Right. So the guy who came up with these stats is saying the "eye test" is extremely important and analytics by themselves can be misleading without that context. Imagine that.
Think of it this way...an average hypothetical defenseman playing 20 minutes a night could be on the ice for 1/3 of the shots for and against, in theory. Off the top of my head and for simplicity's sake, NHL averages are somewhere around 30 shots a game with shooting percentage being around 10%, maybe less. So that hypothetical defenseman might be on the ice for 10 shots for, 10 shots against, and one goal for each team. Again, purely vanilla and hypothetical.
Let's say that d-man is on the ice for 1 more shot for and one less shot against, an 11/9 split. I believe the CF% would then be 55%, which by fancy stat standards is very good, yes?
To me what this means is, with one shot gained per game and one shot against removed, over the course of a season that player is on the ice for about 8 more goals for, and 8 fewer goals against (1 goal every 10 shots, 10 shots every 10 games, about 80 games played). Not necessarily contributing to those situations, just on the ice for them.
So my question is, does that player become "elite" simply based on that 5% distinction over the average hypothetical guy who breaks even statistically? Because of a one shot swing per night?
To me that's where the "eye test" comes in. If you can see over the course of those 80+ games that Mr. 55% is contributing to the possession and the scoring, and isn't a passenger, then the stat confirms your impression. If however you know from experience that that player has benefitted from the work of others or some cupcake matchups, then that 55% in a vacuum means much less. You can try to tease some of that out in QOC and other relative measurements, but as even the founder of the stat says those areas are vastly underdeveloped and you need the "eye test" to give those stats their final meaning.
So, yeah. Welcome stat dude. Tell us something good. But let's hope BT and his experienced eye has the final say.