I think advanced statistics are an invaluable tool, but I think a lot of stats people get so bogged down in numbers that they sometimes loose track of the essence of the game itself. I find instances where the statistician overreaches and claims his stats how much more than they actually do to be unhelpful.
I agree. Some of the best studies and metrics are the simplest in principle. For instance, Overpass' adjusted plus-minus is very simple in principle. Even the math used to estimate SH/PP GF/GA on ice is relatively simple. It's recognizing how to use the available data properly is not always so simple. In contrast, some "all-in-one" metrics that utilize more complicated math without reasoned support for the methodology are of little use and can be misleading.
Two specific criticisms of mainstream hockey analytics:
1) The assumption that "conventional hockey wisdom" is worthless if it can't be statistically proven. Countless times, I have seen conventional wisdom tossed aside by the latest man with "The Answer" only to see later statistical work indicate that yes, the conventional wisdom had something to it. Off the top of my head, "goaltenders have no effect on shots against" and "skaters have no effect on save percentage" to be mindbogglingly ignorant statements (especially the second one), yet for a time were (and in same cases still are) accepting as truisms by some in the hockey analytics community.
I think the responsible thing to do would be to start with the assumption that conventional wisdom has a grain of truth to it, and should only be thrown out in the face of convincing evidence to the contrary (which we do have in quite a few cases). The current assumption seems to be that conventional wisdom should be dismissed off the bat unless convincing (statistical) evidence can be found in favor of it.
Taken to the extreme, the collective opinions of paid NHL GMs and coaches are dismissed as those of a bunch of meatheads stuck a past without the newfangled stats.
I think the opinions of knowledgeable hockey people should be given some respect. However, it should not stop there, but rather be an impetus to further support or (at least partially) disprove that "conventional wisdom."
2) The tendency to dismiss every effect that can't be easily explained statistically as "luck." The easiest example I can think of is the commonly used blanket statement that any increase or decrease in playoff performance is due to random variation. This would make sense of players were simply machines driven by probability engines, but completely ignores the psychological difference between the playoffs and the regular season both in terms of pressure and in terms of playing the same opponent over and over again.
This might be a specific example of #1 (dismissing conventional wisdom as luck out of hand).
Back to me now: In case you haven't noticed, I'm much better at criticizing studies than coming up with my own
While it may be rash to just dismiss conventional wisdom, it's also unwise to blindly accept it. I think what's most important is to use logic when analyzing and assessing data. For instance, playoff performance is difficult to study, because the conditions are unequal. One could say "Messier's playoff PPG is similar to his regular season PPG, which is unusual, so he must be one of the most clutch playoff players." This overlooks some important factors:
- Messier's regular season PPG was lower than a lot of other great players, and it's generally easier to maintain a lower PPG than a higher one.
- Messier was not in the playoffs his last several seasons, so his regular season PPG is decreased by these lesser seasons, while his playoff PPG isn't. Also, a player's playoff games each season varies a lot more than his regular season games, and since league avg. scoring varies each season, this also has an effect.
- Messier was often on strong regular season teams, which were often stronger than their opposition in the playoffs. Such strong teams will tend to outperform their opposition, so the players on strong teams will tend to outperform players on weaker teams.
It shouldn't be expected that everyone perform or cite a study to support their position. What's frustrating is that many abandon simple logic when assessing the data available. I can understand when someone doesn't believe they have the math skills to perform or understand a study. I can't understand when they refuse to use logic.