Philosophy of hockey sabermetrics: Can hockey accurately be measured?

[David Dennison]

http://www.fastcodesign.com/1670059/moneyball-20-how-missile-tracking-cameras-are-remaking-the-nba

The problem is data collection, how come the NHL can't do something like this? It would even be easier for hockey IMO, you could simply put some kind of chip into the equipment. You could calculate speed, find where shot and goals are coming from, zone entries, or just about anything.

 

[Iain Fyffe]

Goalie stats are ALL team dependent to varying degrees.

All individual stats are team dependent to varying degrees. Except maybe PIM.

Much of hockey analytics sees the goalie as completely seperate from the rest of the hockey team and considers save % a perfect measure of goalie and shot differential as a perfect measure of the skaters.

Can you need to quantify "much" here, and avoid spurious claims that any serious hockey analyst would call any hockey stat "perfect", rather than "the best we have at this time"?

At least goals against average isn't nearly so prone to abuse, as quality of team is explicitly a part of the stat

Which makes it a good team stat, but a bad individual goalie stat.

Just go back and watch the games.

Drink!

 

[TheDevilMadeMe]

All individual stats are team dependent to varying degrees. Except maybe PIM.

True to an extent, but goals, assists, and points are less team dependent than save percentage, especially for star-calibre players who tend to drive the offense.

Can you need to quantify "much" here, and avoid spurious claims that any serious hockey analyst would call any hockey stat "perfect", rather than "the best we have at this time"?

You want me to quantify the percentage of hockey analytics people who believe in the single stat theory of goaltending? You realize that's an impossible task, right?

And is not the very premise of the single stat theory of goaltending that everything one needs to know about goaltending is captured by save percentage?

Which makes it a good team stat, but a bad individual goalie stat.

Depends on how you use it. It would be the best stat for complain goalies who play on the same team (assuming similar lineups, etc)

 

[Iain Fyffe]

True to an extent, but goals, assists, and points are less team dependent than save percentage, especially for star-calibre players who tend to drive the offense.

Sorry to be pedantic, but my claim that all stats are team-dependent to varying degrees is true to an extent? What you say above, that some stats of some players are less team-dependent than others, fits exactly with what I said.

You want me to quantify the percentage of hockey analytics people who believe in the single stat theory of goaltending? You realize that's an impossible task, right?

No, I'd rather you stay away from blanket claims about what analysts do and think, especially ones that claim an analyst believes any hockey stats to be "perfect."

And is not the very premise of the single stat theory of goaltending that everything one needs to know about goaltending is captured by save percentage?

I suppose that depends on how you define the "single stat theory of goaltending". Who has defined it this way? Who claims that everything one needs to know about goaltending is captured by save percentage?

I've certainly seen (and made) the claim that if you're going to use a single stat for goaltenders, it should be save percentage, because it is the least team-dependent of the goalie stats. But that's not what you're saying here. You're again claiming that a serious analyst would suggest that save percentage is a "perfect" stat.

Depends on how you use it. It would be the best stat for complain goalies who play on the same team (assuming similar lineups, etc)

Would it? If goalies play on the same team, wouldn't much of the issue of team-dependency of save percentage be erased? When comparing goalies on the same team, why would GAA be better than save percentage?

 

[Alan Jackson]

I'm certainly not an expert on advanced Hockey stats by any means, but my opinion is that these types of numbers are useful in a present context only. That is to say, they are a good indication of how well a player might be performing in current season in his current role.

I don't think advanced stats are a great indicator of future results.

If a player maintains good CORSI or FENWICK numbers playing in a certain role with certain players on a certain team, that does not mean those numbers will be consistent once that player is traded or signs with a new team.

I also wonder sometimes about how such stats are used. I'm a Canucks fan. Again, I'm not a disciple of Sabremetrics, so somebody can correct me if I'm wrong - but I believe Manny Malhotra had some of the worst numbers on the Canucks last season. But, that's okay because he faced tough competition and the vast majority of his shifts started in the defensive zone. So what are those numbers supposed to tell me about Malhotra's season?

Cody Hodgson was apparantly given all the plumb zone starts against weaker opposition, and scored at a 20 goal pace in limited ice-time, but I'm told he was hurting the team because his possession stats were poor, even against the weaker opposition.

My personal opinion is that some people in hockey rely entirely too much on advanced stats, and that the basic point of the game has always been the same. If you score more than your opponent, you're doing something right.

Would like some feedback on the above - I think this is a fascinating topic, and I'm eager to learn from you guys.

 

[overpass]

I suppose that depends on how you define the "single stat theory of goaltending". Who has defined it this way? Who claims that everything one needs to know about goaltending is captured by save percentage?

I've certainly seen (and made) the claim that if you're going to use a single stat for goaltenders, it should be save percentage, because it is the least team-dependent of the goalie stats. But that's not what you're saying here. You're again claiming that a serious analyst would suggest that save percentage is a "perfect" stat.

I think you'll find save percentage is defined as "everything one needs to know about goaltending implicitly rather than explicitly. While few analysts will say that save percentage is perfect, there are some who only use save percentage or derivatives of save percentage when it comes to goalie analysis. Actions speak louder than words, right?

 

[Iain Fyffe]

I think you'll find save percentage is defined as "everything one needs to know about goaltending implicitly rather than explicitly. While few analysts will say that save percentage is perfect, there are some who only use save percentage or derivatives of save percentage when it comes to goalie analysis. Actions speak louder than words, right?

Derivatives of save percentage? Such as those that attempt to capture shot quality as well? That's not the same thing as just using save percentage, because it explicitly includes information not otherwise contained in save percentage.

You have to consider why save percentage might be used as the only stat. Is it because the analyst believes it's the best stat available for the job at the time? Because that's a perfectly valid reason to use it, and does not mean it's believed to be the best possible stat.

 

[overpass]

Derivatives of save percentage? Such as those that attempt to capture shot quality as well? That's not the same thing as just using save percentage, because it explicitly includes information not otherwise contained in save percentage.

You have to consider why save percentage might be used as the only stat. Is it because the analyst believes it's the best stat available for the job at the time? Because that's a perfectly valid reason to use it, and does not mean it's believed to be the best possible stat.

I mean derivatives of save percentage such as "Quality Starts", which is simply GP and SV% with a distribution element. Or Goalie Point Shares, which is again SV% and playing time with a league adjustment.

Using save percentage as a base from which to make adjustments for other factors is certainly more comprehensive.

 

[Iain Fyffe]

I mean derivatives of save percentage such as "Quality Starts", which is simply GP and SV% with a distribution element.

So is the claim that Quality Starts have been presented as "all you need to know" about goaltenders? I know Rob Vollman, who invented the Quality Start stat, quite well and I don't recall him arguing that it's the be-all and end-all of goalie analysis, or indeed that it is a perfect stat. It's an arrow in the quiver, and I don't think it's fair to paint someone who uses it as saying it's all you need.

 

[overpass]

So is the claim that Quality Starts have been presented as "all you need to know" about goaltenders? I know Rob Vollman, who invented the Quality Start stat, quite well and I don't recall him arguing that it's the be-all and end-all of goalie analysis, or indeed that it is a perfect stat. It's an arrow in the quiver, and I don't think it's fair to paint someone who uses it as saying it's all you need.

I have made no such claims about quality starts or Rob Vollman specifically.

I am saying that even if someone does not explicitly say that save percentage explains everything, their analysis and resulting conclusions may be performed under this assumption.

 

[Iain Fyffe]

I am saying that even if someone does not explicitly say that save percentage explains everything, their analysis and resulting conclusions may be performed under this assumption.

I think that if someone, such as TheDevilMadeMe, wants to claim that analysts believe save percentage to be a perfect stat, he/she should need to provide some evidence for that, beyond "they didn't say they don't think that..."

 

[Hivemind]

I think the point TheDevilMadeMe is attempting to make is that we tend to see a lot more goaltender analysis based off of sv% than any other stats. I would agree with that point, but not any conclusions reached past that.

Sv%, or variations off of it (ES sv%, SH sv%), is almost certainly the most widely used statistical metric for goaltenders among stats bloggers. I don't think any of them would argue that it's perfect, and the blogs/sites that track scoring chances often prefer to incorporate that as opposed to raw shot totals. But it's pretty easy to see why it's the best of the commonly available and easily referenced stats (W/L/T, GAA, sv%, MIN, GP, SO, etc.).

 

[Iain Fyffe]

I think the point TheDevilMadeMe is attempting to make is that we tend to see a lot more goaltender analysis based off of sv% than any other stats. I would agree with that point, but not any conclusions reached past that.

I suspect that's true. Given that, of the basic stats, save percentage is the least team-dependent it shouldn't be surprising. And it shouldn't be taken as evidence that the limitations of the stat are not recognized.

 

[habsfanatics*]

I suspect that's true. Given that, of the basic stats, save percentage is the least team-dependent it shouldn't be surprising. And it shouldn't be taken as evidence that the limitations of the stat are not recognized.

I'm not even sure this is true. Sv% like all other goaltending stats is heavily dependent of the quality of team. How do you qualify what is least dependent?

There are numerous examples of goalies putting up great SV% on very good defensive teams. Brian Elliot just did this, so did Halak to a lesser extent.

Even goalies on the same team doesn't necessarily tell you much. Most often, backup goalies play all the weaker opponents, putting up a .920 Sv % vs columbus would be a lot easier than doing it against the Penguins.

 

[Iain Fyffe]

I'm not even sure this is true. Sv% like all other goaltending stats is heavily dependent of the quality of team. How do you qualify what is least dependent?

There are numerous examples of goalies putting up great SV% on very good defensive teams. Brian Elliot just did this, so did Halak to a lesser extent.

Even goalies on the same team doesn't necessarily tell you much. Most often, backup goalies play all the weaker opponents, putting up a .920 Sv % vs columbus would be a lot easier than doing it against the Penguins.

This also affects GAA. Team defence affects both the quantity and the quality of shots on the goalie. Save percentage ignores shot quantity (though quantity and quality are certainly somewhat interrelated in many cases).

 

[Verviticus]

If a player maintains good CORSI or FENWICK numbers playing in a certain role with certain players on a certain team, that does not mean those numbers will be consistent once that player is traded or signs with a new team.

I could say the same about every stat that is or was ever used, though.

I also wonder sometimes about how such stats are used. I'm a Canucks fan. Again, I'm not a disciple of Sabremetrics, so somebody can correct me if I'm wrong - but I believe Manny Malhotra had some of the worst numbers on the Canucks last season. But, that's okay because he faced tough competition and the vast majority of his shifts started in the defensive zone. So what are those numbers supposed to tell me about Malhotra's season?

Cody Hodgson was apparantly given all the plumb zone starts against weaker opposition, and scored at a 20 goal pace in limited ice-time, but I'm told he was hurting the team because his possession stats were poor, even against the weaker opposition.

these are both things that are true, but you're seeing people misrepresent the argument for both of these cases, and then associating that misrepresentation with correct interpretation.

in malhotra's case, you should be taking his 'advanced stats' and apply them to common criticisms of him (creates no offence, always in his own zone) to form an opinion on his value as a player. he's not in a situation to create offence and he starts in his own zone all the time, so why would those complaints be useful or valid?

again, with hodgson, you're taking a bad argument at face value. being up against soft competition and getting easy starts only suggests that his pace is supported by those facts, not that he's inherently a bad player or hurting the team by doing so. the sedins get the same treatment, but nobody would ever suggest that they are not helping the team

 

[Feed Me A Stray Cat]

Statistics can definitely measure hockey in an accurate light. You just need the right statistics.

If/when the NHL starts tracking individual puck possession, puck touches, passes, etc., we'll see a big revolution with this stuff.

 

[habsfanatics*]

I think that the use of advance statistics is most definitely a helpful tool, but it should never be expected to replace advanced scouting, however, I think it would be prudent for most teams to adopt their use and get the best of both worlds.

Sometimes we have predetermined biases that numbers can be useful to counter or augment, however, I find the numbers are often used to justify the predetermined biases we all have, it's a bit of a catch 22. I don't think you can build a roster with sabremetrics, but believe it could be used as a useful tool as long as we realize the limits. For matchups, zone starts ect it can be highly effective imo. A player like MAB can become extremely valuable if used only in a limited role maximizing his strengths.

Some people like to trot it out as the be all end though and I don't think they do analysts much good in doing so. Many argued that Scott Gomez was bound to bounce back, even though he went scoreless an entire year. Gomez could be used as the poster boy of why these numbers are limited in scope and practical use or Gomez could just be the anomaly in all this.

I find them extremely limited in determined future results, especially from season to season in the overall team aspect. Roster turnover ect play a much greater role.

 

[Freudian]

I think advanced statistics could be quite useful when it comes to analyzing hockey, but the problem is that the data we have isn't good enough. Right now we have not very useful data being farmed for correlation and people shouting eureka when they stumble upon something with slightly higher correlation than pure chance.

I don't think we will have access to good enough data for it to be truly meaningful at the moment. There is enjoyment to be had from people into advanced stats speaking in definitive terms about the ability of hockey players purely from what they read into the numbers on behindthenet. I guess there is value to that.

 

[TheDevilMadeMe]

Sorry to be pedantic, but my claim that all stats are team-dependent to varying degrees is true to an extent? What you say above, that some stats of some players are less team-dependent than others, fits exactly with what I said.

Yes, agreed.

No, I'd rather you stay away from blanket claims about what analysts do and think, especially ones that claim an analyst believes any hockey stats to be "perfect."

So you have not seen analysts rank gaolies's by save percentage (or it's derivatives) and rank skaters by things like Corsi or Fenwick without acknowledging the limitations?

I suppose that depends on how you define the "single stat theory of goaltending". Who has defined it this way? Who claims that everything one needs to know about goaltending is captured by save percentage?

Overpass covered this one.

I've certainly seen (and made) the claim that if you're going to use a single stat for goaltenders, it should be save percentage, because it is the least team-dependent of the goalie stats. But that's not what you're saying here. You're again claiming that a serious analyst would suggest that save percentage is a "perfect" stat.

The ones who do certainly consider themselves serious...

Would it? If goalies play on the same team, wouldn't much of the issue of team-dependency of save percentage be erased? When comparing goalies on the same team, why would GAA be better than save percentage?

When comparing goalies on the same team, team effects are removed. This therefore removes nearly 100% of the variables that affect GAA (pretty much everything but chemistry between a goalie and his team), but there are still variables affecting save percentage - goalie style, rebound control, puck handling, poke checking, etc.

The perfect goalie stat would be GAA with team variables removed, but of course that's pretty much impossible to do with any precision for goalie's playing for different teams.

Thinking about it, quality of opponents would affect GAA more than save percentage, so you have a point there.

I think that if someone, such as TheDevilMadeMe, wants to claim that analysts believe save percentage to be a perfect stat, he/she should need to provide some evidence for that, beyond "they didn't say they don't think that..."

I don't know if you're being deliberately obtuse or if you are just unaware of the studies that are out there. I'm not going to take the time to come up with a comprehensive list of names, but you can start with The Contrarian Goaltender and move on to the majority of statistical goalie studies on the history board of this site

I suspect that's true. Given that, of the basic stats, save percentage is the least team-dependent it shouldn't be surprising. And it shouldn't be taken as evidence that the limitations of the stat are not recognized.

No, but ranking goalies by save percentages and using save percentage to make statements like "Martin Brodeur is just Curtis Joseph with a better team" is evidence that the limitations of the stat aren't vein recognized.

 

[TheDevilMadeMe]

Statistics can definitely measure hockey in an accurate light. You just need the right statistics.

A bold statement that stats can "definitely" measure hockey when you recognize in the same post that we don't even have the proper stats yet to begin:

If/when the NHL starts tracking individual puck possession, puck touches, passes, etc., we'll see a big revolution with this stuff.

 

[PresidentCamacho*]

Statistics can definitely measure hockey in an accurate light. You just need the right statistics.

If/when the NHL starts tracking individual puck possession, puck touches, passes, etc., we'll see a big revolution with this stuff.

I definitely agree this is the direction hockey will move in regard to collating useful information for player effectiveness.

Analyzing puck movement is the key to the game itself. I know of certain technologies used in soccer to spatially analyze player velocity and distance by use of chipped jerseys, so I wonder if that could apply to a hockey jersey, stick, and puck?

It'd make analyzing the game easy enough that you won't have to watch the game, necessarily.

 

[almostawake]

Statistics can definitely measure hockey in an accurate light. You just need the right statistics.

If/when the NHL starts tracking individual puck possession, puck touches, passes, etc., we'll see a big revolution with this stuff.

It's not on the NHL to do this, it's on the teams to innovate. That is, more or less, how it has happened in all the other sports. Some teams in the NHL are almost certainly doing it by now anyways.

http://www.fastcodesign.com/1670059/moneyball-20-how-missile-tracking-cameras-are-remaking-the-nba

The problem is data collection, how come the NHL can't do something like this? It would even be easier for hockey IMO, you could simply put some kind of chip into the equipment. You could calculate speed, find where shot and goals are coming from, zone entries, or just about anything.

This can be done for hockey. There are two main reasons STATS inc. hasn't done it yet:

1) They don't feel they're likely to make as much money from hockey as from other sports. Right now they are focusing on developing optical tracking for football, which I think we can all agree is likely a more lucrative venture.

2) Optical tracking of the puck is very hard, perhaps even impossible with out getting the league to agree to implant a chip in it. They did that for FOX, but it's less likely they'd accommodate a company like STATS inc. that plans on selling the data.

I definitely agree this is the direction hockey will move in regard to collating useful information for player effectiveness.

Analyzing puck movement is the key to the game itself. I know of certain technologies used in soccer to spatially analyze player velocity and distance by use of chipped jerseys, so I wonder if that could apply to a hockey jersey, stick, and puck?

It'd make analyzing the game easy enough that you won't have to watch the game, necessarily.

I know they're using chips in soccer now butoptical tracking has been going on there for several years now. The advantage with chips is they are faster, so you can analyse and display the data faster. From what I understand, the way they're going to use the chip data is to generate interesting statistics for broadcast in (essentially) real time, to enhance the viewing experience.

The disadvantage with chips is that the players need to be wearing the special jerseys. Of course that is obvious, and probably doesn't seem like a big deal, but it prevents you from building a large scale scouting program around the technology. It also prevents you from trying to mine historical data.

Granted, current optical recognition techniques require special cameras and several angles, but in the long run those limitations may be overcome through algorithmics. In the case of chipped jerseys, there's simply no way to collect the data for past games.

The funny thing is, as much as people focus on Baseball when talking about analytics, the truth is the current leader is the NBA, by far. The rate at which data collection and analysis techniques are evolving in Basketball is nuts. The traditional Baseball sabre guys sitting around playing with their slide rulers while the the Basketball guys are learning Hadoop.

 

[Iain Fyffe]

So you have not seen analysts rank gaolies's by save percentage (or it's derivatives) and rank skaters by things like Corsi or Fenwick without acknowledging the limitations?

I have seen analysts rank players in this way. Are there any that do so while saying "this is all you need to know about these players"? I know that when using Corsi in player evaluations at Hockey Prospectus, it was just one tool in the toolbox.

When comparing goalies on the same team, team effects are removed. This therefore removes nearly 100% of the variables that affect GAA (pretty much everything but chemistry between a goalie and his team), but there are still variables affecting save percentage - goalie style, rebound control, puck handling, poke checking, etc.

These are characteristics of a goaltender's play, not variables that change from season to season for that goalie or if the goalie moves to a new team.

The perfect goalie stat would be GAA with team variables removed, but of course that's pretty much impossible to do with any precision for goalie's playing for different teams.

And this is why save percentage is considered a better stat than GAA for goalies, because it removes one team variable: shots allowed.

I don't know if you're being deliberately obtuse or if you are just unaware of the studies that are out there. I'm not going to take the time to come up with a comprehensive list of names, but you can start with The Contrarian Goaltender and move on to the majority of statistical goalie studies on the history board of this site

Sorry, chum, but the person making the positive assertion needs to provide the evidence. If you believe that, in a blanket statement, analysts believe certain stats are perfect, you need to back that up. This post, for instance, addresses Brodeur's save percentage and specifically discusses shot quality. The argument made is not that Brodeur is overrated because his save percentage is often low, it's because his save percentage is often low and he faces relatively low-quality shots.

I'm not saying I buy that argument completely, but you are misrepresenting it. And you're not going to win any points by resorting to ad hominems and then refusing to provide evidence where it is required.

No, but ranking goalies by save percentages and using save percentage to make statements like "Martin Brodeur is just Curtis Joseph with a better team" is evidence that the limitations of the stat aren't vein recognized.

I don't doubt someone has said this at some point. Direct your ire at them rather than at analysts in general. Stay away from the blanket statements, as I believe I said some time ago.

 

[alanschu]

Players who are clutch one year are suddenly not clutch the next. Cam Ward is so clutch that he usually can't even get his team into the playoffs. If it were a real thing, it would be observable and predictable, and not rely on ex post facto explanation. If clutchiness can only be identified after the fact, chances are you're attaching meaning to patterns that are not necessarily real patterns.

What are your thoughts on players being "hot" or "cold?" Is it simply a pattern that is based on random chance?


Hockey tends to have a lot of additional variables when it comes to taking a shot on net for a goal, so I'll use basketball since the shot is more isolated.

Wilt Chamberlain scored 100 points one game, and in doing so he ended up shooting an uncharacteristic 28/32 from the free throw line. He was roughly a 50% free throw shooter (using 50% for simplicity), and it we're just using a random variable to determine the likelihood of him going 28/32 from the free throw line, then the odds of him doing that are about 0.000837% (underestimate since his FT% over his career was 51.1%)

The thing is, broken down purely by physics, if someone shoots a free throw and makes it, biomechanically if they do the exact same thing they'll always make the free throw. Is there no physiological difference that could cause a player to behave in a more biomechanically consistent way when performing physical activity. (This is ignoring psychological factors that may be at play as well).


Obviously no player is going to be perfectly consistent in how they perform an action every single time, and other factors such as fatigue and especially defenses will also affect their ability to be consistent.


It's not something that appears to be particularly measurable, and very likely not something that can be easily predicted, but at this point I'm not keen on outright dismissing it. Sam Gagner's 8 point night is likely something he'll never do again and the odds that he ever accomplish it in his career were already quite low, but are there personal factors from within him that may have tipped the scales rather than pure random variation?