Philosophy of hockey sabermetrics: Can hockey accurately be measured?

Mathletic

Registered User
Feb 28, 2002
15,777
407
Ste-Foy
Finally had time to read "The Signal and the Noise" by Nate Silver. Few excerpts I liked from the chapter on baseball.

On the "fear" that analysts would be taking jobs from scouts:

Beane told me the A’s scouting budget is now much higher than it has ever been. Moreover, he said it was the A’s fascination with statistical analysis that led them to increase it. As we’ve seen, baseball players do not become free agents until after six full seasons, which is usually not until they’re at least thirty. As Bill James’s analysis of the aging curve revealed, this often leads clubs to overspend on free agents—after all, their best years are usually behind them. But there is a flip side to this:*before*a player is thirty, he can provide tremendous value to his club. Moreover, baseball’s economics are structured such that younger players can often be had for pennies on the dollar

On objective analysis vs. gut-feel decisions:

“From our standpoint in Oakland, we’re sort of forced into making objective decisions versus gut-feel decisions. If we in Oakland happen to be right on a gut-feel decision one time, my guess is it would be random. And we’re not in a position to be making random decisions and hope we get lucky. If we’re playing blackjack, and the dealer’s showing a four and we have a six, hitting on the sixteen just doesn’t make sense for us.â€

On the decision-making process:

The key to making a good forecast, as we observed in chapter 2, is not in limiting yourself to quantitative information. Rather, it’s having a good process for weighing the information appropriately. This is the essence of Beane’s philosophy: collect as much information as possible, but then be as rigorous and disciplined as possible when analyzing it.

On categorization:

When we can’t fit a square peg into a round hole, we’ll usually blame the peg—when sometimes it’s the rigidity of our thinking that accounts for our failure to accommodate it. Our first instinct is to place information into categories—usually a relatively small number of categories since they’ll be easier to keep track of.

This might work well enough most of the time. But when we have trouble categorizing something, we’ll often overlook it or misjudge it. This is one of the reasons that Beane avoids what he calls “gut-feel†decisions. If he relies too heavily on his first impressions, he’ll let potentially valuable prospects slip through the cracks—and he can’t afford that with a payroll like Oakland’s.
 

Doctor No

Registered User
Oct 26, 2005
9,250
3,971
hockeygoalies.org
When we can’t fit a square peg into a round hole, we’ll usually blame the peg—when sometimes it’s the rigidity of our thinking that accounts for our failure to accommodate it.

When I give modeling presentations, there's a joke that I've probably used to death at this point - we take great care to build these elaborate models that predict human behavior perfectly with great accuracy.

And then imperfect humans have to come along and ruin our perfect model by acting irrationally.
 

Smokey McCanucks

PuckDaddy "Perfect HFBoard Trade Proposal 02/24/14
Dec 21, 2010
3,165
283
I feel like it would take an absolutely immense amount of work to get statistical analysis of hockey to the point where the insight it provides will approach the insight that a knowledgeable and experienced scout would get just from watching and reflecting on a player or team. Of course the benefit is that even if it doesn't approach that level of insight, a collection of stats would be widely available, whether publicly or in-house, you wouldn't be relying on the views and opinions of an individual - you'd be relying on something that's objective and that can be easily disseminated.

Now, if I'm an executive and I'm going to be drafting a kid, I for sure want to have people I trust and whose opinions I value who have seen him play a lot of games in person and got a feel for his game and his tendencies. It's not going to be stats that are ruling the day there, not ever. There are just too many variables in hockey, you can't account for them all in a statistical model, not to the level that you'd have to be able to reach in order to form a real strong opinion.

But for sure those statistical models have a very important place, and it's getting much more significant. It's having an effect already, teams are changing their playstyles and changing their points of emphasis based on - just one thing - the idea that possession correlates to winning over the long haul, it doesn't matter whether you think the possession stats are accurate or useful or not, you still can see that that has had a clear effect on the style of play of a lot of teams.

It's had a huge impact, even when you hear Doughty saying "Corsi's crap," well, a big part of them winning those two Cups was Sutter putting major emphasis on a possession style of play - compare it to when he took the Flames to the Final, or even the first year he was with the Kings, they played a much different style than they do now. Maybe Doughty's right. Maybe he knows it's crap because they have something way better, something that's in Sutter's head - and there's no statistical model that is ever gonna approach the knowledge of a guy that's been in and around the game as long as Sutter. Those models would be for everyone else, who haven't been around that long and don't have that knowledge, or to approximate that knowledge for all the games you just aren't able to watch because you had a game that night or because you want to do a quick overview of some euro leagues you don't get to see or something. That's where the value really lies, to supplement the traditional ways of doing things, not to replace them.
 

Smokey McCanucks

PuckDaddy "Perfect HFBoard Trade Proposal 02/24/14
Dec 21, 2010
3,165
283
I think the biggest problem I have with a lot of these statistical models that are being thrown around now - there's too little attempt to account for the roles different players play and how they are used. Things like qualcomp and zone starts are just scratching the surface and it's almost lip-service, it doesn't go nearly as far in depth as it needs to to provide an accurate measure and give context to some of the other stats. Without that context they can end up being very misleading.

There are a TON of very solid defensive players with very lousy possession stats because they get put out on the ice against opponents' best possession guys and most dangerous scorers. And their role is not to regain and then retain possession; it's to limit the damage those top guys can do and hem them in and then get off the ice.

It's no different than saying (bear with me), why is the checking line's +/- so lousy, they are supposed to be the checking line, right? Well, they are out there against the biggest scoring threats, of course it is going to look lousy. The only way to know whether it is REALLY lousy, though, is to have some theoretical baseline to compare it to, to estimate what the typical line would do against that level of competition, and that's what you base your assessment off of. You can't just say, well Scuderi or Bolland or Callahan, this guy's supposed to be a good defensive guy but his numbers really suck, maybe we were wrong in that assessment. No, the numbers probably suck for that sort of player because the coach knows they are the best defensive guys, the best guys at playing without the puck, so they go out there when the coach figures the other team's top guys are gonna have the puck and he tries to stem the tide, and it's because he's used that way and that's his role that his numbers look bad next to other guys who have other roles that end up giving them pretty numbers.
 

CanadienShark

Registered User
Dec 18, 2012
37,361
10,614
When I give modeling presentations, there's a joke that I've probably used to death at this point - we take great care to build these elaborate models that predict human behavior perfectly with great accuracy.

And then imperfect humans have to come along and ruin our perfect model by acting irrationally.

Pretty sure I've heard that one before. I think economics profs like that one. :laugh:

I myself of 5 years ago saw myself laughing at that now... :facepalm:
 

CanadienShark

Registered User
Dec 18, 2012
37,361
10,614
I feel like it would take an absolutely immense amount of work to get statistical analysis of hockey to the point where the insight it provides will approach the insight that a knowledgeable and experienced scout would get just from watching and reflecting on a player or team. Of course the benefit is that even if it doesn't approach that level of insight, a collection of stats would be widely available, whether publicly or in-house, you wouldn't be relying on the views and opinions of an individual - you'd be relying on something that's objective and that can be easily disseminated.

Now, if I'm an executive and I'm going to be drafting a kid, I for sure want to have people I trust and whose opinions I value who have seen him play a lot of games in person and got a feel for his game and his tendencies. It's not going to be stats that are ruling the day there, not ever. There are just too many variables in hockey, you can't account for them all in a statistical model, not to the level that you'd have to be able to reach in order to form a real strong opinion.

But for sure those statistical models have a very important place, and it's getting much more significant. It's having an effect already, teams are changing their playstyles and changing their points of emphasis based on - just one thing - the idea that possession correlates to winning over the long haul, it doesn't matter whether you think the possession stats are accurate or useful or not, you still can see that that has had a clear effect on the style of play of a lot of teams.

It's had a huge impact, even when you hear Doughty saying "Corsi's crap," well, a big part of them winning those two Cups was Sutter putting major emphasis on a possession style of play - compare it to when he took the Flames to the Final, or even the first year he was with the Kings, they played a much different style than they do now. Maybe Doughty's right. Maybe he knows it's crap because they have something way better, something that's in Sutter's head - and there's no statistical model that is ever gonna approach the knowledge of a guy that's been in and around the game as long as Sutter. Those models would be for everyone else, who haven't been around that long and don't have that knowledge, or to approximate that knowledge for all the games you just aren't able to watch because you had a game that night or because you want to do a quick overview of some euro leagues you don't get to see or something. That's where the value really lies, to supplement the traditional ways of doing things, not to replace them.

I'm of the belief that until an effective metric is devised to rate players, different sorts of "tests" should be run with later round picks. For example, the Habs or Sharks hire more analytics staff with the sole intention of devising a sort of combinational algorithm to find players that typically produce in all stats with a positive correlation to things like goals, assists, etc. Obviously I just summed up what everyone is already trying to do, but I think a team ought to really focus on doing that with 5th, 6th and 7th rounders rather than purely relying on traditional scouting methods (if they aren't already). Even a rebuilding team just stock up on picks in a poor draft year and rely purely on stats for some very late picks.
 

Bomber0104

Registered User
Apr 8, 2007
15,021
6,835
Burlington
Advanced stats should be able to tell me which player is strongest on the puck, and which isn't.

Advanced stats should be able to tell me which player is the best defensively / playing away from the puck

Advanced stats should be able to tell me which player is the best offensively / playing with the puck

Advanced stats is incapable of any of these.

Advanced stats are not advanced at all.

They are gimicky permutations of equally useless stats.
 

jwhouk

Former Cheesehead, Always a Preds Fan
Apr 19, 2004
5,225
49
Valley of the Sun
jwhouk.net
The purpose of advanced statistics is (or at least should be) to paint a picture of what makes for a successful/not successful hockey team.

I would argue that statistics that have little correlation with winning and losing have little use for fans, teams, media, or anyone else with a casual understanding of sport.

Fine, you can tell me a guy's Corsi numbers are above average. What does that mean? Does it have anything to do with scoring goals or wins and losses? Does his "shot differential" mean that he's a floater on a good team, or that he's trying to make plays while on a line with Moe Howard and Doug MacKenzie?

Show me how it affects winning, and I'll pay attention.
 

Bear of Bad News

Your Third or Fourth Favorite HFBoards Admin
Sep 27, 2005
13,361
26,528
I moved these two most recent posts from a thread where they were off-topic, to here.

Enjoy.
 

Ogopogo*

Guest
The real con is convincing so many naïve people that Corsi has some kind of value.
 

Doctor No

Registered User
Oct 26, 2005
9,250
3,971
hockeygoalies.org
Corsi does have "some kind" of value.

If you're disappointed by it, then perhaps you placed too much stock in it to begin with.

But then, you're attempting to derail the thread. Please stick to the topic.
 

Rebuilt

Registered User
Jun 8, 2014
8,736
15
Tampa
Corsi does have "some kind" of value.

If you're disappointed by it, then perhaps you placed too much stock in it to begin with.

But then, you're attempting to derail the thread. Please stick to the topic.

Everything and anything has 'value' if you believe it. I dont think the advanced stats have been able to predict any future events, except the roughly the same corsi stats.
 

Doctor No

Registered User
Oct 26, 2005
9,250
3,971
hockeygoalies.org
Certain aspects of hockey can be accurately measured. Hockey, no.

The nicest thing about never being able to achieve perfection is that we can always continue to make progress towards that level. Indefinitely.

Because hockey will never be able to be perfectly measured, we can always get better at measuring hockey. As a scientist, that's awesome. As someone who loves learning, that's awesome.
 

silverfish

got perma'd
Jun 24, 2008
34,644
4,353
under the bridge
Corsi is not perfect, but it's on to something...

CcgqoUeUMAEEDvS.jpg
 

mrkolice

Registered User
May 26, 2014
1,292
274
Ljubljana, Slovenia
Certain aspects of hockey can be accurately measured. Hockey, no.

that's true, hockey can't be measured directly, almost never. but even rough indirectly-measured estimates about the mechanics of the game can be very useful. we can discuss the reliability or validity of certain parameters, and rightly so, but the bottom-line question is "are these metrics useful".

i'm fascinated about the big advance in hockey stats in recent decade or so. i'm also fascinated about the epistemological side of the questions being asked. but remember, if it's useful - don't throw it away, rather develop it further.
 

Noldo

Registered User
May 28, 2007
1,664
246
Advanced stats should be able to tell me which player is strongest on the puck, and which isn't.

Advanced stats should be able to tell me which player is the best defensively / playing away from the puck

Advanced stats should be able to tell me which player is the best offensively / playing with the puck

Advanced stats is incapable of any of these.

Advanced stats are not advanced at all.

They are gimicky permutations of equally useless stats.

With Coyotes choice of GM, the issue got even more prominent.

Chayka's take here is interesting: calculate the data that shows what experienced hockey minds see, presumably because then you can teach a nobody to track the same thing from video and multiple your effectiveness.

http://www.sportsnet.ca/hockey/nhl/coyotes-john-chayka-nhls-answer-mark-zuckerberg/
 

93LEAFS

Registered User
Nov 7, 2009
33,841
20,903
Toronto
I wonder if anyone would ever come up with a hockey version of WAR.
GVT was a slight attempt at it, and I think Puck shares exist, which is another attempt at it. I don't think a new-age stat will ever gain prominence like WAR has in baseball, because most puck-nerds will acknowledge that luck plays a bigger part of hockey. Another thing is, Baseball is already very statistic based, with every thing being very measurable (only Defence took awhile to measure, and it still difficult), whereas Hockey has a smaller season and has less quantifiable factors that measure success. Even Football, which isn't as fluid as a game as hockey as only recently seen an explosion and in advanced analytics, and my favourite team becoming a guinea pig for it (Browns, and I'm willing to try anything at this point to get a competent QB, whether it be guys with pocket protectors or voodoo priestesses).
 

93LEAFS

Registered User
Nov 7, 2009
33,841
20,903
Toronto
With Coyotes choice of GM, the issue got even more prominent.

Chayka's take here is interesting: calculate the data that shows what experienced hockey minds see, presumably because then you can teach a nobody to track the same thing from video and multiple your effectiveness.

http://www.sportsnet.ca/hockey/nhl/coyotes-john-chayka-nhls-answer-mark-zuckerberg/
I'm interested to see how Chayka does, while slightly envious considering he's a 2 years younger than me and was on the UWO campus as me (Not Ivy though). But I can think of another Ivy grad who did great things for a sports team (Beeston).
 

trentmccleary

Registered User
Mar 2, 2002
22,226
1,100
Alfie-Ville
Visit site
Is anybody else concerned that the prominence of Corsi is making it less reliable?

Like new coaches are are asking players to force more shots from non-scoring areas or UFA's are doing the same to inflate their value? ... Basically creating fraudulent improvements. The Leafs traded down a lot in the past 12 months, but improved their Corsi by 300 (+200sf, -100sa) because they went from an old school coach to a possession coach. This would seem like an example of a fraudulent improvement. Even the league could be showing indications that this is the case; SV%'s increasing while goals remain stagnant or decline (stupid shot inflation?). Perhaps Corsi's downfall will be that it will become difficult to determine the talent from the frauds. Then once Corsi is declines in stature, it will no longer have such a grip on strategy and the measure will be more meaningful.

Or maybe I'm imagining all of this.
 

Doctor No

Registered User
Oct 26, 2005
9,250
3,971
hockeygoalies.org
I don't know if "concern" is the right word, but I'm aware of it.

With any metric that purports to track well with success, until the claim is made you have an advantage in that people aren't incentivized to game it.

It's likely that Corsi tracks well with success because teams that do well in Corsi are also doing other things "right", and Corsi is either a proximate measure, or related to the true effect.

Once teams hear about the correlation, and try to specifically take advantage of it, then they may be doing it at the expense of the thing that's actually valuable.

There's also the likelihood at this point that confirmation and narrative biases sneak into the equation.
 
  • Like
Reactions: Leksand

Random Forest

Registered User
May 12, 2010
14,441
949
Let's also not lose sight of the fact that corsi is still a relatively primitive tool that's been around for almost ten years now. I'd be downright shocked if NHL teams didn't have better, proprietary data and measures that they are more concerned with targeting. I highly doubt NHL GMs and coaches are investing all their resources to simply maximize corsi alone.

The correlation/causation phenomenon referenced above, I would assume, is something that analytically minded managers are keenly aware of (well, they should be keenly aware of). Of course, some managers are not all that bright, so maybe that's not a great assumption. :laugh:
 

Teemu

Caffeine Free Since 1919
Dec 3, 2002
28,756
5,228
Is anybody else concerned that the prominence of Corsi is making it less reliable?

Like new coaches are are asking players to force more shots from non-scoring areas or UFA's are doing the same to inflate their value? ... Basically creating fraudulent improvements. The Leafs traded down a lot in the past 12 months, but improved their Corsi by 300 (+200sf, -100sa) because they went from an old school coach to a possession coach. This would seem like an example of a fraudulent improvement. Even the league could be showing indications that this is the case; SV%'s increasing while goals remain stagnant or decline (stupid shot inflation?). Perhaps Corsi's downfall will be that it will become difficult to determine the talent from the frauds. Then once Corsi is declines in stature, it will no longer have such a grip on strategy and the measure will be more meaningful.

Or maybe I'm imagining all of this.

Goodhardt's law: Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.

That said, there are many more ways at looking at Corsi than simply team CF.
 

Ad

Upcoming events

Ad

Ad

-->