Bear of Bad News
Your Third or Fourth Favorite HFBoards Admin
- Sep 27, 2005
- 13,542
- 27,086
The tricky thing about finding the 'be all, end all' stat
- Thread starter Bear of Bad News
- Start date
will read it but the largest problem is that in other sports, like baseball, where all advanced stats really come from and I love bill James alot, is that one can isolate the hitter and pitcher and get something more accurate from any look at the stats than in a game hockey which which transitions so quickly and has so many more variables in play and thus more chances for variance as well.
hatterson
Registered User
That's one of the significant issues raised in the article. Baseball is basically a perfectly built sport for analytics. It's built on one-on-one matchups that are largely independent. Hockey has the issues of defensemen being influenced by forwards and viceversa.
However, even if hockey was much more one-on-one, the fluid nature of the sport also leaves issues. Baseball is broken down into distinct plays. One guy is offence and one guy/team is defense. A play in hockey can involve rapid shifts from O to D and back again. Sometimes without the puck carrier doing anything different. If I'm rushing through the neutral zone with 3 teammates I'm on offense, but if I'm doing it as their peeling to the bench for a change is that still offense? Is it really just defense by letting them replace themselves?
PenguinMario
Registered User
Plus, virtually every event on a baseball field is recorded, while we're still missing things like zone time, true puck possession, passing data, etc. Even finding out what players were on the ice for a given event can be a chore.
Only football is as difficult, I believe - it's broken into discrete plays like baseball and, to a lesser extent, basketball, but player roles are more highly specialized, and much of the important action takes place away from the ball and isn't recorded. A sample size of 16-20 games per season, roughly 120 plays each, just makes things worse.
There's not a whole lot of interest in building that kind of metric as far as I'm concerned. Even in baseball. Sure WAR is fine when you want a number to tell you how much of an impect a player has. However, for a team in real life, it's close to useless. In order to optimize lineups, defensive strategies and whatnot, you have to use much more meaningful stats. Doesn't matter whether or not baseball is easier to isolate or not.
overpass
Registered User
- Jun 7, 2007
- 5,271
- 2,808
Another major problem with the "all-in-one" metrics is the desire to apply them over any time scale and have consistent results that will sum.
The nature of hockey statistics is that no statistic perfectly describes the exact contribution of a given player to the result that was measured. The best we can do is look at our population of outcomes, determine which statistics describe the contribution of the player toward winning, and assign a corresponding value to those statistics. The resulting metric will be an estimate, not an exact measurement.
Since those values are derived based on the population, they may need to be regressed when applying them to a smaller sample. But the amount of regression that is needed depends on the size of the sample. No all-in-one metrics in hockey will change the amount of regression depending on the size of the sample, because they want the metric to be consistent in terms of game totals adding to seasonal totals adding to career totals.
Let's say an all-in-one hockey metric is created that uses the principle that observed shooting percentage is part randomness and part skill. So player shooting percentage is regressed partially towards league average in order to remove the randomness component, and the amount of regression is picked with reference to the amount of randomness in single season shooting percentage. The problem is that this will overrate the skill component and underrate the randomness component in single game shooting percentage, and it will underrate the skill component and underrate the randomness component in career shooting percentage. When a fixed regression figure is involved in a metric, the metric is only accurate for a particular sample size. In this case the metric would be a valid estimate for single season value (assuming we knew nothing else about the player but his statistics for that season), but not for single game value or career value.
This is an unsolved problem in WAR for baseball. There is not enough regression involved in their fielding metrics over a partial season, which means too much value is given to defensive statistics over a partial season. Similarly, there is too much regression involved in their fielding metrics over a career, which means there is not enough value given to defensive statistics over a career.
The entire problem is based on a misunderstanding of the all-in-one metrics. They are estimates, not counting stats. One possible solution is regressing single season numbers and/or single game numbers to a player mean based on the player's performance in other seasons instead of a general population mean. But I haven't seen any interest in that among the creators of all-in-one metrics - probably because it makes the calculations much more time-consuming.
The nature of hockey statistics is that no statistic perfectly describes the exact contribution of a given player to the result that was measured. The best we can do is look at our population of outcomes, determine which statistics describe the contribution of the player toward winning, and assign a corresponding value to those statistics. The resulting metric will be an estimate, not an exact measurement.
Since those values are derived based on the population, they may need to be regressed when applying them to a smaller sample. But the amount of regression that is needed depends on the size of the sample. No all-in-one metrics in hockey will change the amount of regression depending on the size of the sample, because they want the metric to be consistent in terms of game totals adding to seasonal totals adding to career totals.
Let's say an all-in-one hockey metric is created that uses the principle that observed shooting percentage is part randomness and part skill. So player shooting percentage is regressed partially towards league average in order to remove the randomness component, and the amount of regression is picked with reference to the amount of randomness in single season shooting percentage. The problem is that this will overrate the skill component and underrate the randomness component in single game shooting percentage, and it will underrate the skill component and underrate the randomness component in career shooting percentage. When a fixed regression figure is involved in a metric, the metric is only accurate for a particular sample size. In this case the metric would be a valid estimate for single season value (assuming we knew nothing else about the player but his statistics for that season), but not for single game value or career value.
This is an unsolved problem in WAR for baseball. There is not enough regression involved in their fielding metrics over a partial season, which means too much value is given to defensive statistics over a partial season. Similarly, there is too much regression involved in their fielding metrics over a career, which means there is not enough value given to defensive statistics over a career.
The entire problem is based on a misunderstanding of the all-in-one metrics. They are estimates, not counting stats. One possible solution is regressing single season numbers and/or single game numbers to a player mean based on the player's performance in other seasons instead of a general population mean. But I haven't seen any interest in that among the creators of all-in-one metrics - probably because it makes the calculations much more time-consuming.
Bear of Bad News
Your Third or Fourth Favorite HFBoards Admin
- Sep 27, 2005
- 13,542
- 27,086
Good St. Louis blog about problems with universal player metrics:
http://www.stlouisgametime.com/2014/2/3/5376100/universal-measures-of-player-value-why-they-fail
Basically a crash course in sample size and variations that result.
http://www.stlouisgametime.com/2014/2/3/5376100/universal-measures-of-player-value-why-they-fail
Basically a crash course in sample size and variations that result.
The Mighty Duck Man
R-E-L-A-X
Hell, for how perfect baseball is in analytics, WAR is highly flawed. From overrating defense to seemingly random position values(that have a lot more influence than they should IMO), it's really only good if you want to take a quick glance and get a grasp for where a player's at. At the same time, in any sport, you probably won't do much better when it comes to "universal" stats. Simply put, you're probably not going to get one. Hell, even for quarterbacks in football, QB rating has some very serious flaws.
wgknestrick
Registered User
- Aug 14, 2012
- 5,867
- 2,607
The only tricky thing is figuring out how to separate defensive contributions (or shares) and how they relate towards winning. As this develops, we will start to have a much better grasp on a single "rating" to rule them all.
This is why it is much more difficult to draft defensemen and almost impossible to draft goaltenders compared to forwards. There are not enough data points recorded on defensive plays to work with. For every goal, they record the goal scorer and the 2 players to touch the puck last. They surely don't do anything near that for the defensive players that were scored on. If they start recording these things, the systems will develop behind it.
This is why it is much more difficult to draft defensemen and almost impossible to draft goaltenders compared to forwards. There are not enough data points recorded on defensive plays to work with. For every goal, they record the goal scorer and the 2 players to touch the puck last. They surely don't do anything near that for the defensive players that were scored on. If they start recording these things, the systems will develop behind it.
these data are available already and are being used. For every shot, blocked shot and miss (as well as hits,etc) , we have who was on the ice for both teams. We can link those events with who was on the ice (for and against) to get a sense of who is 'driving' play. This is what the Total Hockey Ratings (THoR) does and what the Expected Goals model of Brian Macdonald does.
Now these data are only available for the NHL and so they don't have much impact on drafting, yet.
wgknestrick
Registered User
- Aug 14, 2012
- 5,867
- 2,607
The same THoR that has Tyler Kennedy as an elite, top 3 forward in the NHL? Or the THoR that list Kimmo Timonen as the best Defenseman in league?
Based upon those data and that methodoloy used, yes.
Isn't it funny that Kennedy has the 2nd highest rate of shots/TOI of anyone in the NHL over that period. Tough for the other team to score when you're putting up so many shots.
Wouldn't be much of a method if we didn't learn something.
Isn't it funny that Kennedy has the 2nd highest rate of shots/TOI of anyone in the NHL over that period. Tough for the other team to score when you're putting up so many shots.
Wouldn't be much of a method if we didn't learn something.
Those real-time stats and inputs being used are recorded with no consistency or standards of regulation, and thus any ratings or models that draw from them are essentially completely worthless.
They are far from the only variables to consider either (in fact, some "stats" that people think suggest defensive ability, have no correlation at all and are sometimes actually more representative of the opposite), so the model would be essentially completely worthless anyway.
wgknestrick
Registered User
- Aug 14, 2012
- 5,867
- 2,607
I wonder what his SH% is? Probably high
I have watched more of TK than most people and his "shot production" needs to be taken with a huge grain of salt. He constantly throws the puck on net with no intent on scoring (Usually without a man even being there for deflections or rebounds). He is hoping for rebounds/misplays etc. We can argue that there is no such thing as a bad puck on net, but he also interrupts many more dangerous chances by blindly throwing pucks on nets from bad angles and huge distances. Many times the goaltenders just freeze it or play goes the other way.
Why doesn't TK have one of the highest 5v5 GF/60 in the league?
Why doesn't' TK have one of the highest 5v5 GF% in the league?
All shots and shot attempts are not equal, but THOR's system assumes they are. This is why it does not correctly identify TK for who he is. We are missing data to weed the TKs out of THOR's ranking. It is even worse for defensive data. Just because THOR uses all the "available data" does not mean it comes out with the proper results. The NHL needs to record more. THOR is too easy to pick apart to use it as a reference for a single stat. It still needs significant development.
Comparison of Crosby vs TK shooting locations (may have to type in players as links doesn't work all the time):
http://www.sportingcharts.com/nhl/icetrack/?r1Id=296786&r1Year=2013&r1eventType=%27SHOT%27%2C+%27GOAL%27&r1type=1&r1strength=&r2Id=325554&r2Year=2013&r2eventType=%27SHOT%27%2C+%27GOAL%27&r2type=1&r2strength=#1
I wonder what his SH% is? Probably high
I have watched more of TK than most people and his "shot production" needs to be taken with a huge grain of salt. He constantly throws the puck on net with no intent on scoring (Usually without a man even being there for deflections or rebounds). He is hoping for rebounds/misplays etc. We can argue that there is no such thing as a bad puck on net, but he also interrupts many more dangerous chances by blindly throwing pucks on nets from bad angles and huge distances. Many times the goaltenders just freeze it or play goes the other way.
Why doesn't TK have one of the highest 5v5 GF/60 in the league?
Why doesn't' TK have one of the highest 5v5 GF% in the league?
All shots and shot attempts are not equal, but THOR's system assumes they are. This is why it does not correctly identify TK for who he is. We are missing data to weed the TKs out of THOR's ranking. It is even worse for defensive data. Just because THOR uses all the "available data" does not mean it comes out with the proper results. The NHL needs to record more. THOR is too easy to pick apart to use it as a reference for a single stat. It still needs significant development.
Comparison of Crosby vs TK shooting locations (may have to type in players as links doesn't work all the time):
http://www.sportingcharts.com/nhl/icetrack/?r1Id=296786&r1Year=2013&r1eventType=%27SHOT%27%2C+%27GOAL%27&r1type=1&r1strength=&r2Id=325554&r2Year=2013&r2eventType=%27SHOT%27%2C+%27GOAL%27&r2type=1&r2strength=#1
THoR doesn't assume that all shot attempts are equal (a la Corsi or Fenwick), it gives different values to different shot locations and different shot types. His scoring rate is low, no doubt and that is not currently in THoR because of the work done on SH% saying that it is too noisy in the short term. Nobody is using THoR blindly.
Here's my question for you: what are the proper results? And how do you know?
Again, I will readily concede THoR is not perfect but that should not be the enemy of the good.
