Point Shares? (Warning: Thread for stat dorks:))

[reckoning]

Hockey-Reference.com has recently added point shares to their statistics.

There's a list of year-by-year leaders here: http://www.hockey-reference.com/leaders/ps_yearly.html

Here's a lengthy explanation of the stat: http://www.hockey-reference.com/about/point_shares.html

Basically it's a system to attempt to determine for each player how many of a teams points he was responsible for, similar to Bill James Win Shares for Baseball. There have been several hockey analysts who have attempted similar systems to this over the past few years.

My opinions on it:

Positives:

- The core of the stat is based on how often the players team wins. That's how it should be since the point of hockey is winning and not accumulating individual statistics. But theoretically a player wouldn't do as well if he was on a stacked team with several players better than him.

- It does attempt to combine offensive and defensive contributions of skaters; something most stats lack


Negatives:

- It uses icetime as a key factor in determining a players defence rating (as it should be) , but (if I'm reading it correctly) doesn't make any distinction between ES, PP or PK minutes, which would be a huge factor considering that PK minutes are far more indicative of a players defensive worth. The defence ratings of top scorers like Lafleur, Gretzky and Jagr seem way out of whack.

- It uses Goals Created for the offence ratings which places more emphasis on goals than assists. It may be to reduce the impact of secondary assists, but this hurts the scores of playmakers who often are the ones most responsible for the goal. They have Milan Hejduk as the top player in '02-'03 instead of Peter Forsberg primarily because of their respective goals/assists ratios. I'd love to meet the person who honestly believes Hejduk was the better player that year.

- They give ratings to all seasons, even though many of the key stats (plus/minus, icetime, save percentage) are only available for the recent years. As a result, this limits how high players from the eras before these stats can separate themselves from the rest of the league. Therefore, it makes it easier for players from the last decade or so to rack up better point share seasons. For example, most of the highest defensive ratings for forwards have been in the last decade http://hkref.com/tiny/r0uHO. This skews the all-time rating lists in favour of modern players.



Overall, I thought it was somewhat interesting. Klein and Reif said in one of their compendiums that a single statistic that could define a players total contribution was the "holy grail" of hockey fans who loved to research stats. I've always felt that a hockey version of Win Shares could be a valid statistic, but there are still some obvious kinks to be worked out. Since there are some hockey sabremetric fans on this board, I'd be curious about their opinions on this stat and suggestions on how it could be better.

Click to read more...

 

[overpass]

I was just looking at this today. IMO it needs some work. GVT, by Tom Awad of Puck Prospectus, is comparable and better implemented.

I also noticed that there was no mention of an ES/PP/PK breakdown. That seems like something that is obviously wrong and should be fixed. GVT is a single number stat that handles this better.

The goals/assists thing is tricky. It's possible that there is some group of hockey players for whom goals should be weighted higher than assists. But when it comes to the stars, I think an assist is as good as a goal. I think Point Shares gets this wrong, and GVT does too.

The defensive adjustment looks bad to me. They include plus-minus numbers. But plus-minus is driven by offence at least as much as defence. The sample calculation uses Paul Coffey as an example. It's clearly incorrect to give Coffey a high number of defensive points based on his scoring-driven plus-minus - yet that is what the sample calculation shows.

I think GVT has already done the one-number stat and done it better. The real issue with these one-number stats is that the available data is insufficient. Hockey stats just don't give enough information for these one-number stats. Even the formulations for recent seasons that have time on ice data fail to take into account quality of competition - which is among the biggest contributions that defencemen provide, and is important for forwards as well.

I noticed hockey-reference is using point shares now to sort the players on a team, instead of points. It's fine for that purpose. I wouldn't use it for anything more.

 

[Infinite Vision*]

I was just looking at this today. IMO it needs some work. GVT, by Tom Awad of Puck Prospectus, is comparable and better implemented.

I also noticed that there was no mention of an ES/PP/PK breakdown. That seems like something that is obviously wrong and should be fixed. GVT is a single number stat that handles this better.

The goals/assists thing is tricky. It's possible that there is some group of hockey players for whom goals should be weighted higher than assists. But when it comes to the stars, I think an assist is as good as a goal. I think Point Shares gets this wrong, and GVT does too.

The defensive adjustment looks bad to me. They include plus-minus numbers. But plus-minus is driven by offence at least as much as defence. The sample calculation uses Paul Coffey as an example. It's clearly incorrect to give Coffey a high number of defensive points based on his scoring-driven plus-minus - yet that is what the sample calculation shows.

I think GVT has already done the one-number stat and done it better. The real issue with these one-number stats is that the available data is insufficient. Hockey stats just don't give enough information for these one-number stats. Even the formulations for recent seasons that have time on ice data fail to take into account quality of competition - which is among the biggest contributions that defencemen provide, and is important for forwards as well.

I noticed hockey-reference is using point shares now to sort the players on a team, instead of points. It's fine for that purpose. I wouldn't use it for anything more.

See Milan Hejduk, 2002-03.

 

[seventieslord]

I saw this yesterday too. My first thought was that it was a poor man's GVT.

 

[Sensfanman]

What I'd like to see is QualComp with the Weighted Average of GVT of team mates and Opposition, then you could compare apples to apples instead of relying on QualComp (and it's variants).

At the end of the day, I'd just like to see a simple "Value Add Play" and "Value Subtract Play" count as well as puck possession time and puck possession plays (pass attempts, etc). Most of the current RTSS stats are so inconsistent between teams that they mean nothing.


That being said, any easy to get, advanced look on players (Tom Awad's historic GVT data base or on behind the net for 2010-11 and 2009-10) is great IMO.

 

[seventieslord]

I actually think I like Corsi better than GVT. It probably does a better job of representing who causes their team to be more in charge of the play when on the ice.

It still has to be weighted with qualcomp and quality of teammates to really tell us something.

 

[TheDevilMadeMe]

I was just looking at this today. IMO it needs some work. GVT, by Tom Awad of Puck Prospectus, is comparable and better implemented.

I also noticed that there was no mention of an ES/PP/PK breakdown. That seems like something that is obviously wrong and should be fixed. GVT is a single number stat that handles this better.

Yeah, GVT is a very imperfect stat, but it seems like it basically already does what hockey reference is trying to do, but better.

The goals/assists thing is tricky. It's possible that there is some group of hockey players for whom goals should be weighted higher than assists. But when it comes to the stars, I think an assist is as good as a goal. I think Point Shares gets this wrong, and GVT does too.

Honestly, it seems like the goals/assists formula for goals created was made by a statistician who doesn't actually know much about the game itself. Technically, there are usually 2 assists per goal, so a goal is worth twice an assist, right? Makes sense mathematically. But I agree that it doesn't say much about the actual offensive performance of players - I think in an actual hockey play, the combined value of the two assists to creating the offense is usually more than the guy who just finished the play.

I think goals + primary assists + (secondary assists / 2) would be a better formula, though still imperfect.

The defensive adjustment looks bad to me. They include plus-minus numbers. But plus-minus is driven by offence at least as much as defence. The sample calculation uses Paul Coffey as an example. It's clearly incorrect to give Coffey a high number of defensive points based on his scoring-driven plus-minus - yet that is what the sample calculation shows.

Again, the simplistic use of plus/minus indicates that the creator of the formula isn't so familiar with what the formula is trying to measure.

I think GVT has already done the one-number stat and done it better. The real issue with these one-number stats is that the available data is insufficient. Hockey stats just don't give enough information for these one-number stats. Even the formulations for recent seasons that have time on ice data fail to take into account quality of competition - which is among the biggest contributions that defencemen provide, and is important for forwards as well.

Agreed. (So says the guy who constantly calls adjusted plus/minus, a better stat than most of the information used to created H-R's new stat, of very limited utility without quality of competition data).

I noticed hockey-reference is using point shares now to sort the players on a team, instead of points. It's fine for that purpose. I wouldn't use it for anything more.

I'd still rather use points. At least it means something.

 

[Roy S]

A main problem is there is so little useful data from years ago. Quality of competition, zone start %, ES/PP/PK time, and penalties drawn/taken all need to be considered and that isn't readily available for the majority of years. Since that is available for the last several years, then it would be easier to make one metric based off that, but even that will have imperfections.

At this point, I think looking at Corsi/Fenwick, the quality of competition metrics, zone start %, penalties drawn/taken, 5 on 5 +-, ESP per 60 minutes and PP/PK time will give a pretty good picture of a player's performance. That doesn't help much from a historical perspective and it isn't one single metric, but advanced hockey stats are still in its infancy so you have to take the good with the bad.

 

[Hockey Outsider]

It's an interesting attempt, but their calculation of defensive point shares ("DPS") is very flawed.

- As Reckoning mentioned, they don't differentiate between ES, PP and SH ice time, which is a fundamental error. A forward can earn a lot of DPS simply by racking up a lot of minutes on the power play (which the system reads as a larger contribution to a team's defense even though, in real life, they're making minimal defensive contributions). For example during the 2000 and 2001 seasons Pavel Bure was 3rd among forwards in DPS; I suspect this is mostly because as he was playing 5 minutes per game on the powerplay, which looks great according to the formula).

- DPS appears to be highly biased in favour of modern players. Eighty-six Of the top hundred DPS seasons from a forward are from the past 12 seasons! Apparently the first fifty-four years of hockey history apparently yielded just one top 100 DPS season from a forward.

- Defensive points is based on plus/minus, which I consider nearly useless as it doesn't take matchups or linemates into account. Also plus/minus, by its nature, is half determined by offense, so I'm not sure why they're treating it as a purely defensive adjustment.

The final results aren't terrible, though there's a lot of notable exceptions. The most ridiculous is Jagr ranking 2nd all-time in DPS among forwards (presumably because he played a lot on the powerplay and had a good plus/minus, mostly due to his offense).

====

Alan Ryder's "Point Allocation" system is the gold standard if you think you can summarize a player's contributions into a single, unified stat. Link. I don't pretend to understand the gory mathematical details but it's obvious that Ryder is a hockey fan who understands the game, and the principles he uses are much stronger than the ones on hockey-reference.com

EDIT: I see that Overpass has basically said the same thing in the ATD thread - great minds think alike, right?

 

[Hardyvan123]

It's an interesting attempt, but their calculation of defensive point shares ("DPS") is very flawed.

- As Reckoning mentioned, they don't differentiate between ES, PP and SH ice time, which is a fundamental error. A forward can earn a lot of DPS simply by racking up a lot of minutes on the power play (which the system reads as a larger contribution to a team's defense even though, in real life, they're making minimal defensive contributions). For example during the 2000 and 2001 seasons Pavel Bure was 3rd among forwards in DPS; I suspect this is mostly because as he was playing 5 minutes per game on the powerplay, which looks great according to the formula).

- DPS appears to be highly biased in favour of modern players. Eighty-six Of the top hundred DPS seasons from a forward are from the past 12 seasons! Apparently the first fifty-four years of hockey history apparently yielded just one top 100 DPS season from a forward.

- Defensive points is based on plus/minus, which I consider nearly useless as it doesn't take matchups or linemates into account. Also plus/minus, by its nature, is half determined by offense, so I'm not sure why they're treating it as a purely defensive adjustment.

The final results aren't terrible, though there's a lot of notable exceptions. The most ridiculous is Jagr ranking 2nd all-time in DPS among forwards (presumably because he played a lot on the powerplay and had a good plus/minus, mostly due to his offense).

====

Alan Ryder's "Point Allocation" system is the gold standard if you think you can summarize a player's contributions into a single, unified stat. Link. I don't pretend to understand the gory mathematical details but it's obvious that Ryder is a hockey fan who understands the game, and the principles he uses are much stronger than the ones on hockey-reference.com

EDIT: I see that Overpass has basically said the same thing in the ATD thread - great minds think alike, right?

Thanks HO for the link. I was going to tackle my taxes today but now I have different plans..lol

 

[Hockey Outsider]

Thanks HO for the link. I was going to tackle my taxes today but now I have different plans..lol

I did mine yesterday... and in some ways reading that link is about as technical and detail-oriented as doing your taxes... though at least you won't have to write a cheque to the government after you're done reading Ryder's article.

 

[TheDevilMadeMe]

Has H-R fixed any of the obvious flaws in their formula?

 

[overpass]

I came across this discussion in another thread and thought I'd post on it here.

Do you know what goes into the formula? Among many issues (documented here), it treats PP/ES/SH time the same (probably because the guys at hockey-reference are employees of sports reference and not actually hockey fan). In other words, playing 2 minutes of PP time, 2 minutes of ES time, and 2 minutes of SH time without getting scored on counts exactly the same for a player's "defensive point shares."

The formula is a steaming hot pile of garbage. Phil Housley of all players finishes 22nd all-time in "defensive point shares," ahead of Rod Langway: http://www.hockey-reference.com/leaders/dps_career.html. Probably mainly because he played a lot on the powerplay (where teams are rarely scored on) and almost never on the penalty kill (so he didn't have shorthanded goals against).

If you want to use an "all in one" formula to evaluate hockey players, at least use one created by someone who understands the sport enough to know what a power play is. Something like points allocation or goals versus threshold. Those aren't perfect, but at least they take actual hockey variables into account.

I don't think that's quite accurate. The formula does treat all minutes played equally for the purposes of defensive point shares. But it doesn't penalize players directly for goals against. Instead, it:

1. Divides team defensive value among players in proportion to their overall ice time (with each defender receiving a x2 position bonus compared to each forward.

2. Applies an adjustment based on plus-minus. Which does not penalize players for goals against while shorthanded, but does penalize them for goals against at even strength or while on the power play.

------------------------------------------------------------------

It's very interesting to compare the Point Shares of Phil Housley and Mike Ramsey when they played together in Buffalo. Buffalo was usually one of the better defensive teams in the league, so there was a lot of defensive value to spread around. So Housley got credit for playing a lot of games at defense on a strong defensive team.

From 1982-83 to 1989-90, Ramsey was credited with 37.7 DPS in 576 GP and Housley was credited with 33.4 DPS in 608 GP. Ramsey was assigned more value because of the plus-minus adjustment - he was +112 over this time and Housley was only +3.

What's wrong with these numbers for Housley and Ramsey?

1. The starting assumption that Housley and Ramsey were equally responsible for team defensive success per-game is absurd to anyone who is familiar with the two players.

Housley had entered the NHL directly out of high school in 1982-83, and this time period covered his age 18-26 years - a time when defenders are still developing, especially defensively. He played parts of some of these seasons at centre, in part because of his defensive weaknesses. He was widely considered to be one of the softest players and weakest defensive players in the league.

Ramsey was in his prime years - age 22-30. He was widely considered to be among the best defensive defencemen in the league. In 1984 a poll of NHL coaches had Ramsey as a runner-up for best defensive defenceman behind Rod Langway, and in 1990 a poll of NHL players named him as the best defensive defenceman in the league.

Penalty killing is just a part of defensive value. But, since it's a pure defensive situation, we can look at deployment in these situations for an idea of how players' defensive abilities were viewed by their coaches. Among all NHL defencemen with 500+ games played, here are the top 5 and bottom 5 players in SH% (the percentage of their teams PPGA they were on the ice for on a per-game basis.)

Rank | Player | GP | SH%
1 | Rod Langway | 585 | 59%
2 | Mike Ramsey | 576 | 59%
3 | Craig Ludwig | 597 | 58%
4 | Joel Quenneville | 531 | 57%
5 | Tom Laidlaw | 546 | 56%
6 | Bob McGill | 434 | 21%
7 | Ron Greschner | 438 | 20%
8 | Jim Benning | 531 | 12%
9 | Phil Housley | 608 | 11%
10 | Petr Svoboda | 416 | 9%

Langway and Ramsey played huge defensive roles on strong defensive teams and might have been considered the top two defensive defencemen in the league during this time period. At the bottom end, only two other defencemen killed penalties as infrequently as Housley. All evidence suggests that Ramsey was considered among the very best players defensively, and Housley was considered among the very worst.

2. Buffalo had a strong penalty killing record for most of this time, with league finishes in PK% of 1, 2, 2, 3, 6, 12, 15, 17. As seen above, Ramsey had a huge part in that record. Housley had almost no part. The only credit either received were in the plus-minus adjustment for the SH goals they were on the ice for - no credit for goal prevention on the PK. I guess you could argue that the SHG / plus-minus credit is enough, but it's an extremely indirect and inaccurate way of giving credit.

3. Ramsey did receive more value according to defensive point shares, so what's the problem? The question is if the gap is as big as it should be.

Ramsey's advantage in DPS is entirely due to his advantage in plus-minus. Ramsey was +112 and Housley was +3 over the same period. Some of that difference is because of shorthanded goals for and against. I estimate that if we back those out, the difference shrinks to +91 to +58 at ES only. So Ramsey did get some advantage from his special teams deployment in plus-minus (and therefore in DPS). Probably about 3 DPS of his 37.4.

BUT Housley scored 305 ES points over this time period. Ramsey scored only 164 ES points. Housley received a ton of credit for these points already in Offensive Point Shares. He had 40.8 OPS from 1982-83 to 1989-90, and Ramsey only had 8.4 OPS. That's not unfair on the surface - after all, Housley was extremely skilled and was considered to be among the best offensive defencemen in the league.

If these points scored by Housley were really valuable in contributing to his team's success, they should have appeared in his team's goal differential. Let's say they did - that he contributed to 141 extra GF than Ramsey did over this time. But Ramsey's ES goal differential was 23 goals better than Housley's overall. If we give credit to Housley for helping to score an extra 141 points, why not subtract credit for an extra 164 goals against as well?

(In fact Housley and Ramsey had relatively close ESGF and ESGA numbers overall during this time - 618 ESGF and 560 ESGA in 608 GP for Housley, and 603 ESGF and 512 ESGA in 576 GP for Ramsey. Either Housley's 141 extra points at ES massively overstates the actual gap in offensive value - probably because Housley's points were simply replaced by the forwards while Ramsey was on the ice - or Ramsey played WAY more ice time at even strength.)

I think you can make a pretty good case that Ramsey was the more valuable player than Housley during their years together in Buffalo. He was an elite defensive defenceman with some offensive skills (i.e. he wasn't a Craig Ludwig or a Ken Daneyko) who played big minutes on a team with an excellent defensive record at ES and on the PK. Housley had a worse overall plus-minus record despite scoring many more ES points, was shifted to forward at times, was considered among the worst defensive defencemen in the league, and scored a lot of points playing big minutes on a power play that wasn't very successful (league PP% finishes of 8, 9, 11, 13, 16, 16, 17, 18 in a 21 team league).

But Point Shares has Housley as over 50% better over this time period, with 74.6 point shares compared to 46.1 point shares for Ramsey. Because Point Shares gives full credit to players for the things Housley was better at (scoring points) and averages out everything Ramsey was better at (everything else).

Edit: I should add that during the 1982-83 to 1989-90 period, both Ramsey and Housley played in 2 Canada Cups, Ramsey and Housley both played in 3 All-Star games, and Ramsey was selected to the NHL team for Rendezvous 87 and Housley was not. So Ramsey was certainly not considered a lesser player at the time.

 

[overpass]

If anyone's interested, it may also be worth pointing the people at H-R in the direction of some of the criticism here. It looks like S-R has someone else running the site now.

 

[Mickey Marner]

Since the 2005-06 season, players have been awarded shares of points earned in the shootout regardless of whether they actually participated in any shootouts... the stupidity of this statistic knows no bounds.

 

[Sens Rule]

Defensive point shared for the "Pizza line". Possibly the best line in NHL over several years.

05/06
Heatley 2.7
Spezza 2.1
Alfredsson 2.6

06/07
Heatley 2.6
Spezza 1.8
Alfredsson 3.0

07/08
Heatley 2.5
Spezza 2.2
Alfredsson 1.8

08/09
Heatley 1.8
Spezza 1.6
Alfredsson 2.5

Total in 4 years:
Heatley 9.6
Spezza 7.7
Alfie. 9.9

This shows the absolute uselessness of DPS. Alfredsson mostly played with Spezza and Heatley for 4 seasons. Alfredsson also played the PK far, far more then Spezza and Heatley. And he was a GREAT defensive player all of those years. A near Selke caliber winger. He was the conscience of the line defensively, and was frankly relentless in trying to challenge the puck owners of opposite teams. He was among the top few forwards all 4 seadons defensively. He has Selke votes each year to prove it.

Heatley and Spezza were average, or even below average forwards defensively. Heatley had moments in 05/06 and 06/07 when he played strong overall, but he also coukd float and later was not good defensively. Spezza just is not great defensively, even if when he tries, he is just average, or is a bit above.

Yet Alfie is 2nd once and 3rd once in 4 years on a line where he is absolutely elite defensively, and he covers for his two linemates? Where even when he is ranked first among his line, it is close? I watched almost all those games. It wasn't close. Alfredsson was a force defensively. He was absolutely relentless, game in and game out, and effective. Plus Alfredsson was used on the PK a lot. And he wad one of the best in the NHL there. Plus Ottawa was a great team 3/4 years. They greatly outscored opposition, so it can't be because Ottawa wasn't good that the DPS got warped.

Heatley almost having an equal DPS over 4 years as a Sen to Alfie is frankly laughable. It is not even remotely close to reality. It shows how a linemate carries another in that stat. And how a top PKer is given virtually no credit over a linemate that rarely is on the PK either.

It nearly completely invalidates the stat as a measure of defensive utility. It shows it is all about goals for and against and measures very little defensive skills.

Offensively the pizza line was basically a wash. All 3 were great and averaged about 90 points+ per 80 games probably over 4 years. Defensively... Alfredsson is in a totally other dimension then Heatley and Spezza... Yet DPS measures them basically a wash. Makes no sense. Unless DPS does not measure defensive performance at all.

 

[LeBlondeDemon10]

I think goals + primary assists + (secondary assists / 2) would be a better formula, though still imperfect.

I'd still rather use points. At least it means something.

I don't like the secondary assists argument. The secondary assist could be the player that created the play, then there's a shot, a save, a rebound and the third player taps in the goal. I agree that points carry much more weight.

 

[plusandminus]

Common sense suggests defensive pointshares are rather useless, and the examples of Housley and Alfredsson "proves" it.

To determine how much a player has contributed offensively, the best "easy" method is to just look at how many points he scored overall.

+/- would be a better stat if the GA could be adjusted based on goaltending. That is problematic too, as for example save percentage seem to be influenced by how good or bad defence the skaters' play (making opponents shoot from bad angles or distances, preventing opponents to score on rebounds, etc).

 

[Hockey Outsider]

I sent an email to the people at Hockey-Reference.com around two months ago, outlining some of these concerns. They've acknowledged receipt. I'll let everyone know if they ever provide a response.

 

[Hockey Outsider]

I'm moving this discussion into this thread so we have all of our feedback on Point Shares in one place:

Here is a link to an article that references it.

These NHL Players Deserve A Raise

Winning the Stanley Cup is about grit, toughness and the will to overcome adversity. But in the NHL’s salary-cap era, it’s also about finding the right mix of c…

fivethirtyeight.com

And here is the brief explanation of modified point shares in a FN in that article, along with a link to Paine's data page:

[Modified Point Shares is] My spin on Hockey-Reference.com’s Point Shares stat, which better balances value between positions by ensuring that forwards get 60 percent of leaguewide value, while defensemen get 30 percent and goalies get 10 percent. (You can find historical MPS data here.)

The link to the historical MPS data opens as unformatted gibberish for me, but it may just be me.

Thanks for the link. I agree - among other issues, Point Shares seems to assign way too much value to goaltending. The thread on Tangotiger's blog (which you linked) is a really good overview of the issue.

Or you can look at some of the clearly unreasonable conclusions:
- Roberto Luongo > Gordie Howe
- Hendrik Lundqvist > Mario Lemieux and Sidney Crosby
- Curtis Joseph > Steve Yzerman
- MA Fleury > Mark Messier, Denis Potvin and Phil Esposito
- Ryan Miller > Bobby Orr
- Sean Burke and Tomas Vokun > Jean Beliveau and Bobby Hull
- Jose Theodore > Mike Bossy
- Olaf Kolzig > Bryan Trottier
- Felix Potvin > Bobbye Clarke
- Don Beaupre > Howie Morenz and Peter Forsberg
- Ron Hetxall > Pavel Bure, Peter Stastny, and Andy Bathgate (I can keep going...)

Tangotiger's conclusion is, among other things, the replacement level is set way too low, which means goalies get a ton of credit for just showing up. This skews the positional ranking of goalies because those with short careers are heavily penalized (and/or those with long careers are rewarded just for being a below-average starter or even a mediocre backup). Any ranking that has Roberto Luongo, Curtis Joseph and MA Fleury ahead of Dominik Hasek (or Don Beaupre and Dwayne Rolson ahead of Ken Dryden) is fatally flawed.

The adjusted point shares from 538 seems to be better in the sense that goalies don't receive such a huge proportion of the Point Shares. I can't download the data set either. But it doesn't sound like they fundamentally change how Defense Point Shares are calculated. Those are flawed (practically to the point of being worthless - will do a separate post about that soon), so even if they adjust the amount allocated to goalies vs skaters, I still think the end result has limited value.

 

[Hockey Outsider]

This turned into a longer and more technical discussion than planned, but I'll post it anyway:

I took another look the 2005-06 season to see if DPS (defensive point shares) makes sense. Some comments:

Defensemen completely dominate the DPS stat, with 99 of the top 100 players. Obviously we'd expect more defensemen than forwards among the leaders in DPS, but 99 of the top 100 players seems extreme. (This implies that a below-average #3 defenseman contributes more defensively than a Selke trophy finalist - that doesn't make sense).

DPS appears to be re-purposed plus/minus (the flaws in that stat are well-documented). The top five forwards in DPS are Jaromir Jagr, Joe Thornton, Brendan Morrow, Michael Nylander and Brian Rolston. I don't think it's a coincidence that four of them were +30 or higher. There were only 34 players with a negative DPS; every one of them finished with a negative plus/minus rating (meaning there's too much stock being placed into an obviously flawed input).

Numerous players in the top 100 in DPS received little ice time on the penalty kill (including Dan Boyle, Joe Corvo, Jaromir Jagr, Ron Hainsey, Nolan Baumgartner, John-Michael Liles, Kurtis Foster, Brian Pothier, Christian Ehrhoff, and Marek Zidlicky). Maybe you can argue that some of them (like Jagr) were so dominant in terms of puck possession that they delivered good defensive value even without being on the PK - but that hardly describes most of these players.

The top seven players for the Selke were Rod Brind'Amour, Jere Lehtinen, Mike Fisher, Daniel Alfredsson, John Madden, Kris Draper, and Stephane Yelle (I'm listing the top seven because there's a big drop-off in the vote totals after that). That's a pretty good list of who the best defensive forwards were at the time. These seven players rank T-22nd, T-22nd, T-38th, T-9th, T-62nd, T-83rd and T-52nd in DPS. I'm not saying there should be a perfect correlation (one of the purposes of these stats is to shed light on things we may have missed). But it seems strange that only one of these players are in the top twenty, but Michael Nylander, Jonathan Cheechoo, Teemu Selanne, Dany Heatley, Martin Straka, and Andy McDonald are.

Here's a deep dive. On the LA Kings, Lubomir Visnovsky has 4.1 DPS and Mattias Norstrom has 2.9 DPS. Visnovsky was a high-risk, high-reward offensive defenseman who scored 67 points and spent more than 7 minutes per game on the powerplay(!), and less than 90 seconds on the PK. Norstrom was a disciplined, responsible shutdown defenseman with minimal offensive talent (this was his career high with 27 points); he spent close to 5 minutes per game on the PK (his 362 PK minutes rank 17th in the league). I can't blame DPS for not "knowing" that Norstrom consistently took tougher defensive matchups. But the system is based largely on total ice time and plus/minus. Visnovsky played about 15% more minutes than Norstrom - but since the system reads all ice time as equal, Visnovsky is getting defensive credit for playing a ton on the PP, and Norstrom loses ground despite being one of the most heavily-utilized penalty killers in the entire NHL. Visnovsky has a higher plus/minus (+7 vs -3). But that's because he was on the ice for far more goals for at ES (69 vs 42). That makes sense, Visnovsky was much better offensively - but he already gets credit for that in OPS (where he's up 6.0 to 1.1). Norstrom was on the ice for only 45 ES goals against compared to 62 for Visnovsky. Taking ice time into account, that's a GAA of 3.14 for Visnovsky and 2.62 for Norstrom.

Essentially, despite Norstrom 1) taking tougher defensive matchups 2) playing far more on the PK and 3) being on the ice for much fewer goals against, Visnovsky ranks well ahead in DPS (because he got a lot of PP ice time and because his plus/minus reflects his offensive production, which is already counted elsewhere). If the system ranked Visnovsky ahead in total point shares (because his offense may have contributed more than Norstrom's defense) - fine, that's not unreasonable. But Visnovsky ranking ahead DPS is absolutely nonsense. We shouldn't use one example to invalidate the whole method, but I wanted to do a deep dive to show how the results just don't reflect what happens on the ice.

I added up the DPS by team and calculated the correlation between Team DPS and team goals against. There's virtually a perfect correlation (R^2 = -0.98). That means the DPS assigned to each team is almost entirely a function of the number of goals the team allows. The problem is this doesn't take into account the relative contributions of the skaters vs the goalies. For example, look at Edmonton. They allowed the fewest shots in the entire league - but also had the worst save percentage from their goalies. That means their skaters did their jobs, and their goalies didn't. If we use shots as an (admittedly imperfect) proxy for defensive performance at the team level, Edmonton should rank first. Instead, based on DPS, they're middle of the pack. Pronger only got 5.6 DPS because he played in front of bad goaltending. My rough calculation suggests that if the Oilers skaters got proper credit for their defense, Pronger would have ranked 1st in the league in DPS (which doesn't seem unreasonable - certainly more supportable than Michael Rozsival). Similarly, the Panthers allowed the 2nd most shots in the league but were bailed out by Luongo. Florida is a bit below average in Team DPS, but they really should be near the bottom. Too much credit went to skaters like Mike Van Ryn (who, as a result, finished within 0.3 DPS (less than 10%) of much stronger defensive players like Chris Chelios and Zdeno Chara).

My conclusion? Defensive Point Shares is virtually meaningless. The more I look at it, the more its flaws become obvious. Any purely stat-based system will have some issues, but these errors are so pervasive that I don't think the output has any value whatsoever.

 

[The Panther]

The other problem with the point-share idea is that it obviously "punishes" great players on great teams, because there are a finite number of points to be shared. I mean, of course the Orr's and Gretzky's of history are still going to do well, but it doesn't make any sense to me that if Orr 1971 or Gretzky 1984 were moved to Philadelphia 1971 and Winnipeg 1984 that they would suddenly have big spikes in their seasonal point shares just because their teams were crappier.

We shouldn't be ranking players higher because they were better than their below-average teammates.

 

[Bear of Bad News]

The other problem with the point-share idea is that it obviously "punishes" great players on great teams, because there are a finite number of points to be shared.

Bill James studied this in baseball extensively and the conclusion was that players on great teams weren't punished with a Win Shares system (nor were players on poor teams punished).

Hockey isn't baseball, but I'll submit that as evidence that if it does punish hockey players on great teams, it's surely not "obvious".

 

[Hockey Outsider]

Not to keep attacking Point Shares, but I noticed another design flaw that I don't think anyone has identified previously.

Offensive Point Shares (from 1999 onwards) takes a player's ice time into account. Effectively, the more ice time a player receives, the more his OPS gets reduced. (In other words, there's a "tax" on ice time, and only a player who produces in excess of the threshold rate is adding value).

We can debate if that makes sense in theory. But the way it's implemented is completely flawed. The reason is, the formula doesn't distinguish between different types of ice time.

For example, look at the 2002 Islanders. Mike Peca (who won the Selke trophy that year) and Mark Parrish (who was skilled, but also one-dimensional) both scored 60 points. Parrish had five more goals and Peca had five more assists. Goals created, which is used rather than points in the calculation of OPS, weighs goals more heavily. So Parrish gets credit for 24 GC, and Peca gets credit for 23 (a difference of about 4%).

Parrish gets credit for 6.1 OPS, and Peca gets 4.8. Parrish's virtually equal production is now worth 27% more than Peca's. Peca received significantly more ice time than Parrish, so he gets penalized in their formula. The problem is, it doesn't consider the type of ice time. Peca played 38 seconds more per game at ES, and Parrish played 20 seconds more per game on the powerplay. Given that scoring rates are much higher on the PP, we'd expect their production (based solely on their ES & PP ice time) to be nearly even. (Actually, we'd expect Parrish's production to be higher; it wasn't tracked by any statistics at the time, but we can be reasonably sure that Peca got more starts in the defensive zone, and spent way more time defending at ES. But we can't fault a simple statistical system for not "knowing" this).

Here's the problem. Virtually the full difference in their TOI (3:26) is due to shorthanded TOI (3:07). Generally, players aren't expected to produce any offense while on the PK (some do, of course, and that's a nice bonus, but it generally isn't the expectation). Parrish got 27% more OPS than Peca. Of that difference, 4% is due to the system weighing goals more than assists (debatable, but let's grant it), and 23% is due to the disparity in ice time (which is nonsensical, because the difference is almost entirely due to the penalty kill, where we don't expect players to produce anywhere near the same level of offense).

There are many other examples. Look at the 2000 Devils. Scott Stevens and Brian Rafalski produced the same level of offense, as measured by the system (10 goals created). But Rafalski had 28% more OPS (2.3 vs 1.8) because Stevens had 29% more ice time (the difference is presumably due to rounding). Stevens got an extra 2:11 TOI per game at ES, but Rafalski got an extra 1:30 per game on the PP (roughly similar). Stevens played a massive 4:07 per game on the penalty kill (top ten in the league), while Rafalski played only 16 seconds. The formula is quite obviously penalizing Stevens' offense because he was so good defensively (and therefore played extensively on the PK).

The reason I'm posting this is because, although I think it's well-established that defensive point shares are meaningless, I used to think that at least OPS had some value. Now that I've noticed this, I think OPS can (in certain situations) be misleading, as it clearly penalizes players who play on the penalty kill. Unless you believe that players should produce offense at the same rate (regardless of whether it's ES, PP or PK time), this is a fundamental error in how Offensive Point Shares are calculated.