Non-traditional metrics: PDO

[Czech Your Math]

What is IPP?

I wouldn't call PDO an "advanced statistic", I'd call it a new statistic that is very simple and very misleading. It has two components, one of which the individual has much control over and may indicate his skill to a large degree, the other of which he has very little control over (except to prevent high quality shots) and therefore has little to do with his skill. It's like putting sauerkraut and Oreos together and calling it an advanced food.

Look at the shooting %s of Brett Hull or Stamkos. They jumped in their second seasons, but that doesn't mean they didn't improve further a season or two later. Of course such levels are not sustainable for most players over longer periods, but it doesn't mean it was mostly luck.

 

[Czech Your Math]

No, it would not. One is a measurement of efficiency, the other of result. GF is the result of SF * S%, and GA is the result of (1 - Sv%) * SA.

As many people have stated many times in many ways using strategies firmly rooted in sound statistical reasoning, shot volume (SF/SA) is something that good teams tend to be good at.

The efficiency percentages over the long term for ALL teams tends to regress back to league averages. In other words, outliers of the efficiency percentages are not generated by the teams ability since they will always tend to head back to league averages.

So again, no GD, GF, and GA are not the same pieces of information. PDO is a lot more useful when looking at unsustainable performances.

I still don't see the difference. If a team is scoring 4 GPG in today's NHL, or giving up 1.5 GPG, it will also likely regress to the mean. However, if the 80s Oilers were scoring 1 GPG more than the league, I wouldn't hold your breath waiting for it to "regress to the mean." It's being proposed that PDO should be 1000 for all teams and individuals, but that's not even close to being true.

I gave examples of teams whose overall S% +SV% generally stayed above or below the mean over multiple seasons:

Islanders were 985, 986, 989, 998, and 984 the last 5 seasons.
Columbus was 990, 999, 994, 984, and 984 the last 5 seasons.
Toronto was 986, 981, 975, 997, and 998 the last 5 seasons.

These teams weren't unlucky, they were just below average. I doubt Toronto was "less unlucky", but instead was "less bad."

When Boston won the Cup two years ago, they were 1023 during the season. They improved to 1042 in the playoffs. How is that regressing to the mean? Yes, teams will regress to the mean over multiple seasons, because there's a lot of parity and teams change from year to year. They add players, they lose players, players get better, players get injured.

I gave examples of teams whose overall S% + SV% generally stayed above or below the mean over multiple seasons:

Vancouver was 1006, 1018, 1019, 1026 and 1019 the last 5 seasons.
Boston was 1004, 1036, 998, 1023 and 1019 the last 5 seasons.

Those teams weren't lucky, they were good, especially their goalies.

How is it primarily luck-driven, when the best teams tend to be higher and the bad teams tend to be lower over multiple seasons? The bad teams provide better examples in general, because it's a lot easier to be bad than good. Somehow, certain teams tend to remain lucky:

Oilers
'84 1054
'85 1042
'86 1050
'87 1039
'88 1035

I know some people that will be glad to know that Gretzky was simply lucky and not great.

 

[mindmasher]

What is IPP?

I wouldn't call PDO an "advanced statistic", I'd call it a new statistic that is very simple and very misleading. It has two components, one of which the individual has much control over and may indicate his skill to a large degree, the other of which he has very little control over (except to prevent high quality shots) and therefore has little to do with his skill. It's like putting sauerkraut and Oreos together and calling it an advanced food.

Look at the shooting %s of Brett Hull or Stamkos. They jumped in their second seasons, but that doesn't mean they didn't improve further a season or two later. Of course such levels are not sustainable for most players over longer periods, but it doesn't mean it was mostly luck.

IPP is individual point percentage, is a calculation of the number of times an individual player gets a point (either a goal or an assist) compared to the number of total goals scored while he's on the ice. Very high values of this measure tend to regress back to league averages.

Arguing over the usage of 'advanced' is useless. It is pretty simple, but it is certainly advanced enough for the vast majority of fans that I don't see issue with the use of the word in regards to PDO. Either way, who cares, how we categorize it complexity wise means nothing to the overall debate of usability.

PDO of course is on-ice team shooting percentage and team save percentage, and this is much different than individual shooting percentage, which you keep on bringing up for some reason.

 

[mindmasher]

I still don't see the difference. If a team is scoring 4 GPG in today's NHL, or giving up 1.5 GPG, it will also likely regress to the mean. However, if the 80s Oilers were scoring 1 GPG more than the league, I wouldn't hold your breath waiting for it to "regress to the mean." It's being proposed that PDO should be 1000 for all teams and individuals, but that's not even close to being true.

I gave examples of teams whose overall S% +SV% generally stayed above or below the mean over multiple seasons:

Islanders were 985, 986, 989, 998, and 984 the last 5 seasons.
Columbus was 990, 999, 994, 984, and 984 the last 5 seasons.
Toronto was 986, 981, 975, 997, and 998 the last 5 seasons.

These teams weren't unlucky, they were just below average. I doubt Toronto was "less unlucky", but instead was "less bad."

When Boston won the Cup two years ago, they were 1023 during the season. They improved to 1042 in the playoffs. How is that regressing to the mean? Yes, teams will regress to the mean over multiple seasons, because there's a lot of parity and teams change from year to year. They add players, they lose players, players get better, players get injured.

I gave examples of teams whose overall S% + SV% generally stayed above or below the mean over multiple seasons:

Vancouver was 1006, 1018, 1019, 1026 and 1019 the last 5 seasons.
Boston was 1004, 1036, 998, 1023 and 1019 the last 5 seasons.

Those teams weren't lucky, they were good, especially their goalies.

How is it primarily luck-driven, when the best teams tend to be higher and the bad teams tend to be lower over multiple seasons? The bad teams provide better examples in general, because it's a lot easier to be bad than good. Somehow, certain teams tend to remain lucky:

Oilers
'84 1054
'85 1042
'86 1050
'87 1039
'88 1035

I know some people that will be glad to know that Gretzky was simply lucky and not great.

Sigh. You take a team measurement and then use it to logically conclude that Gretzky is lucky? Do you see the logical fallacy there?

The teams you picked have some of the best shot differentials over the past 5 seasons - a much better indicator of success - and also feature two of the most extreme examples of shooting percentage and save percentage outliers respectively: the Sedins and Tim Thomas.

Why don't you take a look at all 30 teams at once, like a true statistical analysis would. Maybe you would be surprised to find that on average teams regress to 1000.

 

[Czech Your Math]

IPP is individual point percentage, is a calculation of the number of times an individual player gets a point (either a goal or an assist) compared to the number of total goals scored while he's on the ice. Very high values of this measure tend to regress back to league averages.

Thanks for the definition. An extreme value for just about any metric would be expected to regress to the mean.

Arguing over the usage of 'advanced' is useless. It is pretty simple, but it is certainly advanced enough for the vast majority of fans that I don't see issue with the use of the word in regards to PDO. Either way, who cares, how we categorize it complexity wise means nothing to the overall debate of usability.

Agreed, but when people start giving metrics fancy names and calling them advanced, some start believing they are more useful than may actually be.

PDO of course is on-ice team shooting percentage and team save percentage, and this is much different than individual shooting percentage, which you keep on bringing up for some reason.

Somehow I missed that it was on-ice team S%, sorry for the mistake. However, team shooting %s don't necessarily regress to the mean, especially certain lines. Do you think the team S% while Gretzky-Kurri & Coffey were on the ice regressed to any sort of mean (league avg.)?

Sigh. You take a team measurement and then use it to logically conclude that Gretzky is lucky? Do you see the logical fallacy there?

The Oilers/Gretzky. It doesn't really matter, they weren't lucky, they were good. Where's the regression to the mean? Average teams are going to regress to the mean, good and bad ones aren't. How do we know which is which, simply by looking at this metric?

The teams you picked have some of the best shot differentials over the past 5 seasons - a much better indicator of success - and also feature two of the most extreme examples of shooting percentage and save percentage outliers respectively: the Sedins and Tim Thomas.

Why don't you take a look at all 30 teams at once, like a true statistical analysis would. Maybe you would be surprised to find that on average teams regress to 1000.

I did look at all 30 teams over the last 4 years in terms of overall SV% + S%, but I don't even know where to find 5v5 team S% and SV%. I've already given several examples of teams that stayed above or below the mean for each of multiple seasons. How does this agree with what is supposed to happen? So the handful of consistently good and the handful of consistently bad teams are "outliers", but the fact that a bunch of near-average teams will tend to have near-average data over larger samples is some sort of revelation?

The graph in the OP's line shows it regressing to the mean and then regressing back away from the mean. Is that what one would expect? Why do I need to disprove what isn't even proven by the metric's proponents?

I know conceptually that this metric is highly flawed. If others find it to be useful, that's fine.

However, if it's truly driven by randomness, wouldn't one expect the results to be unpredictable from one season to another for all teams, not just the average ones?

I see I missed an example on the linked site: 2011 Sharks. It says the Sharks were unlucky through Jan. 13 and post graphs to show how that was so. Interesting that Nittymaki (.896 overall SV%) played in 22/45 games through Jan. 13 and 2/37 after that, while Niemi (.920 overall SV%) took over in full after that. Whether Nittymaki was injured or the Sharks decided he was "unlucky", apparently this has no bearing on the matter? Good grief, if I can refute the best examples without even trying hard, how does one really take this seriously?

 

[TheDevilMadeMe]

Sigh. You take a team measurement and then use it to logically conclude that Gretzky is lucky? Do you see the logical fallacy there?

The teams you picked have some of the best shot differentials over the past 5 seasons - a much better indicator of success - and also feature two of the most extreme examples of shooting percentage and save percentage outliers respectively: the Sedins and Tim Thomas.

Why don't you take a look at all 30 teams at once, like a true statistical analysis would. Maybe you would be surprised to find that on average teams regress to 1000.

Well if good teams tend to have consistently have high PDOs and bad ones tend to have consistently low ones, then I don't think it should be very revealing that "on average," they regress to the mean

Edit: if CYM's numbers are correct, that would seem to be what they are showing

 

[seventieslord]

But who relies on simple plus-minus, esp. over a full season or less, as a supreme measure of value? At most, one might use simple +/- as a supplemental stat over a full season or more. Anyone serious would use adjusted plus-minus, preferably over multiple seasons, as one of the metrics of value. On a team level, I would use GF/GA ratio or differential. One doesn't need PDO to know that a large change will usually regress towards the previous level the following year, Bill James proved that decades ago for baseball.

GF/GA is the goal in the end. You want to score more and be scored on less. But, they are a pretty small sample to draw conclusions from. Thousands of shots are better than a couple hundred goals.

I think there's too much put into stats like Corsi too. It's at least measuring something, but outshooting the opposition does not equal success. Outshoot Hasek, all you want, it might not help one bit. I think Corsi might tell you something about possession, but much less about overall effectiveness.

No, CORSI does tell you a lot about possession.

Also, I am not arguing that PDO is any kind of catch all or super stat. It is pretty much an auxiliary or supplementary tool to stats that actually mean something.

Boston was outshot two years ago, but they won the Cup. I think they'll take the goals and leave the shots for their opponents.

If that is the case, then they were “lucky” in terms of goaltending, or there was something strategic about it (i.e. take only high% shots and allow only low% shots). It doesn’t change the fact that taking even more of those high% shots is a good idea, and allowing even less of those low% shots is a good idea. In Boston’s case I would be more interested in which players were getting outshot less (i.e. relative Corsi) and who they were up against in doing so (Corsi Rel QoC)

I still don't see the difference. If a team is scoring 4 GPG in today's NHL, or giving up 1.5 GPG, it will also likely regress to the mean. However, if the 80s Oilers were scoring 1 GPG more than the league, I wouldn't hold your breath waiting for it to "regress to the mean." It's being proposed that PDO should be 1000 for all teams and individuals, but that's not even close to being true.

I gave examples of teams whose overall S% +SV% generally stayed above or below the mean over multiple seasons:

Islanders were 985, 986, 989, 998, and 984 the last 5 seasons.
Columbus was 990, 999, 994, 984, and 984 the last 5 seasons.
Toronto was 986, 981, 975, 997, and 998 the last 5 seasons.

These teams weren't unlucky, they were just below average. I doubt Toronto was "less unlucky", but instead was "less bad."

When Boston won the Cup two years ago, they were 1023 during the season. They improved to 1042 in the playoffs. How is that regressing to the mean? Yes, teams will regress to the mean over multiple seasons, because there's a lot of parity and teams change from year to year. They add players, they lose players, players get better, players get injured.

I gave examples of teams whose overall S% + SV% generally stayed above or below the mean over multiple seasons:

Vancouver was 1006, 1018, 1019, 1026 and 1019 the last 5 seasons.
Boston was 1004, 1036, 998, 1023 and 1019 the last 5 seasons.

Those teams weren't lucky, they were good, especially their goalies.

How is it primarily luck-driven, when the best teams tend to be higher and the bad teams tend to be lower over multiple seasons? The bad teams provide better examples in general, because it's a lot easier to be bad than good. Somehow, certain teams tend to remain lucky:

Oilers
'84 1054
'85 1042
'86 1050
'87 1039
'88 1035

I know some people that will be glad to know that Gretzky was simply lucky and not great.

Teams themselves aren’t necessarily lucky. Players can be situationally lucky though. It is important to know these things when looking at a player’s GF and GA. PDO is much more useful for comparing players on the same team, IMO. I have read many good examples of this. I think it was about Koivu’s 2010 season where he was excellent but had his numbers destroyed by bad goaltending – significantly worse than what the rest of the team experienced. These kinds of things do happen within a team over the small sample size of a season. These are things PDO can catch.

However, if it's truly driven by randomness, wouldn't one expect the results to be unpredictable from one season to another for all teams, not just the average ones?

It is random, but not necessarily in the way you’re looking at it.

Take some player from some team. Last year he was in line with the rest of his team from a Corsi standpoint, but for whatever reason his goalie stunk at even strength when he was on the ice, posting a .882 sv% when he was otherwise .920. he was situationally unlucky, and this badly damaged his +/-. It is predictable that this will likely regress to the mean for this player. It’s not predictable that he will continue to have bad puck luck.

I see I missed an example on the linked site: 2011 Sharks. It says the Sharks were unlucky through Jan. 13 and post graphs to show how that was so. Interesting that Nittymaki (.896 overall SV%) played in 22/45 games through Jan. 13 and 2/37 after that, while Niemi (.920 overall SV%) took over in full after that. Whether Nittymaki was injured or the Sharks decided he was "unlucky", apparently this has no bearing on the matter? Good grief, if I can refute the best examples without even trying hard, how does one really take this seriously?

No, the players “doing” these events aren’t lucky or unlucky, these things are based largely on their own skill. It’s the effect on the other players on the ice that is more random and luck-based.

I think that you think this stat is trying to be something it’s not. I hate that I’m not good at explaining stuff like this.

 

[mindmasher]

I see I missed an example on the linked site: 2011 Sharks. It says the Sharks were unlucky through Jan. 13 and post graphs to show how that was so. Interesting that Nittymaki (.896 overall SV%) played in 22/45 games through Jan. 13 and 2/37 after that, while Niemi (.920 overall SV%) took over in full after that. Whether Nittymaki was injured or the Sharks decided he was "unlucky", apparently this has no bearing on the matter? Good grief, if I can refute the best examples without even trying hard, how does one really take this seriously?

I'm not sure how this refutes anything? Just because there are some examples that show PDO values consistently in outlier regions does not logically lead to the conclusion the statistic is useless.

Generally speaking, teams with a high PDO are experiencing luck because it means their shooting and save percentage rates are in an unsustainable territory. This is not always the case, and certainly you would never take a look at PDO and exclaim 'aha, now I recognize all of the factors of luck that played into their success/failure'. Use it as an indicator that you need to look more closely at their underlying numbers.

 

[TheDevilMadeMe]

Take some player from some team. Last year he was in line with the rest of his team from a Corsi standpoint, but for whatever reason his goalie stunk at even strength when he was on the ice, posting a .882 sv% when he was otherwise .920. he was situationally unlucky, and this badly damaged his +/-. It is predictable that this will likely regress to the mean for this player. It’s not predictable that he will continue to have bad puck luck.

But why do you need PDO to show you that? Isn't "his goalie's save percentage was unusually crappy just when he was on the ice, and as a forward, he probably had relatively little effect on on-ice save percentage" an adequate explanation?

Edit: or better yet, don't even bother to use single-season plus-minus as indicative of very much

 

[mindmasher]

Well if good teams tend to have consistently have high PDOs and bad ones tend to have consistently low ones, then I don't think it should be very revealing that "on average," they regress to the mean

Edit: if CYM's numbers are correct, that would seem to be what they are showing

This is simply not the case. Past PDO does not correlate with future PDO. In other words, a team with a low PDO in one season does not accurately predict they will have a low PDO in the upcoming season - it is much more likely they will move towards 1000.

I believe Vancouver and Boston are the number 1 and 2 teams respectively for PDO for the last five years, and repeatable results indicate that it is more than luck driving this success. I don't think anyone in the 'stats' community will refute that.

I've said it before and I'll say it again: PDO is not, and never will be, a perfect proxy for team luck. It's merely a useful indicator. A red flag if you will that helps people like you and I draw up a quick list of possible over- or under-performing teams or players.

Finally, I will also say I think all the new statistics should have descriptive and appropriate names. Cryptic titles help nobody.

 

[Czech Your Math]

I think the metric has quite a bit of predictability on a team level. It's not just the top two teams, but other teams that either have systems designed to prevent high quality shots or teams that are just not very good. Sure if you take the middle group of teams, they will tend to regress to the mean, as they would in many other metrics of overall value, because they tend to be average.

When I look at the featured examples that are supposed to illustrate the metric most clearly on a team level, 2012 Minnesota and 2011 San Jose, I immediately looked at the most likely culprits (goaltending in both cases, offense in Minnesota's case, etc.) and these explained the change in the fortune of each team much more than this metric. If the biggest examples of teams regressing are so obviously flawed, that doesn't give me much faith in the metric on a team level. Give me the PDO data for the past few years for all teams and I would guess I could predict who the top and bottom teams would be next year with decent accuracy. This isn't what would be expected from the metric.

It wasn't just Vancouver and Boston, but teams like Phoenix, Rangers and Nashville. Their overall PDOs may have been generally lower or higher for a shorter time, but there's a clear correlation to some degree. The Islanders, Columbus, and Toronto didn't have one overall PDO of 1000 or higher in 5 years between them. That's no coincidence.

I can see some use on an individual level, but it seems overblown. I mean, if a superstar is having a rough patch, most of us understand it's just bad luck and don't need to check their PDO to tell us this. If a scrub goes on a tear, it's usually not sustainable, and we intuitively know this, we don't need to consult PDO. As TDMM said, it's best not to rely on raw plus-minus for a single season or a partial season, and anyone worth their salt knows this. It doesn't mean raw plus-minus is useless, and maybe PDO isn't either, but that also doesn't automatically make either particularly useful.

Remember, this stat was touted as the "single most useful stat in hockey" and described as "primarily driven by luck." By disproving the latter, along with the featured examples, it becomes evident that the former statement is pure hyperbole.

 

[mindmasher]

I think the metric has quite a bit of predictability on a team level. It's not just the top two teams, but other teams that either have systems designed to prevent high quality shots or teams that are just not very good. Sure if you take the middle group of teams, they will tend to regress to the mean, as they would in many other metrics of overall value, because they tend to be average.

When I look at the featured examples that are supposed to illustrate the metric most clearly on a team level, 2012 Minnesota and 2011 San Jose, I immediately looked at the most likely culprits (goaltending in both cases, offense in Minnesota's case, etc.) and these explained the change in the fortune of each team much more than this metric. If the biggest examples of teams regressing are so obviously flawed, that doesn't give me much faith in the metric on a team level. Give me the PDO data for the past few years for all teams and I would guess I could predict who the top and bottom teams would be next year with decent accuracy. This isn't what would be expected from the metric.

It wasn't just Vancouver and Boston, but teams like Phoenix, Rangers and Nashville. Their overall PDOs may have been generally lower or higher for a shorter time, but there's a clear correlation to some degree. The Islanders, Columbus, and Toronto didn't have one overall PDO of 1000 or higher in 5 years between them. That's no coincidence.

I can see some use on an individual level, but it seems overblown. I mean, if a superstar is having a rough patch, most of us understand it's just bad luck and don't need to check their PDO to tell us this. If a scrub goes on a tear, it's usually not sustainable, and we intuitively know this, we don't need to consult PDO. As TDMM said, it's best not to rely on raw plus-minus for a single season or a partial season, and anyone worth their salt knows this. It doesn't mean raw plus-minus is useless, and maybe PDO isn't either, but that also doesn't automatically make either particularly useful.

Remember, this stat was touted as the "single most useful stat in hockey" and described as "primarily driven by luck." By disproving the latter, along with the featured examples, it becomes evident that the former statement is pure hyperbole.

I don't know who touted it as this kind of stat, but like most statistics only gives a certain amount of information.

It's a lot more useful than most vanilla stats in use by the average fan, that's for sure. I'll take my composite multi-stat analysis over yours any day

 

[Fourier]

Hunh? What is incorrect about his statement? I'm a huge fan of the Oilers and Eberle; I wish him the best, but it's very simple: he performed at a historically unsustainable level and it's very unlikely that he repeats it. Possible, as most things are, but unlikely.

Not even to mention you are trying to someone hold this up as an indicator of advanced statistic misuse - and I see *zero* arguments as to why this is the case. Why is this a ridiculous conclusion? Where are your arguments that his IPP and S% are sustainable and not influenced heavily by a season of good bounces. I would love to see some contrary arguments based on logic, not hand waving.

No one would argue that Eberle is likely to duplicate his shooting precentage. I also do not think it is at all unreasonable to question whether his numbers will drop next year. But so what? The prediction itself is trying to tell us that a player who is almost universally recognized as having elite offensive skills with one of the best shots that I have seen from a player his age in many years is somehow going to regress to the 45-50 point range because his shooting percentage and his IPP is higher than average. This after he put up prorated seasons of 21g and 51pts, and 38g and 81 pts in his first two NHL campaigns. (I use prorated numbers because so far as I know shooting % and IPP are not good predictors of when a player will get injured.) And he did this despite being 5th and 6th respectively amongst forwards in TOI/G, Moreover in his first year he played with a dogs breakfast of line mates and he was 7th on the team amongst forwards in pp TOI/g behind even Linus Omark.

And lets say for a minute that Eberle's S% drops to 12% next year. How many goals will he score? With more ice time does he get more shots. And even if he only scored 22 goals next year, to get into the 45-50 point range his assits total would have to drop by a minimum 14. Why is this so likely. Because of IPP? Is he suddenly going to no longer be a key player in the Oilers offense? Or is it becuase players like Hall and Nuge are going to see big dips in their games as well?

I really wonder if you polled 50 NHL coaches and GM's if you might find one who would actually predict that Eberle would be expected to score in the 45-50 point range next year if he is healthy. If the answer is no, how do you justify such a prediction?

Now you also asked me about why I do not think the use of these stats in the manner we typically ses is appropriate.

At this point we have no idea what Jordan Eberle's mean shooting % will be, but given the nature of his game it is reasonable to believe that it will be comparable with other first line players with very good to excellent shots. But even if we knew what to expect from his S% it is still not possible to predict accurately how many goals he would score. To illustrate this I picked three names of players who I thought might fit this mold at random to look at to see how shooting% might impact their numbers. Here is what there prorated season look like over stretches of their careers (in all cases these three actually played full seasons most years).

Player 1

G	A	PT	S%
18	36	54	11.8
22	49	71	10.8
36	48	84	15.3
29	45	74	11.7
31	51	82	10.9

Player 2

G	A	Pts	S%
28	23	51	13.3
31	36	67	11.3
33	43	76	13.5
52	44	96	16.7
38	35	73	11.1
42	33	75	15.5
35	32	67	11.9
46	64	110	14.8

Player 3

G	A	PTS	S%
18	22	40	12.8
25	29	54	15.2
33	37	70	16.4
38	56	94	17.9
31	30	61	14
43	59	102	15.8

While there is no doubt that S% and goal scoring are correlated, it seems clear that S% alone is actually quite a poor predictor of an individuals actual goal totals.

In player 1 we have two seasons of almost identical S% and yet the goals scored varied by 11 goals. The players linemates were mostly constant over these two seasons at least to the degree that in both years this player played nearly all the time with the same elite playmaker.

Player 2 had far more variance in his linemates and is the type of player that tends to make much of what he gets happen on his own. His numbers are actually closer tied to his S% but the relationship is still far from obvious. For example, we still have two years of 38 and 42 goals respectively despite the fact that the first was his lowest S% of this strecth at 11.1% and thet second was his second highest S% at 15.5%

Player 3 has a lot of similarities in his game to Eberle, but I'd say that it is possible that Eberle's shot is actually better. He has also played a lot with elite plays though these days he plays with someone who is certianly a top end triggerman. Certainly his goal scoring has fluctuated a lot but S%'s of 15.2% vs 15.8% do not explain the difference between a 25 goal season and a 43 goal season.

So I will ask you this. What is are the key characteristics of the statistic S% as a means of predicting goals scored? What does its distribution look like, both for a typical individual and in the cummulative pool? Do we know anything about the variance we might have in using the S% to predict output? What would an 80% confidence interval look like. Is it possible that the top 40 scorers in the league may not follow similar patterns to the masses or is it possible that this analysis is being done using the characteristics of a fair coin when what we have is one that is significantly loaded? .

Without understanding the characteristics of its distribution what predictive value is a statistic really? Without communicating these characteristics why should we put any significant value in any preditions that result from the data if they do not agree with empirical evidence? Can you personally give me any clear mathematical evidence, and by this I mean evidence that a pro might actually accept, that these stats have any real predictive value? I've never seen it.

I should add though that I have no problem with people trying to find gems in the huge piles of data that the NHL generates. There is some interesting stuff. But it has been my experience that too often people read into it more than it is actually there.

 

[Fourier]

I don't know who touted it as this kind of stat, but like most statistics only gives a certain amount of information.

It's a lot more useful than most vanilla stats in use by the average fan, that's for sure. I'll take my composite multi-stat analysis over yours any day

The guy who said that was Gabe Desjardins.

 

[theoil]

It was very hot the first 15 days of April hitting 30 degrees every single day without a drop of rain.

On the sixteenth of April a cold front moved in and the high every day was plus 10 but there was still no rain until the final two days of the month when it just poured.

The meteorologist on the local TV station did his monthly wrap up on May 1st by announcing that April was a typical month with an average 20 degree temperature and 85 mm of rain.

I carried my unbrella with me most of the month but was awfully hot in my wool sweater for the first two weeks and a pretty chilly the last two. I forgot my unbrella the last two days of the month and got drenched.

Looking back at the averages though I have to agree that it was a pretty typical and predictable month of weather. Of course if it wasn't I could just go to annual weather averages which would give me more data points. With an average high annual temperature of 12 I would know a lot more about how to dress every morning when I got up.

 

[Hynh]

It was very hot the first 15 days of April hitting 30 degrees every single day without a drop of rain.

On the sixteenth of April a cold front moved in and the high every day was plus 10 but there was still no rain until the final two days of the month when it just poured.

The meteorologist on the local TV station did his monthly wrap up on May 1st by announcing that April was a typical month with an average 20 degree temperature and 85 mm of rain.

I carried my unbrella with me most of the month but was awfully hot in my wool sweater for the first two weeks and a pretty chilly the last two. I forgot my unbrella the last two days of the month and got drenched.

Looking back at the averages though I have to agree that it was a pretty typical and predictable month of weather. Of course if it wasn't I could just go to annual weather averages which would give me more data points. With an average high annual temperature of 12 I would know a lot more about how to dress every morning when I got up.

That's wrong.

The PDO disbelievers would expect the hot weather to continue forever and use the 15 days as proof. The PDO users would expect the average temperature over the next 15 days to be 20, for an average of 25 degrees.

If you flip a coin 100 times and it comes up 75 heads and 25 tails, you don't bet that the next 100 will be 75 tails and 25 heads. You bet that there will be 50 heads and 50 tails, or 125 and 75 total. Flip the coin until you have 1000 results and the PDO users would expect there to be 525 heads and 475 tails. Do it until there are 10,000 results and the expected split is 5025 and 4075.

If you flip that coin 1 million times, the expected result would be 500,025 heads, 499,975 tails. Instead of 75% of results being positive, only 50.0025% are.

 

[Czech Your Math]

The PDO disbelievers would expect the hot weather to continue forever and use the 15 days as proof. The PDO users would expect the average temperature over the next 15 days to be 20, for an average of 25 degrees.

This PDO disbeliever (to a large degree at least) would expect the weather to remain hotter than normal at least for a while. Not necessarily as hot as it has been, but not completely random either.

I might say "expect temperatures to average 23-25 in the coming days." No one can forecast the results with complete accuracy, but it's not all luck either.

 

[Fourier]

That's wrong.

The PDO disbelievers would expect the hot weather to continue forever and use the 15 days as proof. The PDO users would expect the average temperature over the next 15 days to be 20, for an average of 25 degrees.

If you flip a coin 100 times and it comes up 75 heads and 25 tails, you don't bet that the next 100 will be 75 tails and 25 heads. You bet that there will be 50 heads and 50 tails, or 125 and 75 total. Flip the coin until you have 1000 results and the PDO users would expect there to be 525 heads and 475 tails. Do it until there are 10,000 results and the expected split is 5025 and 4075.

If you flip that coin 1 million times, the expected result would be 500,025 heads, 499,975 tails. Instead of 75% of results being positive, only 50.0025% are.

You are of course assuming that the coin you flip is fair. What if it is not? This is part of the problem with PDO.

 

[theoil]

That's wrong.

The PDO disbelievers would expect the hot weather to continue forever and use the 15 days as proof. The PDO users would expect the average temperature over the next 15 days to be 20, for an average of 25 degrees.

If you flip a coin 100 times and it comes up 75 heads and 25 tails, you don't bet that the next 100 will be 75 tails and 25 heads. You bet that there will be 50 heads and 50 tails, or 125 and 75 total. Flip the coin until you have 1000 results and the PDO users would expect there to be 525 heads and 475 tails. Do it until there are 10,000 results and the expected split is 5025 and 4075.

If you flip that coin 1 million times, the expected result would be 500,025 heads, 499,975 tails. Instead of 75% of results being positive, only 50.0025% are.

Though, of course, no matter how many times you flip a coin the chances of it being 50% heads and 50% tails on a number that ends with 3 or more zeros is frightfully insignificant given that coin flipping is considered the paradigm for the proposition that defines regression to the mean.

I am not sure what to say about PDO that hasn't already been stated. Formulas that prove that the middle of the pack will act like the middle of the pack given enough data points while telling us nothing pertinent about outliers (other than that they can't be included because they are outliers) doesn't seem to have anything very profound to add to a discussion in which there are only 30 sets of data involved.

Telling me after the fact that my team's collapse was predictable because they had an unsustainable PDO to start the season would be more impressive if really good teams didn't sustain their high PDO over the season.

And extending the data collection to 5 or 10 years in order to make the data prove the conclusion may be esthetically pleasing I don't think you would find many hockey fans surprised at the notion that their team tends to suck after a number of years of being very good or that bad teams improve given enough time.

 

[Hammer Time]

The problem is, how would you determine what number the PDO should regress to in the long run? For a team like the Rangers it should be significantly higher than 1000, while I would expect the opposite for the Leafs.

 

[Czech Your Math]

The problem is, how would you determine what number the PDO should regress to in the long run? For a team like the Rangers it should be significantly higher than 1000, while I would expect the opposite for the Leafs.

Yes, if one used an estimate, it might actually have some value. Pretending it will regress to 1000 just makes it that much less valuable.

 

[PDO]

My only comment at the moment is about the unfortunate tendency to give new advanced stats impenetrable names. There's no reason at all for this to be called PDO. It's not descriptive in the least; it's not even an attempt to be descriptive. The same goes for Corsi and Fenwick. I don't want to believe it's an attempt at exclusiveness, where a thing is called something non-descriptive so that only those in the know will understand it. But sometimes I do wonder. There's any number of things I developed over the years that I could have just called "the Fyffe ratio" or what have you, but I use descriptive terms, because the purpose of new stats is to explain and clarify things, not to obfuscate them.

Sorry, that's presumably not the type of thought you were looking for.

Unless some guy named PDO came up with it. THEN...

 

[PDO]

I've looked at this a bit more and I still don't get it.

First, the statement of it being "the single most useful statistic in hockey" is so flabbergasting, I honestly don't know how to even begin.

Second, the basis of it is said to be that both SV% and shooting % are "primarily luck-driven." Really? Maybe in the short term, but over any longer period, that's clearly false IMO. SV% is considered as or more important than any statistic for judging goalie performance by those that specialize in this. Shooting % isn't one of my favorite stats, but to say it's primarily luck-driven also doesn't seem correct IMO.

There's a graph on the link in the OP for this thread, but I'm not sure what to conclude from it. If it shows regression to the mean over time, why does it start rising at some point, plateau (2000-2500 shots) and then fall (2500-3000 shots)? Shouldn't it continue rising as the sample size increases?

I don't know where to find all the relevant team 5v5 data (shooting%, SV%, shots F-A, etc.), so I'm looking at overall data here. Here are the examples given in the link to PDO:

1) 2012 Minnesota- This seems to be the featured keystone example in the linked article and the whole basis of this example is that Minn's SV% was an unreasonably high .944 as of Dec. 11. This does seem quite high, but am not sure how 5v5 SV% varies from overall SV%, so not sure how high the variation was. What else was happening besides "luck regressing to the mean"?:

A) The top 2 Minn. goalies each had .932 overall SV%s as of Dec. 11. It seems odd that they would have basically identical SV%s given that this is supposedly all driven by luck.

B) Backstrom's .932 overall SV% to that point was .018 better than the year-end league avg. of .914. In his first 3 seasons ('07-'09) his overall SV%s were .024, .012 and .015 better than league avg. How is .018 better than league avg. so different from some of his previous performances?

C) Backstrom was the starter or at least the biggest part of the tandem/trio. He misses large chunks of games in both January and March. This means he may have been rusty, still partially injured, or (looking at the data) possibly overworked upon his return. This also means the remaining goalie(s) may have been overworked during his absence.

D) Through Dec. 11, excluding SO goals, Minn. averaged 2.47 GPG, while after Dec. 11 they averaged 1.77 GPG.

E) They finished with an overall SV% + S% of 995, so they must have been very unlucky the last 2/3 of the season, right?

Attributing Minnesota's collapse purely to "luck regressing to the mean" seems quite simplistic and likely incorrect IMO. I'm going to quickly address the other examples, since they were basically glossed over in the linked article, and after the featured example, my impression of this metric, or at least the article supporting it, went from skeptical to "does not pass inspection."

2) 2012 St. Louis- They were supposedly unlucky at the start of the season before luck regressed to the mean. Maybe so, but according their overall SV% + S% was the fourth highest in the league at 1014, even with such an "unlucky" start. How does a stat, that supposedly regresses to the mean, fluctuate so wildly that it goes from unlucky to not just normal but very lucky? How does one draw valid conclusions from such a stat?

3) 2011 Dallas & New Jersey- Again, Dallas was supposedly very lucky at the start and New Jersey very unlucky. Yet Dallas finished 7th or 8th in overall SV% + S% at 1012 and New Jersey finished last at 983. I don't see where their luck regressed to the mean, at least enough to explain their different fates.

4) 2010 Colorado- Once again, Colorado was supposedly lucky at the start, but finished second with an overall SV% + S% of 1021. I don't see where their luck regressed to the mean, at least enough to explain their fall.

Other team examples of overall SV% + S% (which is supposedly random and driven by luck, at least the 5v5 version):

Vancouver was 1018, 1019, 1026 and 1019 the last 4 seasons.
Boston was 1036, 998, 1023 and 1019 the last 4 seasons.
Nashville was 1016 and 1024 the last 2 seasons.
Phoenix was 1006, 1010, and 1013 the last 3 seasons.
Rangers were 1006, 1013 and 1019 the last 3 seasons.

Islanders were 986, 989, 998, and 984 the last 4 seasons.
Columbus was 999, 994, 984, and 984 the last 4 seasons.
Toronto was 981 and 975, before improving to 997 and 998.
Edmonton was 989 and 990 before improving to 1007.

To me, it's a mix of competing factors:

- high shooting skill
- teams waiting for high quality shots
- teams forcing low quality shots
- great goaltending

vs. the opposites of each of those vs. luck

It's a patchwork of various factors, which are often muddled by a large dose of luck. That makes its use severely limited as either a measure of total skill by goalies/shooters on a team, or as measure of luck. The individual version is even more prone to fluctuation and so basically even more useless. Why would I use the goalie's SV% over a very small sample as half of a metric for an individual skater, whether it's supposed to indicate the skater's skill or luck?

I think if one used it on a team basis and compared it to how that team did in the past, it might indicate that either the team is getting better/worse or luckier/unluckier. However, as I said in a previous post, why wouldn't one just use GF/GA or some such metric that is at least as easy to find and is more directly indicative of performance?

What is IPP?

I wouldn't call PDO an "advanced statistic", I'd call it a new statistic that is very simple and very misleading. It has two components, one of which the individual has much control over and may indicate his skill to a large degree, the other of which he has very little control over (except to prevent high quality shots) and therefore has little to do with his skill. It's like putting sauerkraut and Oreos together and calling it an advanced food.

Look at the shooting %s of Brett Hull or Stamkos. They jumped in their second seasons, but that doesn't mean they didn't improve further a season or two later. Of course such levels are not sustainable for most players over longer periods, but it doesn't mean it was mostly luck.

It's not the individuals shooting %, it's the teams shooting % with him on the ice. Essentially, its the other teams goalies SV %. Extremely different things, and blows up most of your argument..

 

[PDO]

The problem is, how would you determine what number the PDO should regress to in the long run? For a team like the Rangers it should be significantly higher than 1000, while I would expect the opposite for the Leafs.

That would be a heckuva lot more work .

Ironically, a lot of the teams that benefit from great goaltending (LA, NYR both come to mind) are low shooting pct teams.

However, if you take a quick look at each player on the team, sort it... even then, you can figure out on a player by player basis. It's a baseline, and something that has become more popular than I ever imagined (for example, it's being used in European Soccer... I have no idea how that came about...), but when you get right down to it.. it's just a baseline. Back of the envelope test. It's not supposed to be some sort of perfect stat that explains everything... it's just a quick "something might not add up here."

And that's about it.

FWIW, I made an absolute mint fading Minnesota the second half of the season.

Lastly, when I first came up with this, it was at a player-level. A site called Irrelevant Oiler Fans had a post up on the end of season numbers, believe it was 07-08, I know Robert Nilsson was on the team. All the players on-ice sv% and on-ice s% numbers were listed, and the post glossed over guys at either end of the spectrum who may have got the short end of the stick. Seemed intuitive to me to pop those numbers together, on the belief that bounces could even out at either end of the ice for some guys, but not for others...

 

[Master_Of_Districts]

Some in this thread are claiming that PDO is a primarily luck-driven statistic.

Others are claiming that some teams will have significantly better PDO numbers than others over the long run.

These statements are not necessarily incompatible. Both can be true, depending on the context.

With respect to the first statement, it is true that, at the level of a single NHL regular season, most of the variation between teams is luck.

This can be measured with some precision - if we look at each season from 2003-04 to 2010-11, for example, we find that 61% of the team variation in PDO was due to luck, and 39% due to skill.

This carries implications for estimating a team's true talent level with respect to PDO. For example, during the above noted period, if we single out the teams with the best PDO number in the league in each of the seven seasons, they posted an average mark of 1.025. Because most of the league-wide variation is due to luck, however, our best estimate is that a team with a 1.025 PDO number actually has a true talent PDO number of only 1.00975.

This corresponds well with the observed data - the teams in the above study posted an average PDO number of 1.0083 in the following season.

Of course, merely because most of seasonal variation in PDO is due to the luck isn't to say that all teams have the same underlying talent with respect to PDO. The skill standard deviation at the team level is about 0.0074.

So if the team with the best underlying PDO talent in the league is two standard deviations above the mean, it would have a true talent PDO number of 1.0148. That's a sizable advantage. The problem is that because most the variation over a single season is due to luck, that team's observed results will vary significantly around that mean.

The Bruins had an average PDO number of 1.011 in each season from 2007-08 to 2010-11, but the range over that period was considerable (0.991 - 1.029).