Non-traditional metrics: PDO

[tarheelhockey]

After running quite a few numbers and comparing with the Wild's performance, I am leaning towards PDO being an indicator of quality scoring chances and less an indicator of luck.

Odd man rushes and breakaways are better scoring chances than perimeter shots. Good teams box out the opposition and limit odd man rushes. Good teams transition on the breakout quickly and force turnovers leading to odd man rushes. All these things drive up PDO in a way that's not related to luck.

I like that interpretation. PDO has always rubbed me the wrong way in that it describes success and failure as "luck", across the board, no questions asked. That just isn't good deductive logic in my opinion.

 

[Master_Of_Districts]

I'm not sure it's a predictive stat, but it sure seems to be a descriptive stat.

Last night, Wild got killed. Just terrible. In every sense of the word. They gave up a breakaway first shift of the game and LA scored. Then they gave up a 3-on-1 two shifts later and LA scored. Lots of turnovers and crappy shots from the perimeter.

PDO was 786, shot differential +9. Can't really tell anything from Corsi because LA scored on their first two shots and Wild never scored.

But if you look at the PDO, you think, wow the Wild were terribly unlucky. You watch the game, probably their worst game of the year.

Not to be a dick - and I mean that sincerely - but the problem with your interpretation is that it's contradicted by all of the available evidence.

For example, quite a few bloggers counted scoring chances for a variety of teams over the 2010-11 NHL season. I happened to have collected data from all games for which scoring chances were counted. It turns out that there was scoring chances data for 386 out of the 1230 games played that year, which isn't bad.

For that 386 game sample, I took the trouble of calculating each team's:

1. Scoring Chance Ratio
2. Fenwick Ratio
3. Corsi Ratio
4. PDO Number

The correlation between Scoring Chance Ratio and Fenwick Ratio was substantial, at 0.82. The correlation between Scoring Chance Ratio and Corsi Ratio was high as well, although somewhat less so, at 0.69.

The correlation between PDO Number and Scoring Chance Ratio, however, was negative 0.37. If PDO was reflective of how well a team was playing, we would have expected to obtain a significant positive correlation, like with Fenwick and Corsi. But we didn't.

This result is actually not too surprising, given that the majority of variation in PDO at the team level over the course of a single season is due to luck. If a majority of the variation in a statistic is explained by luck, it, by definition, is unlikely to be correlated with measures of ability. So one would expect PDO to be uncorrelated with measures of team ability, such as scoring chance ratio.

This is actually susceptible to precise calculation - if you look at data from the 2007-08 to 2010-11 NHL seasons, it appears that variation in PDO at the team level is 69.6% luck, 30.4% skill.

If you expand the sample to include the 2003-04 to 2006-07 NHL seasons, the skill component increases to 39.1%, but the majority of the variation from team to team is still explained by luck.

All of this is not to say that PDO is entirely a reflection of luck - it's not, and the above data bears that out. Some teams will sustainably do better than others with respect to PDO over large samples.

It's simply that when we're talking about 20, 30, or 40 game samples, or even an entire regular season, the noise tends to dominate the signal.

 

[Master_Of_Districts]

And just to expand on the above, given that we're dealing with a relatively small of games (386), it's likely more sensible to use weighted rather than raw correlations.

For example, there were only nine Chicago games in the sample, but 82 games from Montreal and Edmonton. This has the potential to skew things.

Fortunately, the correlations don't change much if we weight each team's data by number of games played (or more precisely, aggregate fenwick for and against).

The correlation between Fenwick Ratio and Scoring Chance Ratio drops slightly to 0.81, whereas the correlation between PDO Number and Scoring Chance Ratio increases to negative 0.17 (which makes sense - in theory, we would expect something closer to zero).

 

[Jarick]

How are you calculating Scoring Chance ratio?

 

[Master_Of_Districts]

How are you calculating Scoring Chance ratio?

(Scoring Chances For)/(Scoring Chances Against)

 

[Jarick]

Possible explanation would be that PDO is a measurement of quality of scoring chances, independent of the number of them for or against a team.

 

[Jarick]

An update:

Through 44 games for the Wild, there's been a nearly perfect correlation between PDO and wins/losses.

Only once did they lose in regulation with a PDO of over 1000 (1006).
Only once did they lose in overtime with a PDO of over 1000 (1004).

Only once did they win in regulation with a PDO under 1000 (986).
Only once did they win in overtime with a PDO under 1000 (965).

Under the old interpretation of PDO, that would mean the Wild were lucky on all but two of their wins and unlucky on all but two of their losses.

Doesn't that strike you as odd?

 

[Master_Of_Districts]

An update:

Through 44 games for the Wild, there's been a nearly perfect correlation between PDO and wins/losses.

Only once did they lose in regulation with a PDO of over 1000 (1006).
Only once did they lose in overtime with a PDO of over 1000 (1004).

Only once did they win in regulation with a PDO under 1000 (986).
Only once did they win in overtime with a PDO under 1000 (965).

Under the old interpretation of PDO, that would mean the Wild were lucky on all but two of their wins and unlucky on all but two of their losses.

Doesn't that strike you as odd?

PDO is primarily a measure of luck over small sample sizes, like a single NHL game.

And over small sample sizes, luck is heavily correlated to results.

So it's not surprising at all.

Like I said earlier, some teams will surely post PDO numbers than others over the long run (~3 seasons).

But in the short term (i.e. half a season), the stat is dominated by luck. To the point where people can reasonably use the stat as a proxy for luck.

If PDO is a measure of how well a team is playing, doesn't it strike you as odd that there's no correlation between PDO and scoring chance differential - which everyone would agree is a reasonable proxy for how well a team is playing - at the team level?

 

[Jarick]

If it's only useful in one game or over a full season, then what's the point of this stat?

PDO is highly, highly correlated with win percentage. It's not much different from just looking at the win/loss column.

What bugs me is attributing those wins or losses to luck.

What's your correlation between scoring chance differential and win percentage?

 

[Jarick]

Just ran some quick numbers using the latest standings and NHL Numbers PDO stats...

Correlation between...

PDO and Fenwick = -0.179
PDO and Points = 0.644
Fenwick and Points = 0.445
Goal Differential and Points = 0.946

There's a more positive correlation between PDO and Points than Fenwick and Points.

Fenwick is an estimation of scoring chance differential, and you had measured scoring chances in some games...both Fenwick and scoring chances had a negative correlation with PDO.

It comes back around as PDO being a descriptive statistic of how well a team played. If you out-shoot and out-save an opponent, you're going to win. A very high or low PDO would regress to the mean just as a 10-game win or loss streak would.

But not necessarily the same mean for all teams. Some teams will be better than others and have higher or lower win percentage (and PDO) as a result.



When I'm looking at the standings vs PDO or Fenwick, there's some outliers that are interesting.

Tampa has a high PDO, low Fenwick, and low standings, yet higher goal differential than anyone around them in the standings. So...they are lucky yet terrible?


More correlations...

PDO-GF/G = 0.627
PDO-GA/G = -0.484
PDO-GDiff = 0.701

FW-GF/G = 0.121
FW-GA/G = -0.460
FW-GDiff = 0.394

Higher correlation with PDO and Goal Differential than Fenwick and Goal Differential.

Interesting that Fenwick and Goals per Game is so close to zero?

 

[Master_Of_Districts]

Just ran some quick numbers using the latest standings and NHL Numbers PDO stats...

Correlation between...

PDO and Fenwick = -0.179
PDO and Points = 0.644
Fenwick and Points = 0.445
Goal Differential and Points = 0.946

There's a more positive correlation between PDO and Points than Fenwick and Points.

Fenwick is an estimation of scoring chance differential, and you had measured scoring chances in some games...both Fenwick and scoring chances had a negative correlation with PDO.

It comes back around as PDO being a descriptive statistic of how well a team played. If you out-shoot and out-save an opponent, you're going to win. A very high or low PDO would regress to the mean just as a 10-game win or loss streak would.

But not necessarily the same mean for all teams. Some teams will be better than others and have higher or lower win percentage (and PDO) as a result.



When I'm looking at the standings vs PDO or Fenwick, there's some outliers that are interesting.

Tampa has a high PDO, low Fenwick, and low standings, yet higher goal differential than anyone around them in the standings. So...they are lucky yet terrible?


More correlations...

PDO-GF/G = 0.627
PDO-GA/G = -0.484
PDO-GDiff = 0.701

FW-GF/G = 0.121
FW-GA/G = -0.460
FW-GDiff = 0.394

Higher correlation with PDO and Goal Differential than Fenwick and Goal Differential.

Interesting that Fenwick and Goals per Game is so close to zero?

Luck is correlated with PDO.

Luck is correlated with results.

PDO is correlated with results.

Questions:

How well does PDO predict future results at even strength (one half of the year to the next)?

How well does PDO repeat from one half of the year to the next?

 

[Master_Of_Districts]

Luck is correlated with PDO.

Luck is correlated with results.

PDO is correlated with results.

Questions:

How well does PDO predict future results at even strength (one half of the year to the next)?

How well does PDO repeat from one half of the year to the next?

This might illustrate my point a little better.

Consider the 2007-08 to 2010-11 NHL seasons - the range of seasons for which detailed EV team data is readily available.

Let's divide each of those seasons into two halves, and examine each half season separately.

As you've noted, PDO is substantially correlated with results.
The correlation between 1st half PDO and 1st half EV goal differential is 0.75 at the team level.

Unfortunately, PDO has no predictive power at all.

The correlation between 1st half PDO and 2nd half EV goal differential is effectively zero (-0.007).

And the correlation between 1st half PDO and 2nd half PDO is also effectively zero (0.007).

In summary, PDO:

- does not repeat from one half of the season to the next
- cannot predict future EV goal differential

That looks an awful look like a statistic that is primarily luck driven.

 

[Czech Your Math]

Unfortunately, PDO has no predictive power at all.

The correlation between 1st half PDO and 2nd half EV goal differential is effectively zero (-0.007).

And the correlation between 1st half PDO and 2nd half PDO is also effectively zero (0.007).

In summary, PDO:

- does not repeat from one half of the season to the next
- cannot predict future EV goal differential

That looks an awful look like a statistic that is primarily luck driven.

What is the correlation from one season to the next? For instance, from '11 to '12, the last two full seasons?

 

[Micklebot]

What is the correlation from one season to the next? For instance, from '11 to '12, the last two full seasons?

Would that really be worth looking at? Lots of change in personel from year to year, seems like a lot of unaccounted for variables.

Anyhow, I plugged ES numbers from 11 and 12 into excel and got this:

11 GF to 12 PDO 0.5874343331
12 GF to 11 PDO 0.332863218

11 PDO to 12 PDO 0.4429501639
11 GF to 12 GF 0.6158271089

 

[PSGJ]

I see no real reason to use PDO. It obfuscates more than it clarifies. It's much better to look at sv% and shooting% and analyze those numbers on their own.

Shooting% for an entire team will regress simply due to the number of players involved.

There is no reason however to believe the same about goal tending. One goalie who plays the majority of the games will have a significant impact and will be able to post better or worse numbers depending on how good he is.

 

[GuineaPig]

I see no real reason to use PDO. It obfuscates more than it clarifies. It's much better to look at sv% and shooting% and analyze those numbers on their own.

Shooting% for an entire team will regress simply due to the number of players involved.

There is no reason however to believe the same about goal tending. One goalie who plays the majority of the games will have a significant impact and will be able to post better or worse numbers depending on how good he is.

You're right, for the most part, about goaltending. There's little reason to suspect that a team like Vancouver or New York, to pick two examples, will converge upon the mean save percentage for the season. But it's still useful to include in PDO because it will frequently wildly diverge from the expected mean over small sample sizes.

The thing is that one can't use PDO as a blunt object. It needs to be properly contextualized. Not every team should be expected to regress to 1000.

 

[Jarick]

Luck is correlated with PDO.

Luck is correlated with results.

PDO is correlated with results.

By that logic, if PDO is a measurement of luck, and 75% of wins could be described by PDO, then 75% of a team's results are luck.

Doesn't make a bit of sense to me.

The correlation between 1st half PDO and 2nd half EV goal differential is effectively zero (-0.007).

And the correlation between 1st half PDO and 2nd half PDO is also effectively zero (0.007).

In summary, PDO:

- does not repeat from one half of the season to the next
- cannot predict future EV goal differential

That looks an awful look like a statistic that is primarily luck driven.

What's the correlation between first half and second half win percentage?

You're right, for the most part, about goaltending. There's little reason to suspect that a team like Vancouver or New York, to pick two examples, will converge upon the mean save percentage for the season. But it's still useful to include in PDO because it will frequently wildly diverge from the expected mean over small sample sizes.

The thing is that one can't use PDO as a blunt object. It needs to be properly contextualized. Not every team should be expected to regress to 1000.

And save percentage is highly system-driven. A trapping team that collapses in the defensive zone will tend to have a higher save percentage than a team that plays run-and-gun. PDO says the trapping team's opposition is less lucky, or that the trapping team is more lucky.

The question I'm asking then is, what number should teams regress to? If not 1000, then what?

 

[Master_Of_Districts]

By that logic, if PDO is a measurement of luck, and 75% of wins could be described by PDO, then 75% of a team's results are luck.

Doesn't make a bit of sense to me.

A couple of points:

One - we're talking about even strength goal differential, not wins.

Two - when you say "75% of wins could be described by PDO," you're implicitly referring to variance.

0.75 is the value of the correlation. You have to square the correlation to obtain the variance value (assuming the involved variables are normally distributed, which is approximately true here).

So PDO explains about 56% of the variation in 1st half EV goal differential, not 75%.

Three - as has been stated before, PDO =! luck. Rather, a substantial percentage of the variation in Team PDO is attributable to luck over small sample sizes, thereby allowing PDO to serve as a proxy for luck over the same small sample sizes.

Four - when you say "75% of a team's results are luck," the truth value of that statement depends on the sample size in question. Over 82 games, results at the team level are roughly 65% skill - 35% luck (in the post-lockout era).

There is a sample size for which results at the team level would be 75% luck - 25% skill. In fact, that would be true at approximately the 12 game mark of the regular season.

What's the correlation between first half and second half win percentage?

It's relatively low in the post-lockout era - roughly 0.4 - 0.5. Maybe even a touch lower.

 

[PDO]

As a Wild fan, it shouldn't be so hard for you to be convinced of PDO.... the Wild ran on an unbelievably hot/lucky streak right up until Christmas last year. When they stopped converting at ludicrous numbers and getting HHOF goal-tending every night, they invariably fell out of the playoffs to nobodies surprise...

A team level, it can show you outliers and let you know how "real" a teams results are. Teams running with very low PDO's are likely to hit a hot streak soon (this year the Rangers are a very good example, as are the Blue Jackets) while teams running a very high one are likely to hit a cold streak soon (this years Leafs would be the poster child, but the season won't be quite long enough... if management believes their results this year are from the hockey magick â„¢ being perpetuated, look for a lot of confusion and anger when they miss the playoffs next year).

It's the same at an individual level really. Guys with very large PDO's tend to get overpaid while teams that pick up guys with very low PDO's tend to have them get a "bounce-back" year. Sean Couturier for example didn't forget how to play defense, he just can't buy a save when he's on the ice.

 

[EVBetting Site]

And save percentage is highly system-driven. A trapping team that collapses in the defensive zone will tend to have a higher save percentage than a team that plays run-and-gun. PDO says the trapping team's opposition is less lucky, or that the trapping team is more lucky.

The question I'm asking then is, what number should teams regress to? If not 1000, then what?

Every team would have their own "true" or "pure" expected PDO value but it would (I would assume and can say with 99+% confidence) take such a large sample size that it is, in a practical sense, impossible to determine for a variety of reasons, the most obvious of which is that no team is kept intact for long enough to converge to that "true" value (even if the required sample size is no larger than 2 or 3 seasons worth, though I believe it's larger than that even).

A defensive collapse team, in your words, would probably (I will throw some random numbers out that generalize things here) converge to a true value that is closer to 1001 (now, it may still be quite different from 1001, but that doesn't make this statement untrue) than it is to 1000. It's easier to influence (to what degree I don't have the slightest clue) team SV% compared to team SH%, or at least I believe it is. Now, say this hypothetical team's true PDO value is 1005. We can never actually determine this, so it's more accurate to just simply keep in mind that it is more likely to converge to a number slightly greater than 1000 than it is to converge to a number slightly less than 1000.

Now, if we are looking to determine to what degree, or how much higher or lower than 1000 a team's PDO "should" converge to, it's an unfair question. A very simple question (I believe this was your question in a general sense) with a very complex answer that would take hours on hours of research and statistical manipulation that could very well lead to nothing productive in the end. We have no way of collecting enough data due to the fluctuations of a single team's make-up even over the course of a single season, never mind over two or 3 or more so we need to extrapolate and use assumptions to reach any meaningful (often not even meaningful) conclusions.

If someone could answer your question, they would have one of the most valuable pieces of information related to the NHL in their possession I do believe. You could then very easily rank performance vs expected results with a few extra pieces of contextual information as well.

 

[SmellOfVictory]

I like that interpretation. PDO has always rubbed me the wrong way in that it describes success and failure as "luck", across the board, no questions asked. That just isn't good deductive logic in my opinion.

I'd say that more a distaste for the terminology than the spirit of the description. "Luck" just means events or trends that are not consistently repeatable; if we're talking about a player playing "hot", for example, it doesn't mean he's not playing well (e.g. maybe he is generating a substantial number of breakaway opportunities proportional to what would be expected of him normally) but rather that he does not have the ability to consistently drive that effect in a repeatable fashion. It feels less random, in terms of "luck", than the roll of a die, but the overall effect is roughly equivalent.

It is acknowledged that PDO isn't the same across the board, either. Players on either end of the spectrum will have consistently high or low PDOs, to an extent, and this is a reflection of some aspect of their skill level in relation to the average player in the league (or the fact that they happen to play in front of a really good goalie). It's that the vast majority of it, beyond a couple of percentage points either way, is variance - whether that be hot/cold streaks, goalie issues, or straight up luck in the traditional sense (e.g. shooting from the same areas as normal, but simply having the hockey gods favour you and a greater success rate per shot).

 

[TKB]

...A very simple question (I believe this was your question in a general sense) with a very complex answer that would take hours on hours of research and statistical manipulation that could very well lead to nothing productive in the end. We have no way of collecting enough data due to the fluctuations of a single team's make-up even over the course of a single season, never mind over two or 3 or more so we need to extrapolate and use assumptions to reach any meaningful (often not even meaningful) conclusions.

If someone could answer your question, they would have one of the most valuable pieces of information related to the NHL in their possession I do believe. You could then very easily rank performance vs expected results with a few extra pieces of contextual information as well.

Time, that could be used actually watching and analyzing the games/players themselves...

 

[Cunneen]

Quick think to note about PDO, research has shown that while shooting percentages are basically random and can't be predicted over time, Save percentage has far less variance. Teams with good goaltending (like the Rangers) can consitantly over time obtain an above average save percentage, and this makes sense since they have King Henrik.

As a result, PDO is slightly skewed because while shooting percentage is random, save percentage is far less random. PDO as a result does only a decent job at revealing how lucky a team is. While it can generally tell if a team is getting unsustainable results (a team with a high PDO might have both a unsutaibable shooting percentage and save percentage (for their goalie), while the Rangers might have the same PDO # and only their shooting percentage can be shown to be lucky.

Here is the link for the above mentioned research http://www.boysonthebus.com/2013/05/02/formulating-a-new-statistic-for-team-luck-part-i/

I think the that PDO needs to be revised a bit to fix the fact that Save percentage is not as luck driven as shooting percentage for specific teams. But thats a project for another day

 

[Cunneen]

I think PDO is far more valuable at the individual level because Save percentage when one player happens to be on the ice is definitely more random over a course of a season than a goalie's full year save percentage.

PDO has shown to be high in players playing above their talent level for a year while it has been able to partially predict bounce back/breakout years for players with low PDOs. Thus, in my opinion it is far more valuable to look at on the individual level.

 

[Cunneen]

and as previous posters have stated, PDO also is a better revealer of luck at small sample sizes. PDO over the first 15 games of a season will tell us more about how lucky a teams has been compared to PDO after 70 games. The larger the sample size, the more truth we can see in numbers like save percentage, thus the less luck we can determine from PDO value.