Hockey Reference's Playoff GSAA - why don't they add up?

[seventieslord]

Something I noticed about Hockey Reference's Playoff GSAA numbers today:

Ed Belfour Stats | Hockey-Reference.com

Take a look at Ed Belfour. Look at his season by season regular season GSAA numbers. Just to keep it simple, look at his FLA and SJS career totals. They match his single season numbers. Also, he was with TOR for just three seasons and you can see that his 24.6, 10.1 and -13.3 add up to his career total of 22.6.

But then look at his playoff numbers. His TOR GSAA numbers of 2.4 and -1.0 should add up to 1.4, but they come out to 8.6. His Chicago numbers add up to 27.5 but in the career totals line, they show as 40.9.

Same thing with Curtis Joseph:

Curtis Joseph Stats | Hockey-Reference.com

Regular season numbers add up perfectly. But in the playoffs, his EDM numbers add up to 0.9 but the career totals line says 9.5. His DET numbers should add up to 3.1 but they say 6.6. In TOR he adds up to 4.2 but the career line says 21.7!

Can anyone identify why the numbers come out like this? Is it a site glitch, a calculation error or is it somehow correct?

Click to read more...

 

[Doctor No]

I don't spend too much time there (and hadn't noticed this yet) - based on how they describe the statistic, it should be additive (and clearly is for regular season).

Interestingly enough, even if you take just Joseph's six "positive" GSAA playoffs, you don't get to the 62.1 that they have for his playoff career.

My calculations (slightly different methodology so naturally aren't identical) yield 161.0 for regular season and 24.8 for playoffs.

 

[Hockey Outsider]

There's definitely a formula error.

I looked at Patrick Roy as an example. You need to multiple his season-by-season results by 4.6 to add up to his career totals. But that approach doesn't work for Joseph or Belfour. (Nor does it work for Nabokov - another goalie who played on two playoff teams). Just a bug in their website.

 

[seventieslord]

There's definitely a formula error.

I looked at Patrick Roy as an example. You need to multiple his season-by-season results by 4.6 to add up to his career totals. But that approach doesn't work for Joseph or Belfour. (Nor does it work for Nabokov - another goalie who played on two playoff teams). Just a bug in their website.

I wonder if it would be possible to identify exactly what it's doing to calculate wrong. I doubt it.

 

[ContrarianGoaltender]

I wonder if it would be possible to identify exactly what it's doing to calculate wrong. I doubt it.

Shouldn't be too hard to figure that out, just look at a goalie with only one playoff season. I picked Alex Nedeljkovic, for example. Hockey Reference has him at 0.3 GSAA in the 2021 playoffs, but 3.3 under his career playoff totals. That gives us an implied league average save percentage of .919 for the former and .908 for the latter. Considering that .908 was the regular season league average in 2020-21, it seems pretty clear that the season-by-season playoff GSAA stats are measured against the playoff average, while the career GSAA playoff stats are measured against the regular season average. Different baselines will obviously give different results, and since save percentages tend to rise in the playoffs most guys will end up with better stats compared to the regular season average.

Just double-checked Darren Pang, since he's another guy who had almost his entire playoff career in one postseason, and it just so happens that was one of the few where the average save percentage actually dropped. Pang has -1.0 GSAA in his 1988 line and -1.8 for his playoff career, which supports my theory since the 1987-88 regular season average was .880 while all goalies in the 1988 postseason combined for just .869, making Pang look relatively worse if you judge him by the regular season standard in that specific year.

 

[seventieslord]

Shouldn't be too hard to figure that out, just look at a goalie with only one playoff season. I picked Alex Nedeljkovic, for example. Hockey Reference has him at 0.3 GSAA in the 2021 playoffs, but 3.3 under his career playoff totals. That gives us an implied league average save percentage of .919 for the former and .908 for the latter. Considering that .908 was the regular season league average in 2020-21, it seems pretty clear that the season-by-season playoff GSAA stats are measured against the playoff average, while the career GSAA playoff stats are measured against the regular season average. Different baselines will obviously give different results, and since save percentages tend to rise in the playoffs most guys will end up with better stats compared to the regular season average.

Just double-checked Darren Pang, since he's another guy who had almost his entire playoff career in one postseason, and it just so happens that was one of the few where the average save percentage actually dropped. Pang has -1.0 GSAA in his 1988 line and -1.8 for his playoff career, which supports my theory since the 1987-88 regular season average was .880 while all goalies in the 1988 postseason combined for just .869, making Pang look relatively worse if you judge him by the regular season standard in that specific year.

Well done. It makes one wonder what is the correct way to calculate a stat like GSAA. I've heard the case before that the regular season league average should be used, because the playoff average is more volatile and based on a small sample (and any goalie who goes deep is, himself, producing a relatively large chunk of that small sample). Also, in an environment where save percentages almost always rise in the playoffs, a stat based on the regular season average will credit more goalies with a positive contribution, which intuitively makes more sense to me. But on the other hand, that's what "versus threshold" or "versus replacement" stats are for.

 

[Hockey Outsider]

Well done. It makes one wonder what is the correct way to calculate a stat like GSAA. I've heard the case before that the regular season league average should be used, because the playoff average is more volatile and based on a small sample (and any goalie who goes deep is, himself, producing a relatively large chunk of that small sample). Also, in an environment where save percentages almost always rise in the playoffs, a stat based on the regular season average will credit more goalies with a positive contribution, which intuitively makes more sense to me. But on the other hand, that's what "versus threshold" or "versus replacement" stats are for.

I looked into this about 5 years ago (the difference between calculating adjusted playoff save percentage using the regular season average vs the playoff average as the benchmark).

To quote myself from an old post: "In total, 37 of the 43 goalies experienced no more than a 0.3% change in their save percentage, and more than half (25 of 43) experienced a change of no more than 0.1%." I believe I used a threshold of 1,000 adjusted shots (roughly 35 games).

It's debatable which method is better, but the results should be very similar. Personally I like using the playoff average, simply because that way the sum of goals versus average will equal zero (but maybe that's having the math drive the decision a little too much). The swings are likely to be most significant for a goalie who had relatively limited postseason experience.

 

[ContrarianGoaltender]

Well done. It makes one wonder what is the correct way to calculate a stat like GSAA. I've heard the case before that the regular season league average should be used, because the playoff average is more volatile and based on a small sample (and any goalie who goes deep is, himself, producing a relatively large chunk of that small sample). Also, in an environment where save percentages almost always rise in the playoffs, a stat based on the regular season average will credit more goalies with a positive contribution, which intuitively makes more sense to me. But on the other hand, that's what "versus threshold" or "versus replacement" stats are for.

I think a reasonable case can be made for both sides. When I'm doing some quick and dirty comparison I almost always use the regular season average because it's way easier to look up. However, if I had to choose one to use for an in-depth analysis, I think I'm probably on Team Playoff Average actually.

One of the reasons is that I think it's less era biased. The regular season/playoff differential has not been consistent throughout all of history. Sometimes that is because 16 of 21 teams made the playoffs so the effort level went up massively when the games actually mattered (and so the 1980s had consistently large season/playoff splits in general). Sometimes it is because the regular season stats are skewed by teams pounding on a bunch of bad expansion teams during the regular season (e.g. Patrick Roy's 1993 is unquestionably overvalued by using .885 as an benchmark, even if it very possibly still is the best goalie postseason ever). And sometimes I think the competitive environment is simply different in the playoffs, because of rules enforcement or some other factor (more recently, 2021 was notable for greatly improved defensive success in the playoffs relative to the regular season, which could be related to the refs putting away the whistles to an extreme degree).

My only other possible suggestion is to look at each era as a whole and figure out the average save percentage differential between the regular season and playoffs, and then use that to adjust the regular season average? Although it is a bit more complicated, it would smooth the numbers a bit from season-to-season while still taking into account the playoff environment to some extent. It still probably wouldn't be enough to correctly adjust for things like expansion seasons, but nothing's going to be perfect.

 

[Doctor No]

I haven't implemented this yet on my site (two small kiddos leaves less time for dalliances), but I think you want to adjust for the environment (shot selection is typically more goalie-favorable in the playoffs) but correct for the fact that the average goaltender in the playoffs is better than the average goaltender overall.

What I'm closest to buying into (insert standard save percentage caveats):

Benchmark postseason save percentage = League-average regular season save percentage * (playoff postseason differential)

Where:
(playoff postseason differential) is the weighted (by playoff shots faced) relativity of playoff save percentage to regular season save percentage.

This would correct for the fact that stronger goaltenders disproportionally advance further in the playoffs, but would account for the change in those goaltenders' environments.

On the other hand, this would further reduce the already slim sample size.

 

[Doctor No]

The multi-year averaging suggested by CG could be a nice approach; presumably the secular change in relative playoff environment won't change by too much from year to year. Would be a useful exercise to look at that distribution by year and see how much it fluctuates.