Goaltender Game-to-game Consistency

[Doctor No]

I certainly hope so! The main reason that I developed this (and strength of schedule) is as inputs to my predictive model.

So far, I'm getting a decent amount of lift from both (or rather, from each).

 

[hatterson]

It will be interesting to see how the Shot Quality project from sportsnet turns out, more specifically the enhanced data around a shot.

Clearly Reimer stopping 6 shots on one Washington powerplay is significantly less impressive when those shots are unscreened from the point by Carlson instead of one-timers from the slot by Ovechkin.

Reimer's game was great on Saturday, but it certainly wasn't a '3 deviation above average' all time great performance, despite stopping 49 of 50 shots against an elite offense. It would be interesting to incorporate shot quality data into expected save percentage, although we're talking about a whole nother level of tracking akin to the stuff the NBA is starting to use.


Regarding trade and contract value mentioned earlier, similar information would help that as well. Is Tuukka Rask really an elite, MVP caliber goalie worthy to be the highest paid at his position? Or is it possible he "just" very good and playing in front of a defense that specializes in not allowing hard shots to get through?

 

[Doctor No]

That's a really good point, and a key thing to note - everything in this analysis is predicated upon save percentage, and therefore carries with it all of the advantages and disadvantages of that metric.

The approach that I describe could be applied to any number of adjusted save percentage methods, and it's something that I would like to look at in the future.

Ultimately, the biggest advantage of (raw) save percentage in these calculations is that it's available to some degree for many years in the past (although I've been spending hours reconciling discrepancies in my 1982-83 and 1983-84 game logs, so I guess it's relative. ).

One broad brush approach to estimating shot quality for past seasons would to be look at the number of power plays and number of times shorthanded (both of which are somewhat available for these seasons), and develop a proxy adjustment based upon that.

Of course, doing it at the detailed level would be better yet.

 

[LTIR Trickery]

I'm really curious to see how the limited shot attempts like someone mentioned affects goaltender performance negatively.

 

[Doctor No]

I'm really curious to see how the limited shot attempts like someone mentioned affects goaltender performance negatively.

Tell me more about what you're looking to see (since I'm probably interested in it as well).

Are you talking about genuine low levels of activity for goaltenders, or the under-reporting of (true) shots on goal in certain rinks (or something else entirely)?

 

[Doctor No]

I've got the 2013-14 goaltender variation metrics up on the site now - these will wiggle slightly as the overall shooting percentages twist a bit in the postseason.

For now, the variation metrics for Vezina contenders (I just took the six who were 40 or more goals better than replacement level this year).

|Variation|Bel Avg|Average|Ab Avg
Semyon Varlamov |0.91|17%|35%|48%
Tuukka Rask |1.01|20%|23%|57%
Carey Price |0.95|23%|26%|51%
Ben Bishop |0.92|19%|37%|45%
Jonathan Bernier |0.87|19%|38%|43%
Sergei Bobrovsky |0.94|22%|42%|36%

A variation of 1.0 is about league average; lower means more consistent. It should be noted that you can be "consistent" without being necessarily being good; this is a measure of how predictable a goaltender's performance is.

Average performances are the percentage of time that a goaltender performed within 0.5 standard deviations of what they were "expected" to do (given the opponent's shooting percentage). Below average and above average follow from that.

 

[Doctor No]

Here's a different way of looking at goaltender variance (something that I'd toyed with in the past):

http://www.sportingnews.com/nhl/sto...ff-stats-elite-henrik-lundqvist-rangers-kings

This article compares rolling 7-game save percentage samples between Quick and Lundqvist. The author's conclusion is that Quick's consistency isn't enough (although he acknowledges that now is Quick's chance).

I do like the concept presented, although I prefer my own method (described above) for two reasons - it doesn't account for the fact that a goaltender might have a tougher stretch at one point of the year than another (where you would expect him to have a lower save percentage despite playing just as well), and it doesn't allow for easy cross-generation comparisons.

My metrics:
http://hockeygoalies.org/bio/quick.html
http://hockeygoalies.org/bio/lundqvist.html

Have Quick's variation at 0.93 this past regular season (so about 7% more consistent than random), and Lundqvist's at 0.95 this past regular season. On the other hand, this was Lundqvist's least consistent season since 2008-09, where Quick's variability is on the low side compared to his career.

These numbers will move a bit, although not much, once I update them at the end of the regular season (since I use a composite [regular season plus playoffs] non-empty net shooting percentage to determine a goaltender's "expected" number of saves in any given game). I'll also be updating my two strength of schedule metrics at the same time.

 

[Doctor No]

I've finished the 2014-15 regular season update for the NHL, so am posting some interesting numbers (all totals, including goal support, goaltender variation, hot/cold, are on my site including the links below).

Most consistent game-to-game (min. 500 shots faced):

• Anders Lindback (BUF) 0.61
• Curtis McElhinney 0.75
• Thomas Greiss 0.78
• Craig Anderson 0.79
• Devan Dubnyk (MIN) 0.82
  

Least consistent game-to-game (min. 500 shots faced):

• Ryan Miller 1.21
• Darcy Kuemper 1.18
• James Reimer 1.17
• Petr Mrazek 1.15
• Ben Scrivens 1.14

All totals are separated by team (so Dubnyk gets totals for the Coyotes and for the Wild separately).

 

[Doctor No]

I've updated the website for 1975-76, 1976-77, and 1977-78. Will dig into interesting tidbits next.

 

[Doctor No]

1974-75 NHL totals are now up on the website.

Among goalies with 25 or more games played, Ken Dryden (54%) and Bernie Parent (53%) were second and third in terms of percentage of games with an above-average performance. Who was first (with 66%)?

On the other end of the scale, Michel Belhumeur (58%) led the league in below-average performances.

Dan Bouchard (0.81) was the most consistent regular goaltender. Pete LoPresti (1.28) was the least consistent.

 

[Doctor No]

In the 1975 playoffs, Glenn Resch (69%) and Bill Smith (44%) were one/two in terms of above-average performances.

Tony Esposito (55%) led in terms of below average performances (although check out his strength of schedule in the other thread).

Roger Crozier (0.59) was most consistent, and Glenn Resch (1.20) was least consistent (this last part seems to contradict the first part, although I traced it through, and his span of performances is a bit wild).

 

[charlie1]

I'm really curious to see how the limited shot attempts like someone mentioned affects goaltender performance negatively.

Tell me more about what you're looking to see (since I'm probably interested in it as well).

Are you talking about genuine low levels of activity for goaltenders, or the under-reporting of (true) shots on goal in certain rinks (or something else entirely)?

I know this is old but I am also interested in this. I have seen conflicting studies on the effect of shot volume on goaltender save %. Here is one which supports the idea that more shots lead to a higher save % (from the Shot Quality Project in 2014):

http://www.sportsnet.ca/hockey/nhl/nhl-goalies-better-with-high-shot-volumes/

Do you have anything to add to this, Dr No?

 

[Doctor No]

This thread:
http://hfboards.mandatory.com/showthread.php?t=1632645

Contains the work that I've published on the matter (as well as the work of some others).

My conclusion was that, after adjusting for the strength of the opponent, the effect wasn't significant (and can probably be explained by score effects).

 

[Doctor No]

A poster in another thread asked what I think is a very interesting question (I usually define interesting as "I don't know the answer", and very interesting as "I don't know the answer and I'd never thought of the question"):

I have to wonder how much of this is shot total based. In a single game, lower shot totals make the margin for error smaller, but these would correct themselves over the larger sample size.

So what I decided to do was run some simulations to see if there was an effect there - for goaltenders who face fewer shots/game, is their consistency metric impaired (do they appear more variable by this metric). Then if so, is this something that evens out as the number of games increases.

I looked at hypothetical league average goaltenders in an environment where the league average save percentage is 91%. I then simulated the same number of games (with shots following a Poisson distribution where the expected value is the goaltender's average shots/game, and saves following a binomial).

Interlude: for anyone trying to do this and are frustrated that there isn't an inverse Poisson distribution, recall that the binomial distribution approaches a Poisson, and so the following will produce a random sampling of shots faced per game, where the theoretical mean is in cell $C$2: =BINOM.INV(1000000,$C$2/1000000,RAND())

I ran 10,000 seasons under each situation, and then looked at the mean consistency score, as well as 5th/25th/50th/75th/95th percentiles of the score.

My conclusions (with an asterisk) are this:

• When the number of games a goaltender plays increases, their distribution of expected consistency scores decreases (you will see fewer consistency outliers the greater the number of games played) - this is something that's obvious when viewed empirically.
• At a given number of games played, the mean and median expected consistency scores are the same for any number of shots/game.
• At a given number of games played, the distribution of expected consistency scores is *very slightly* tighter for a goaltender facing more shots/game. This looks to be so slight that it would be impossible to verify empirically.
• The above effect appears to be mitigated further once the sample of games increases.

The asterisk will come in the next post.

Here are the simulation results.

For goaltenders playing five games in a season:

Shots/Game|5th|25th|50th|75th|95th|Mean
20|0.37|0.60|0.80|1.02|1.40|0.83
22|0.37|0.60|0.81|1.03|1.41|0.83
24|0.36|0.60|0.80|1.03|1.38|0.83
26|0.37|0.60|0.80|1.03|1.40|0.83
28|0.37|0.61|0.81|1.04|1.40|0.84
30|0.37|0.61|0.81|1.03|1.39|0.84
32|0.38|0.61|0.81|1.03|1.39|0.84
34|0.37|0.61|0.81|1.03|1.39|0.84
36|0.37|0.62|0.81|1.03|1.38|0.84
38|0.38|0.62|0.82|1.04|1.39|0.84
40|0.37|0.61|0.81|1.02|1.38|0.83

For goaltenders playing 20 games in a season:

Shots/Game|5th|25th|50th|75th|95th|Mean
20|0.70|0.84|0.95|1.07|1.25|0.96
22|0.70|0.84|0.95|1.07|1.25|0.96
24|0.70|0.84|0.95|1.06|1.24|0.96
26|0.70|0.84|0.95|1.07|1.24|0.96
28|0.70|0.85|0.95|1.07|1.25|0.96
30|0.70|0.85|0.96|1.07|1.24|0.96
32|0.70|0.85|0.95|1.07|1.24|0.96
34|0.71|0.84|0.95|1.06|1.24|0.96
36|0.70|0.84|0.96|1.07|1.24|0.96
38|0.71|0.85|0.95|1.06|1.23|0.96
40|0.70|0.85|0.95|1.07|1.25|0.96

For goaltenders playing 40 games in a season:

Shots/Game|5th|25th|50th|75th|95th|Mean
20|0.79|0.90|0.98|1.06|1.19|0.98
22|0.79|0.90|0.97|1.06|1.19|0.98
24|0.79|0.90|0.98|1.06|1.18|0.98
26|0.79|0.90|0.98|1.06|1.18|0.98
28|0.80|0.90|0.98|1.06|1.18|0.98
30|0.79|0.90|0.97|1.06|1.18|0.98
32|0.80|0.90|0.98|1.06|1.18|0.98
34|0.80|0.90|0.98|1.06|1.18|0.98
36|0.80|0.90|0.98|1.06|1.18|0.98
38|0.79|0.90|0.98|1.06|1.17|0.98
40|0.80|0.90|0.98|1.06|1.18|0.98

For goaltenders playing 60 games in a season:

Shots/Game|5th|25th|50th|75th|95th|Mean
20|0.83|0.92|0.98|1.05|1.15|0.99
22|0.83|0.92|0.98|1.05|1.15|0.99
24|0.83|0.92|0.98|1.05|1.15|0.99
26|0.83|0.92|0.98|1.05|1.16|0.99
28|0.84|0.92|0.98|1.05|1.15|0.99
30|0.83|0.92|0.98|1.05|1.15|0.99
32|0.83|0.92|0.99|1.05|1.15|0.99
34|0.83|0.92|0.98|1.05|1.15|0.99
36|0.83|0.92|0.98|1.05|1.15|0.98
38|0.83|0.92|0.98|1.05|1.15|0.99
40|0.84|0.92|0.98|1.05|1.15|0.99

 

[Doctor No]

Okay, here's the asterisk - the above analysis assumes "all shots are equal".

However, different shot totals can either be genuine (goaltender A faces more shots than goaltender B), or an artifact of shot counters (goaltender A and goaltender B faces the same number of shots, but scorekeeper A recorders fewer shots on goal).

In this second case, it's typically shots going wide that some scorekeepers count as shots, and others do not - shots going wide cannot go into the net (unless the goaltender does something to it). So in this case, consider these two goaltenders:

Goaltender A faces 25 "real" shots and 5 shots going wide, and allows three goals. The scorekeeper records 25 shots on net.

Goaltender B faces 25 "real" shots and 5 shots going wide, and allows three goals. The scorekeeper records 30 shots on net.

The effect on save percentage here is (hopefully) obvious - both goaltenders did the same job, but Goaltender A's save percentage is 88% and Goaltender B's save percentage is 90%.

In the effect on this goaltender consistency metric, though, Goaltender B is going to be measured as "more consistent" - and here's why.

Goaltender A faces 25 "shots" and is expected to stop 88% of them under a binomial distribution.

Goaltender B faces 25 "shots" where he is expected to stop 88% of them under a binomial distribution, and an additional 5 "shots" where he will stop 100% of them no matter what.

The second situation inherently presents less variance and less risk to the goaltender.

 

[Doctor No]

So my conclusion is that when you're looking at these numbers, they aren't *directly* influenced by the number of shots faced per game, but you should think about the effect that I describe above - it's likely as important here as it is with save percentage.

Of course, I've also heard that goaltenders like Martin Brodeur also see fewer shots because they stickhandle well, and decrease the number of quality opportunities that opponents face. This effect would artificially *increase* the save percentage and lower the variability (for the same rationale as above).

There are also score effects to consider - goaltenders on good teams will be leading in the third period more often, and will probably see additional low-quality shots as the trailing team makes a desperate attempt to tie.

Overall summary: it's important to think about numbers when looking at them (which should perhaps be something that all who enter this sub-forum appreciate).

 

[ElfanuReinhard*]

Sorry for being a moron, but where did you learn these statistics? Because it's really confusing for a noob like me haha. Either way it's very impressive.

 

[Doctor No]

Thanks! (And you're not a moron - this stuff isn't obvious. And to the extent that it's not obvious, I should be explaining it better.)

This is something that I built myself, when I realized that I wanted to know something that wasn't out there publicly, and that I was trying to do manually (I actually did the first few of these manually).

Since we know a goaltender's save percentage, I wanted to know who was more consistent. For instance, consider two goaltenders each with the same (90%) save percentage, and they each play four games.

Goaltender A does this:
3 goals on 30 shots
3 goals on 30 shots
3 goals on 30 shots
3 goals on 30 shots

Goaltender B does this:
0 goals on 30 shots
0 goals on 30 shots
6 goals on 30 shots
6 goals on 30 shots

Which one would you rather have on your team? Collectively, they "did the same thing" (as measured by save percentage), but the second goaltender gave his team an awesome chance to win two times and essentially no chance to win twice. The first goaltender may have kept his team in all four games (or may not have, depending on the league scoring environment).

Regardless, Goaltender A is clearly more consistent than Goaltender B, and I wanted to measure that.

This number measures how variable (game to game) a goaltender's performance is within a season - so a 1.00 means league average, and a 1.20 means that a goaltender is 20% more variable (20% less consistent) than league average.

In the calculation, I adjust for things that I've figured out how to adjust for (like the varying strength of opponents - if you play a good team and then a bad team, you'd expect a consistent goalie to do better against the bad team), and haven't adjusted for things that I haven't figured out how to adjust for (like general inadequacies of save percentage).

One of the consequences of me building this from scratch is that not a lot of people have studied it - which is both fun and scary, since we're still learning what it's good at (and what it's not good at).