Goalie effectiveness formula?

Status
Not open for further replies.

Hockey Outsider

Registered User
Jan 16, 2005
9,056
13,987
BM67 said:
One big problem with most of these formulas is they reward goalies for stopping more shots. Facing fewer shots also brings with it it's own difficulties. Would you rather face a shot after making a flurry of saves a minute ago, or after not seeing a shot for 3-6 minutes?

It's true that there's no way to account for the psychological effects of making a lot of saves vs making fewer saves. However, the data shows over the history of the NHL, and especially this era, the number of shots a goalie faces has no effect on a goalie's ability to make saves.

BM67 said:
That's a minor thing next to this "shot quality" thing. He says himself the data is wonky, then when you add in all the factors that aren't taken into account, it's value is very much in doubt. PP shots are of a "higher" quality, but what about shots just after a penalty expires? What kind of shot is it when a shot bounces off two legs and goes to a wide open player beside the net on the other side from the goalie? Is a player more likely to score if the goalie hasn't made a save in several minutes?

I agree that shot-quality neutral save percentage (SQN Sv%) doesn't (and can't) tell you everything. (The examples of potential problems you mentioned were legitimate, though probably immaterial over the course of a season). But as long as it's an improvement over save percentage, it's more valuable and should be used. Since SQN SV% can tell us things like which teams usually give up shots from far away, which give up more rebounds, which give up more wraparounds, etc., it tells us more than normal save percentage does.

Not sure if you're aware, but I didn't invent shot quality neutral save percentage, Alan Ryder did. So you may want to ask those questions to him.

BM67 said:
Also why if you're going shot by shot, don't you break it down goalie by goalie rather than just lumping them by team?

If Mr. Ryder is calculating the data based on the source I think he's using, there are no records of the goalies involved. Going back and manually determining which goalie was in net for each shot (a total of roughly 68,000 shots) would be a phenomenally time-consuming effort. Though I fully agree with you in principle: the data quality would be improved if it was broken down on a goalie-by-goalie basis.
 

bruins4777*

Guest
Stats dont' tell how good a goalie is, A stat tells how good a stat is. There are cases when stats represent how amazing a goalie is, ex. luongo. But there are cases where it doesn't, ex. brodeur. Typically in the NHL its very hard to judge, but in the AHL it is significantly easier.
 

mudcrutch79

Registered User
Jul 5, 2003
3,903
0
The Big Smoke
www.mc79hockey.com
I'm a believer in save percentage as the evaluative tool for goalies-I attach virtually no weight to things like clutch saves, wins, Cups and GAA. Brodeur is the most overrated goalie of all time, unless he is somehow scaring teams from taking shots, which I doubt.

That said, I have a couple of questions for you. Someone pointed out that the guy who posted the #'s for goalies in low percentage games didn't screen out games where the guy was having a tough night. Thanks, whoever that was. You say though that you've gone back and looked at the data since 1950 to determine that there isn't a relationship between shots and save percentage...how are you making this determination? Can you explain your methodology?

I've looked at that shot neutral stuff, and the problem is, the NHL data is screwed up. I remember looking at some of the data from CHI home games-if it wasn't a slap shot or a back hand, it was coded as a snap shot. One wrist shot all game. I don't trust the data collection, and consequently, I have a hard time trusting the results on that.
 

BM67

Registered User
Mar 5, 2002
4,775
279
In "The System"
Visit site
trentmccleary said:
But facing more shots tires a goalie out. Who could stop a shot better? A fresh goalie or a tired one?

Mental work can be just as tiring as manual work, a goalie still has to stay focused, even when he isn't stopping pucks. The difference in shots faced between Brodeur and Luongo is less than 10 a game. Is 5 shots over average really enough to wear a goalie out? I'd say the difference is more like facing a breakaway at the start of a game after a warmup or just after replacing an injured goalie without a warmup. Which do you prefer?

trentmccleary said:
Shot quality would be a real problem in a smaller sample ... but should be much less of a difference (between goalies) as the sample of games increases.

The samples are still pretty small. Take any goalie or team, take away their worst game of the year. Marty will jump at least 4 spots on the sv% list without his worst game. Take a middle of the pack PK unit, remove that horrid 4/8 night they had and they jump from 14th to 6th in PK%, or something close to that.
 

Hockey Outsider

Registered User
Jan 16, 2005
9,056
13,987
mudcrutch79 said:
That said, I have a couple of questions for you. Someone pointed out that the guy who posted the #'s for goalies in low percentage games didn't screen out games where the guy was having a tough night. Thanks, whoever that was. You say though that you've gone back and looked at the data since 1950 to determine that there isn't a relationship between shots and save percentage...how are you making this determination? Can you explain your methodology?

Sorry if this gets too complicated, but I want it to be thorough...

I'm using the Correlation Coefficient to see if the two variables are related. Basically it's a mathematical formula that calculates how two variables relate to each other. It ranges from -1 to +1. A correlation coefficient of +1 means that the two data sets are perfectly related; if one increases by a given amount, the other will always change by a given amount. A correlation coefficient of around 0.6 means that there's a moderate relationship; there's probably some relationship, though it doesn't explain all the data and there may be other factors to consider. Anything under 0.3 is a weak to nonexistent relationship. +.5, for example, means the two data sets are positively related (they both increase together) while -.5 means the two data sets are negatively related (when one goes up, the other goes down). The correlation coefficient squared ("coefficient of determination") tells us how much one variables explains the behaviour of the other.

So I looked at the correlation coefficient between shots per game and save percentage, for every goalie-season from 1952-2004 (minimum 600 adjusted minutes played). Note: I adjusted save percentage and shots against for era since it has changed so widely over times. After all, it's not fair to compare 33 shots against, .860 sv% goalies in the 80's to modern 27 shots against, .920 sv% goalies today. That could really skew the data.

The correlation coefficient is a miniscule -0.121, which is extremely small. Coefficient of determination is: 1.5%. So knowing the number of shots taken will, on average, only explain less than 2% of the behaviour of save percentage. This number is so small we cannot infer a meaningful relationship from it.

mudcrutch79 said:
I've looked at that shot neutral stuff, and the problem is, the NHL data is screwed up. I remember looking at some of the data from CHI home games-if it wasn't a slap shot or a back hand, it was coded as a snap shot. One wrist shot all game. I don't trust the data collection, and consequently, I have a hard time trusting the results on that.

Definitely, there are certainly flaws in the game data. But the sample size is so huge (around 68,000 shots per year) that I'm not sure how material those errors are.
 

BM67

Registered User
Mar 5, 2002
4,775
279
In "The System"
Visit site
Hockey Outsider said:
It's true that there's no way to account for the psychological effects of making a lot of saves vs making fewer saves. However, the data shows over the history of the NHL, and especially this era, the number of shots a goalie faces has no effect on a goalie's ability to make saves.

Just because you can't measure it doesn't mean it isn't there. Your way of measuring goalies might be the best, and may be very accurate. The fact that you measure workload by shots faced does weigh more in the favor of goalies that face more shots, which, as it says in an article on perseverance at puckerings, is a team stat not a goalie stat, and not what you want to measure.


Hockey Outsider said:
I agree that shot-quality neutral save percentage (SQN Sv%) doesn't (and can't) tell you everything. (The examples of potential problems you mentioned were legitimate, though probably immaterial over the course of a season). But as long as it's an improvement over save percentage, it's more valuable and should be used. Since SQN SV% can tell us things like which teams usually give up shots from far away, which give up more rebounds, which give up more wraparounds, etc., it tells us more than normal save percentage does.

IF it's an improvement. The data is questionable, and therefore the use of it is questionable. Ryder raises doubts himself of the data, and I can see lots of other problems with it, so I doubt very much that it does improve the stats any.

Hockey Outsider said:
Not sure if you're aware, but I didn't invent shot quality neutral save percentage, Alan Ryder did. So you may want to ask those questions to him.

I question your use of them to "improve" your formula. You seem to have less trouble with them than he does, based on what he says in his original article.

Hockey Outsider said:
If Mr. Ryder is calculating the data based on the source I think he's using, there are no records of the goalies involved. Going back and manually determining which goalie was in net for each shot (a total of roughly 68,000 shots) would be a phenomenally time-consuming effort. Though I fully agree with you in principle: the data quality would be improved if it was broken down on a goalie-by-goalie basis.

I'm not familiar with what he's using, but since one goalie plays the entire game most nights, much of the work is done just by noting the goalie of each game as you compile the totals. I'd think breaking the work down by location, team and goalie would be very telling of it's accuracy. By the sounds of it, the data is even worse than those of hits and such that the NHL wanted to drop.
 

Hockey Outsider

Registered User
Jan 16, 2005
9,056
13,987
BM67 said:
Just because you can't measure it doesn't mean it isn't there. Your way of measuring goalies might be the best, and may be very accurate. The fact that you measure workload by shots faced does weigh more in the favor of goalies that face more shots, which, as it says in an article on perseverance at puckerings, is a team stat not a goalie stat, and not what you want to measure.

I certainly agree that just because something can't be measured, it doesn't exist. However, I find it unlikely (though not impossible) that with over 50+ years of data, there's still no indication of any relationship between shots against and save percentage.

I measure workload by shots faced because it varies so much on a team-to-team basis; however I only do this so I can later try to compensate for these effects. I try to eliminate this discrepancy by calculating the number of shots a goalie would have faced on an average team. For example Luongo's team allows about 25% more shots than average. Since I measure goalie performance in terms of "Goals Saved", more shots against means that Luongo got more opportunities to accumulate Goals Saved, by 25%. So I lower his actual Goals Saved by 25%. This compensates for the fact that he had so many opportunities to get GS because he faced so many shots. Conversely I raised Brodeur's score by 10% because the Devils give up so few shots. Fewer shots understate his impact; I am trying to compensate for it. Is this method perfect? No. But it helps remove some of the distortion caused by the number of shots allowed (which as you correctly pointed out, is beyond the goalies control).

BM67 said:
IF it's an improvement. The data is questionable, and therefore the use of it is questionable. Ryder raises doubts himself of the data, and I can see lots of other problems with it, so I doubt very much that it does improve the stats any.

I agree that there are errors, but it still helps sort out some of the discrepancies. If a team routinely allows tip-ins, rebounds, wraparounds and shots from 5 feet out, life will be harder for the goalie (ie Rangers, Blues, Thrashers). If teams force shooters to consistently take low-probability shots or shots from far away (Devils, Wild, Flyers), things will be easier for their goalie. SQN Sv% helps us sort out some of these problems.

BM67 said:
I question your use of them to "improve" your formula. You seem to have less trouble with them than he does, based on what he says in his original article.

I know that Ryder questioned the quality of the data, but it's telling that he ended up using it for his Player Contribution (similar to Win Shares) and so did Iain Fyffe for his version. Ryder's probably more knowledgeable about statistics than anyone on this site and is a quality control specialist. If he's satisfied with the quality of the data, I'm inclined to believe him.

BM67 said:
I'm not familiar with what he's using, but since one goalie plays the entire game most nights, much of the work is done just by noting the goalie of each game as you compile the totals. I'd think breaking the work down by location, team and goalie would be very telling of it's accuracy. By the sounds of it, the data is even worse than those of hits and such that the NHL wanted to drop.

I haven't seen the game-by-game data for hits, etc., so I'm not sure how accurate the shot data is, compared to that. I can send you the shots data file if you're curious.

It records each game by the official NHL games number, 1-1230. If there was some sort of reference that matches up the goalies for each team to the game number, it would make finding the individual goalies stats much easier.

This has been an enjoyable discussion. You raise many good points.
 

BM67

Registered User
Mar 5, 2002
4,775
279
In "The System"
Visit site
Hockey Outsider said:
For example Luongo's team allows about 25% more shots than average.

That reminds me.

Luongo's vs other's shots/60:

04: Luongo 2475/4252= 34.92 Shields 346/732= 28.36 +6.56
03: Luongo 2011/3627= 33.27 Hurme 707/1376= 30.83 +2.44
02: Luongo 1653/3030= 32.73 Flaherty/Kidd 1005/1928= 31.28 +1.45
01: Luongo 1333/2628= 30.43 Kidd 1217/2354= 31.02 -0.59
00: Luongo 730/1292= 33.90 Valiquette/Flaherty/Potvin/Weekes 2003/3674= 32.71 +1.19
Total: Luongo 8202/14829= 33.19 Others 5278/10064= 31.47 +1.72

A little over 5% more shots/60 minutes. Does that mean anything? Is it just the expected variance, or bad rebound control?
 

Hockey Outsider

Registered User
Jan 16, 2005
9,056
13,987
BM67 said:
That reminds me.

Luongo's vs other's shots/60:

04: Luongo 2475/4252= 34.92 Shields 346/732= 28.36 +6.56
03: Luongo 2011/3627= 33.27 Hurme 707/1376= 30.83 +2.44
02: Luongo 1653/3030= 32.73 Flaherty/Kidd 1005/1928= 31.28 +1.45
01: Luongo 1333/2628= 30.43 Kidd 1217/2354= 31.02 -0.59
00: Luongo 730/1292= 33.90 Valiquette/Flaherty/Potvin/Weekes 2003/3674= 32.71 +1.19
Total: Luongo 8202/14829= 33.19 Others 5278/10064= 31.47 +1.72

A little over 5% more shots/60 minutes. Does that mean anything? Is it just the expected variance, or bad rebound control?

This is where breaking the data down on a goalie-by-goalie basis would be really helpful. However I can think of four reasons why Luongo may be consistently facing more shots than his backups:

1) Randomness in the data. The sample size is still fairly small.
2) Luongo gives up a lot of rebounds (relative to his backups).
3) The Panthers have confidence in Luongo, so when he's in net, they open up their play in order to try to score more goals. This results in them allowing more shots. The Panthers are less confident in their backups, so they play more cautious and allow fewer shots.
4) Luongo tends to play against harder opponents (I think), and they tend to shoot more. The backups tend to play against weaker teams that tend to shoot less.

I think all four of these reasons help explain why Luongo faces so many shots. I don't have any data to prove any of them, though I'm inclined to believe it's some combination of all four.
 

BM67

Registered User
Mar 5, 2002
4,775
279
In "The System"
Visit site
Hockey Outsider said:
I haven't seen the game-by-game data for hits, etc., so I'm not sure how accurate the shot data is, compared to that. I can send you the shots data file if you're curious.

I'd be more interested in the shot totals for the years you said you have. What form are they in pdf or excel?
 

Dr Love

Registered User
Mar 22, 2002
20,360
0
Location, Location!
Jason MacIsaac said:
A former poster Degroat made an excellent post how SV% goes up based on higher shot counts.
Which he used only one season, it doens't hold any water.

Also tell me why many goaltenders say they would rather face more shots then 20 or less.....because they are in the game.
And many baseball players will tell you that bunting runners over leads to winning, when in fact it has a negative corrolation to winning.
 

Dr Love

Registered User
Mar 22, 2002
20,360
0
Location, Location!
Hockey Outsider said:
There's a small but growing number of hockey analysts. If you're at all interested in the topic, you should check out:
-- The Hockey Analysis Group (http://sports.groups.yahoo.com/group/HAG_list/)
-- Puckerings (http://www.puckerings.com/)
-- Hockey Analytics (http://members.rogers.com/hockeyanalytics/Research.htm)
-- The Hockey Project (http://members.shaw.ca/hbtn/)
-- The Hockey Outsider (my humble site) (http://www.geocities.com/thehockeyoutsider/)
-- "The Hockey Compendium" by Jeff Z Klein and Karl Eric Reif (available for under $10 in every bookstore I've gone to--can't recommend it enough)
These are all interesting sites, but IMO they all lack strong writing, and thus don't hold the reader's interest. This stuff is very interesting to me, but I get rather bored with it a quarter of the way through. It's not stimulating, it's lecturing. Just my two cents.
 

Greenback

Registered User
Feb 2, 2005
109
0
MB.CA
I would love a copy of that DB Hockey Outsider. I have sent an email.

I will be reading you pdf tonight and hope to contribute to this thread a little later.
 

Hockey Outsider

Registered User
Jan 16, 2005
9,056
13,987
BM67- the data's in Excel format, I can convert it to PDF if you want. Send me an e-mail and I'll send you the file.

Dr Love- I agree that some of the technical stuff can be dull. (Though it's imperative that whatever formulae or methods are being used are explained in detail). Often I find that the sections that show the results of the formula are much more interesting and stimulating. In general "The Hockey Compendium" by Klein and Reif is very interesting and funny, but this often comes at a cost of a less rigorous analysis.

Greenback- I sent you the file. Enjoy. And as always I'd love to hear any feedback.
 

Go Flames Go*

Guest
http://www.goalies.com/site/Stats.pasp?List=all&League=ahl

here is a QB like rating for goalies taking in winning %, save %, and GAA. i don t know the exact formula nor would it mean much. but it just takes 3 imperfect statistics and just compounds them.

I'm a believer in save percentage as the evaluative tool for goalies-I attach virtually no weight to things like clutch saves, wins, Cups and GAA. Brodeur is the most overrated goalie of all time, unless he is somehow scaring teams from taking shots, which I doubt.

at first this statement is bunk, but as the average goalie gets bigger and blocks the net opposed to reacting save % should become more important. a 93 save % means that you place your body in a position where you stop 93 out of a 100 shots. this goalie is better than the goalie who stops 92 shots using the same tactic. thats what the butterfly is: playing the percentages. the better you do this the better of would be on any team.
 

trentmccleary

Registered User
Mar 2, 2002
22,226
1,100
Alfie-Ville
Visit site
BM67 said:
Mental work can be just as tiring as manual work, a goalie still has to stay focused, even when he isn't stopping pucks. The difference in shots faced between Brodeur and Luongo is less than 10 a game. Is 5 shots over average really enough to wear a goalie out? I'd say the difference is more like facing a breakaway at the start of a game after a warmup or just after replacing an injured goalie without a warmup. Which do you prefer?



The samples are still pretty small. Take any goalie or team, take away their worst game of the year. Marty will jump at least 4 spots on the sv% list without his worst game. Take a middle of the pack PK unit, remove that horrid 4/8 night they had and they jump from 14th to 6th in PK%, or something close to that.

The problem here, is that after every possible supposition is made to discredit every statistic... Andre Racicot is the best goalie who ever lived (or at least try disputing that).

Some suppositions could affect the data, but 2% is truly the difference between shots and SV%... then the difference is irrelevant.
I mean, you could cling to every very faint hope that Jason MacIsaac seems to cling to... to prove that Brodeur is the best ever. But the problem is that if those extremely remote possiblies (maybe every shot on Brodeur = Mario break-away & every shot on Hasek = shot from the other teams goalie) are explored... Arturs Irbe could be the best goalie of this generation.
 

Greenback

Registered User
Feb 2, 2005
109
0
MB.CA
Thanks for the DB and sorry for the long delay...

I have a couple of issues with your formula but it does show that much reflection went in to attempting to interpret the stats.

Issue 1:
The comparison of a goalie to the seasonal league average is valid, to a point, but the extension of that comparison to anything beyond the dataset (season) is beyond what the data can infer. For example, the fact that Dryden stands out among goalies in '76 in no way should be interpretted to mean that he would stand out among goalies of the '95. The data shows that Dryden was X goals saved (GS) above the league average in '76. The GS in no way relates to seasons outside the one in question and has little to no meaning outside its own dataset.

Issue 2:
There is no adjustment for rule changes and how that may influence the GS calculation. In the '50's, players ceased to serve a full two minutes for penalties regardless of whether the opposing team scored or not. In the '70's, the blue line face-off dot was moved from the center of the ice, affording the luxury of prime attacking position, to make two dots, one on either side. In the late '80's, teams no longer played four-on-four when coincidental minors were called. The 90's brought the end of tag-up offsides forcing the forechecking team to refrain from pressing the defenders and stacking players in the neutral zone. Every one of these changes affected the way the game was played and how many chances to score were generated. I am sure there are more but this is off the top of my head. The point is that the game being played today is very different than the one being played in years past.

Issue 3:
There is no allotment for the "timing" of a save. The GS formula assumes that all saves must be made. This is not the case. Soft goals allowed in lopsided games for both sides are not that meaningful while spectacular saves made when the team needs it most are invaluable. Grant Fuhr, for example, would allow certain goals in the first two periods but that was in stark contrast to what he could stop in the third period. By stipulating that all goals are equal, we neglect to credit "money" goalies who raise their ability to play when it is needed. If you set a standard of "all goals are equal" as a rating for goalies, then disregard this objection. Also, having re-read the thread, I see that the issue of intagibles has been brought up so I will leave it at that.


I also may have issue with how the formula is applied even within a given dataset, but I have yet to read the SQN essay so I will leave that for now.


All criticisms aside, I do like the way you went out and stated your conclusions and put forth their result with confidence. Not many people would outright state that Sawchuk or Brodeur are overrated, even if they had cooked up a formula to back them up. It was entertaining and made me rethink how I rate goalies and how I, at one time, unsuccessfully tried to brew a formula that would reflect goaltending ability rather than collective team contribution. It was a good read.
 

Greenback

Registered User
Feb 2, 2005
109
0
MB.CA
trentmccleary said:
The problem here, is that after every possible supposition is made to discredit every statistic... Andre Racicot is the best goalie who ever lived (or at least try disputing that).

I can dispute that. I've seen him play. I am sure that there are hundreds of scouts that would agree with me and all without the need for statistical analysis :razz:
 

VanIslander

A 19-year ATDer on HfBoards
Sep 4, 2004
35,134
6,429
stop the puck = effective

do so more than others = good goalie

it's that simple

That's why Hasek was a very good goalie
 

I Hate You All*

Guest
To be a good goalie all you have to do is be able to do the splits.
 

VanIslander

A 19-year ATDer on HfBoards
Sep 4, 2004
35,134
6,429
Staze19 said:
To be a good goalie all you have to do is be able to do the splits.
Uh, maybe in the minors.

In the NHL too many players can lift the puck effortlessly, over the blocker, under the blocker, under the glove hand, over the glove hand, top shelf!
 

Greenback

Registered User
Feb 2, 2005
109
0
MB.CA
Actually, that works until about midget or early junior. After that, players learn what to look for.
 

trentmccleary

Registered User
Mar 2, 2002
22,226
1,100
Alfie-Ville
Visit site
Greenback said:
I can dispute that. I've seen him play. I am sure that there are hundreds of scouts that would agree with me and all without the need for statistical analysis :razz:

Didn't Darren Pang call Hasek the worst goalie ever at one point? (and I don't remember him prefacing it with "worst technical goalie".)
 

Greenback

Registered User
Feb 2, 2005
109
0
MB.CA
and your point is....


Pang, while apparently benefitting from the TV contract signed with the goaltending union guaranteeing all retired goalies can move into the broadcast booth, most likely made those statements before Hasek was given a chance to show his stuff. Plus, Pang, to my knowledge, is not a scout. He is an "analyst".

Racicot's career is over. Nothing left to do but pass judgement.


No idea what your objection is trentmccleary.
 

trentmccleary

Registered User
Mar 2, 2002
22,226
1,100
Alfie-Ville
Visit site
Greenback said:
and your point is....


Pang, while apparently benefitting from the TV contract signed with the goaltending union guaranteeing all retired goalies can move into the broadcast booth, most likely made those statements before Hasek was given a chance to show his stuff. Plus, Pang, to my knowledge, is not a scout. He is an "analyst".

Racicot's career is over. Nothing left to do but pass judgement.


No idea what your objection is trentmccleary.

Pang was an NHL goalie... err, maybe that's debateable... I mean a goalie who played in the NHL. His opinion should have some merit.
In the extreme, without any statistic whatsoever... I might think that Peter Schaefer was one of the top offensive players in the NHL. Schaefer might have the highest "highlight real appearances/ total goals" ratio in hockey right now.
Meanwhile, players like Nash, Ciccerelli, etc... are ugly players. Nearly every goal, a forgettable one. Yet one type of player is far more effective on ice.
 
Status
Not open for further replies.

Ad

Upcoming events

Ad

Ad

-->