Goalie effectiveness formula?

[Human Blender]

I have searched the archive file to no avail (there was some discussion about how to judge goalies stats in the San Jose thread), so if this topic had been covered in the past, please give me a link, and I'd be happy.

I am trying to put together (if at all possible) a unified goalie effectiveness formula (GEF) to be able, based only on statistical data, to find out which goalies are, again, statistically, better than others. I am aware of the Brodeurs, Luongos and Turcos of the world (to name a few), but a mathematical formula would help me divide the subjective from the intersubjective (since, the objective does not exist in judging value of goalies).

I think the most important constituents for such a GEF would be (n no particular order): Save Pct, GAA, Shots per Game, and probably Games Played as the preliminary indicator to exclude marginal and occassional performers. What other criteria would you take into consideration? Is Games Lost any better than Games Won? Actually, the goalie has more impact on games lost than games won, right? How would you weigh any of these criteria against each other? And ultimately, what do you think a GEF should look like?

Food for thought. Thanks for your input.

CEO later
--Nieszpory

Click to read more...

 

[wint]

I think the problem is that people very much disagree on what a good performance is. For instance, who has done better, a goalie who lets in 3 goals on 52 shots or 2 goals on 22 shots? The first goalie has a better save percentage, but many would argue that the second job is harder because it's more difficult to stay in the game mentally without any sustained pressure. In fact, I believe goalie save percentages tend to go up when a goalie sees a greater number of shots in a game.

Another problem is the importance of winning. Clearly that's a team stat, and you don't want to punish the Luongos of the world for having a sub-par team in front of him, but some goalies win a lot more often than their backups, or simply get better when the game is tied or his team is up a goal.

Basically, I don't think you're going to find a way to determine objectively whether one goalie is better than another. But I do have one suggestion that can help you judge goalies fairly based on your own preferences: Do the process in reverse first. Rank the goalies according to your own opinion from 1-12 or so, and then try to come up with a formula that manipulates the statistics of those goalies, such that when you plug in their stats into the formula, you get a similar if not identical list. Then you can apply the formula you came up with to the stats of more guys (or even the same guys in previous years) and see if the ranking you come up with looks about right.

 

[Hockey Outsider]

Nieszpory: I have a database of goalie statistics from 1952-present. Please PM or e-mail me if you would like a copy of this database.

I divised a formula that rates goalie performance, independent of the year they played in, and the team they played for. You may find it interesting, and I welcome comments. It includes data from 1952-present. It's available on my website: http://www.geocities.com/thehockeyoutsider/Goalie1.pdf. The document is fairly long but I believe the discussion is worth it. I go into a detailed discussion of the pro's and con's of "conventional" goalie stats, and how I incorporate some data into one final rating.

For the curious the top 10 goalies ever are:

1 Patrick Roy 1,040 469.1
2 Dominik Hasek 603 384.0
3 Jacques Plante 948 350.8
4 Tony Esposito 908 319.8
5 Ken Dryden 402 314.8
6 Glenn Hall 1,031 296.2
7 Bernie Parent 621 244.6
8 Johnny Bower 620 222.6
9 John Vanbiesbrouck 873 204.4
10 Billy Smith 659 201.2

EDIT: You may also want to check out "Goaltender Rating" by Daryl Shilling (http://members.shaw.ca/hbtn/player_study/goaltender_rating.htm) and "Perseverance Rating" by Jeff Z Klein and Karl Eric Reif.

 

[Jason MacIsaac]

Nieszpory: I have a database of goalie statistics from 1952-present. Please PM or e-mail me if you would like a copy of this database.

I divised a formula that rates goalie performance, independent of the year they played in, and the team they played for. You may find it interesting, and I welcome comments. It includes data from 1952-present. It's available on my website: http://www.geocities.com/thehockeyoutsider/Goalie1.pdf. The document is fairly long but I believe the discussion is worth it. I go into a detailed discussion of the pro's and con's of "conventional" goalie stats, and how I incorporate some data into one final rating.

For the curious the top 10 goalies ever are:

1 Patrick Roy 1,040 469.1
2 Dominik Hasek 603 384.0
3 Jacques Plante 948 350.8
4 Tony Esposito 908 319.8
5 Ken Dryden 402 314.8
6 Glenn Hall 1,031 296.2
7 Bernie Parent 621 244.6
8 Johnny Bower 620 222.6
9 John Vanbiesbrouck 873 204.4
10 Billy Smith 659 201.2

EDIT: You may also want to check out "Goaltender Rating" by Daryl Shilling (http://members.shaw.ca/hbtn/player_study/goaltender_rating.htm) and "Perseverance Rating" by Jeff Z Klein and Karl Eric Reif.

Sorry that Method sucks.....if a goaltender sees alot of shots and plays alot of games he will be seen as the best goaltender that year which is not the case. Winning games, holding leads, clutch saves, goals allowed are not included in that that formula.

I'm also sorry but you math dorks need to get a life and watch a hockey game. The higher the shots the better the sv% usually is.

 

[Hockey Outsider]

Sorry that Method sucks.....if a goaltender sees alot of shots and plays alot of games he will be seen as the best goaltender that year which is not the case.

No offense, but did you even read anything I wrote? Had you done that, you'd see that's not the case at all. I'm really interested where I "said" that a goalie who faces a lot of shots will get rated higher.

A goalie does get credit for playing in more games. Who contributed more to his team- a goalie who plays 10 games or a goalie who plays 70?

Winning games, holding leads, clutch saves, goals allowed are not included in that that formula.

Winning games are extremely dependent on your team. Therefore it doesn't tell us much aboout individual performance. Saying that a goalie on a winning team must be a good goalie is the same thing as saying that an employee on a profitable company must be a competent employee.

"Clutch saves" aren't objectively measured, and "holding leads" is also team dependent.

Goals allowed are included in my formula. Again, did you even read it?

I'm also sorry but you math dorks need to get a life and watch a hockey game.

Why resort to childish insults? If you don't have anything positive to contribute, don't say anything. I asked for comments, not personal attacks, which are completely uncalled for. I have no desire to start trading insults with people I don't know over the net.

For the record: I go to about 20 NHL games per year, watch several more games per week, and play in a recreational league on a weekly basis, so I'm really not sure what that comment refers to.

The higher the shots the better the sv% usually is.

This is completely incorrect and over 50 years of shot-and-save data prove this point. Correlation between shots against per game and save percentage: -0.099. The R^2 value is less than 1%. There is NO relationship between shots and sv%.

Anyway, if you want to make personal attacks against me and criticize things you've obviously never read, go ahead. I won't participate in that. I'm here to talk about hockey and possibly make new friends.

 

[Jason MacIsaac]

No offense, but did you even read anything I wrote? Had you done that, you'd see that's not the case at all. I'm really interested where I "said" that a goalie who faces a lot of shots will get rated higher.

A goalie does get credit for playing in more games. Who contributed more to his team- a goalie who plays 10 games or a goalie who plays 70?



Winning games are extremely dependent on your team. Therefore it doesn't tell us much aboout individual performance. Saying that a goalie on a winning team must be a good goalie is the same thing as saying that an employee on a profitable company must be a competent employee.

"Clutch saves" aren't objectively measured, and "holding leads" is also team dependent.

Goals allowed are included in my formula. Again, did you even read it?



Why resort to childish insults? If you don't have anything positive to contribute, don't say anything. I asked for comments, not personal attacks, which are completely uncalled for. I have no desire to start trading insults with people I don't know over the net.

For the record: I go to about 20 NHL games per year, watch several more games per week, and play in a recreational league on a weekly basis, so I'm really not sure what that comment refers to.



This is completely incorrect and over 50 years of shot-and-save data prove this point. Correlation between shots against per game and save percentage: -0.099. The R^2 value is less than 1%. There is NO relationship between shots and sv%.

Anyway, if you want to make personal attacks against me and criticize things you've obviously never read, go ahead. I won't participate in that. I'm here to talk about hockey and possibly make new friends.

I read both methods and they depend mostly on SV% which is a poor stat. The two worst seasons ever where by an Ottawa early 90's team and a washington team early 70's. That itself shows a flaw.

A former poster Degroat made an excellent post how SV% goes up based on higher shot counts. Also tell me why many goaltenders say they would rather face more shots then 20 or less.....because they are in the game. What a goaltender like Brodeur can do with that little shots is amazing. You also said Stanley Cups are based on team perforence. How can a goaltender like Cechmanek do so well in the regular season then flop in the playoffs with Philly. How can a player like Cujo go from a horrible defense in Toronto with good stats and play behind Detroi and not put up great stats. How can Hasek do the same from Buffalo to Detroit.

I don't care what anyone says....how good a player truely was is shown by his championships. Brady is a winner, Jordan was a winner, Roy was a winner and so is Brodeur. He won 3 stanley cups, 1 world cup (something Roy couldn't do) and an Olympic gold medal. It doesn't have to be fancy, as long as he gets the job done.

 

[mymkovski]

A former poster Degroat made an excellent post how SV% goes up based on higher shot counts.

If you knew anything about Statistics you would realize that Hockey Outsider just proved that there is not a correlation between shots and SV%

 

[Jason MacIsaac]

If you knew anything about Statistics you would realize that Hockey Outsider just proved that there is not a correlation between shots and SV%

If you knew anythnig about hockey you would know that stats mean little in a game where winning is everything.

 

[Hockey Outsider]

I read both methods and they depend mostly on SV% which is a poor stat.

No statistic is perfect. This is especially true in a complex, fluid game like hockey as opposed to, say, baseball, where everything can be broken down into little pieces. But save percentage is the best record of a goalie's performance. After all, a goalie's job is to stop shots. Save percentage measures that. I agree that it doesn't take shot quality into account (it didn't until 2002, now some new research does, which I've included...) but no other statistic does either.

The two worst seasons ever where by an Ottawa early 90's team and a washington team early 70's. That itself shows a flaw.

I understand your point, but I disagree with the conclusion. Based on the work done by Daryl and I, I would come to the conclusion that the mid-70's Capitals and expansion Senators were bad because they got bad goaltending.

A former poster Degroat made an excellent post how SV% goes up based on higher shot counts.

I have not seen this post, so I don't know exactly what arguments he made. However I have a database of every single shot, save and goal from 1952-present. There's less than a 1% correlation between shots against and save percentage. That tells me they're virtually independent of each other.

Also tell me why many goaltenders say they would rather face more shots then 20 or less.....because they are in the game. What a goaltender like Brodeur can do with that little shots is amazing.

I'm a goalie... I'd rather face 0 shots per game! I know that whenever I face a lot of shots I'm "in the zone", but I also get fatigued and my level of play drops. Still, regardless of what I (or any other goalie, amateur to NHL) says, the data clearly shows there's no relation between sv% and shots against.

You also said Stanley Cups are based on team perforence. How can a goaltender like Cechmanek do so well in the regular season then flop in the playoffs with Philly. How can a player like Cujo go from a horrible defense in Toronto with good stats and play behind Detroi and not put up great stats. How can Hasek do the same from Buffalo to Detroit.

It would be pretty boring if every goalie played exactly as well, every season.

Hasek and Cujo left their teams past their primes... that's why their stats got worse. In fact their stats were on a downward trend before they left Buffalo and Toronto, respectively.

The jury's still out on Cechmanek. The guy's only played 23 playoff games in his career and was fabulous in '02 (it's not his fault the Flyers couldn't score any goals that year). He had bad playoff runs in '01 (his rookie season) and '03. I'd hesistate to call a goalie a "playoff choker" on a sample of just 19 games.

I'm not denying that goalies can play a huge role in how well their team does in the playoffs, but there are so many great goalies stuck on terrible teams, and many not-so-great goalies who are lucky enough to be on championship teams. For example Roy should be commended for being such an amazing playoff performer... but if he played just as well, but was stuck on awful teams, he'd still be just as good a goalie.

 

[mymkovski]

If you knew anythnig about hockey you would know that stats mean little in a game where winning is everything.

I entirely agree that stats mean little in a game where winning is everything and I never said otherwise. But even winning is a stat.

I simply stated that Hockey Outsider statistically and mathematically proved that higher shots are not related to a higher save %.

 

[Jason MacIsaac]

I disagree with your first point. A goaltenders job is not really to stop shots....it is to win. If a goaltender can get it done night in and night out letting in 3 goals...all the power to him.

There are intangibles that you can't place in that formula. A player like Luongo is on a team that is usually trailing. There is less pressure on a goaltender down by 2 or 3 then there is with a goaltender who is up by one or two.

Another thing would be types of defense, NJ is known to collaps in front. They do block alot of shots but the ones that get through get deflected or screen Brodeur. Other teams where the defense is not as brave the goaltender may see more shots but the majority of them he can see.

Many things can't be included in the stats, hockey is not like baseball where everything is run by stats. In hockey there are far too many intangibles that can't be added up. So you just have to watch the game and say to yourself who the better goaltender is.

 

[Jason MacIsaac]

I entirely agree that stats mean little in a game where winning is everything and I never said otherwise. But even winning is a stat.

I simply stated that Hockey Outsider statistically and mathematically proved that higher shots are not related to a higher save %.

That point can't be proven over a an average of 50 years. Since the game is changing that means you would ahve to do the same test over the last 5 years. 10 would even be acceptable.

 

[KOVALEV10*]

For me you cant judge which goalie is better simply by GAA and Save percentage. I mean David Aebisher and Marty Turco have better numbers then Theodore for instance but that doesnt make them any better. IMO the true way to compare 2 goalies is to watch them play, see which is making more GREAT saves, look at their team's stats whenever the goalie has been in nets, look at his personal accomplishments (trophies, being called on an internation team, etc.) and judge based on that. That's my opinion at least.

 

[Hockey Outsider]

I disagree with your first point. A goaltenders job is not really to stop shots....it is to win. If a goaltender can get it done night in and night out letting in 3 goals...all the power to him.

I still think that on each team, each player is supposed to do their job... assuming the jobs are allocated fairly, this will help a team win. A goalie stops shots... this helps a team win. A penalty-killer kills penalties... this helps a team win. A goal-scorer scored goals... this helps a team win. I don't think we should evaluate the best penalty killers on the number of games their team wins, we should evaluate them on how well they play defense and kill penalties. Similarly, I don't think we should evaluate goal-scorers or goalies on how much their team wins... otherwise Lemieux's Hart/Art Ross/Pearson/168 point season in '88 was bad because his team missed the playoffs.

I think we will have to agree to disagree on this point... that's perfectly ok, I don't claim to know everything.

There are intangibles that you can't place in that formula. A player like Luongo is on a team that is usually trailing. There is less pressure on a goaltender down by 2 or 3 then there is with a goaltender who is up by one or two.

Yes, I agree that intangibles can't be placed in formula. But looking at an objective record of goalie performance, along with considering other factors like intangibles, is the fairiest way. Using only intangibles is purely subjective, so there's not much point in comparing your list to someone else's, since it would all boil down to opinion.

Another thing would be types of defense, NJ is known to collaps in front. They do block alot of shots but the ones that get through get deflected or screen Brodeur. Other teams where the defense is not as brave the goaltender may see more shots but the majority of them he can see.

I agree that save percentage traditionally hasn't taken shot quality into account (though new research shows that it is possible to quantify shot quality). But I don't understand why this means we should throw out over 50 years of data. We need to take it with a grain of salt and understand it's limitation, but not throw it all away and rely purely on subjectivity.

Many things can't be included in the stats, hockey is not like baseball where everything is run by stats. In hockey there are far too many intangibles that can't be added up. So you just have to watch the game and say to yourself who the better goaltender is.

If I was able to watch every single game played in NHL history, I would do that; but I can't. Stats do have a role. They are the sole, objective record of player performance. Of course intangibles are worth considering too. You need to look at both.

That point can't be proven over a an average of 50 years. Since the game is changing that means you would ahve to do the same test over the last 5 years. 10 would even be acceptable.

Fair enough. The R^2 over the last 5 years is: 0.1%. The R^2 over the last 10 years is 0.0%!! There is absolutely no relationship between shots against and save percentage.

 

[gr8haluschak]

I read both methods and they depend mostly on SV% which is a poor stat. .

How is that a poor stat - IT IS THE ONLY STAT A GOALIE CAN CONTROL ON HIS OWN. Can he control the amount of wins - nope that is team dependent, as is goals against average, he himself can only control the amount of saves he make.

 

[trentmccleary]

A former poster Degroat made an excellent post how SV% goes up based on higher shot counts.

First of all, would you even be fighting this if you didn't believe that every one of these theories was a personal attack on one of your favorite players?

2nd, it's a fairly small sample... but:

Luongo vs. back-up:
00-01 Luongo = [2.44 / .920], Trevor Kidd [3.31 / .893]
01-02 Luongo = [2.77 / .915], Trevor Kidd [3.21 / .895]
02-03 Luongo = [2.71 / .918], Jani Hurme [2.88 / .907]
03-04 Luongo = [2.43 / .931], Steve Shields [3.44 / .879]

Before and after these seasons,
Steve Shields has been declining in SV% for about 6 years now.
Trevor Kidd has had 2 big seasons, but has hovered around .895 - .905 for most of his career.
Jani Hurme put up the same identical SV% he put up in Ottawa the year before.

I didn't do the win %... but believe me, it looks like it's going to to f-ugly.

 

[Jason MacIsaac]

How is that a poor stat - IT IS THE ONLY STAT A GOALIE CAN CONTROL ON HIS OWN. Can he control the amount of wins - nope that is team dependent, as is goals against average, he himself can only control the amount of saves he make.

It measures goals per shots. If there was a goals per scoring chance stat that would be one step in the right direction. Some teams system allows all the shots from the outside but nothing lethal from the inside. That is going to change the stats in a large way.

 

[Fish on The Sand]

I read both methods and they depend mostly on SV% which is a poor stat. The two worst seasons ever where by an Ottawa early 90's team and a washington team early 70's. That itself shows a flaw.

A former poster Degroat made an excellent post how SV% goes up based on higher shot counts. Also tell me why many goaltenders say they would rather face more shots then 20 or less.....because they are in the game. What a goaltender like Brodeur can do with that little shots is amazing. You also said Stanley Cups are based on team perforence. How can a goaltender like Cechmanek do so well in the regular season then flop in the playoffs with Philly. How can a player like Cujo go from a horrible defense in Toronto with good stats and play behind Detroi and not put up great stats. How can Hasek do the same from Buffalo to Detroit.

I don't care what anyone says....how good a player truely was is shown by his championships. Brady is a winner, Jordan was a winner, Roy was a winner and so is Brodeur. He won 3 stanley cups, 1 world cup (something Roy couldn't do) and an Olympic gold medal. It doesn't have to be fancy, as long as he gets the job done.

you lost all your credibility when you said Roy was never able to win the world cup. I mean its pointless, and holds no value. There is very little in the way of international hockey that means squat. Actually strike that. There is absolutly nothing in the way of international hockey that can effectively measure a players career.

 

[Jason MacIsaac]

First of all, would you even be fighting this if you didn't believe that every one of these theories was a personal attack on one of your favorite players?

2nd, it's a fairly small sample... but:

Luongo vs. back-up:
00-01 Luongo = [2.44 / .920], Trevor Kidd [3.31 / .893]
01-02 Luongo = [2.77 / .915], Trevor Kidd [3.21 / .895]
02-03 Luongo = [2.71 / .918], Jani Hurme [2.88 / .907]
03-04 Luongo = [2.43 / .931], Steve Shields [3.44 / .879]

Before and after these seasons,
Steve Shields has been declining in SV% for about 6 years now.
Trevor Kidd has had 2 big seasons, but has hovered around .895 - .905 for most of his career.
Jani Hurme put up the same identical SV% he put up in Ottawa the year before.

I didn't do the win %... but believe me, it looks like it's going to to f-ugly.

Did I say Luongo was a bad goaltender. He is miles ahead of those goaltender. Kidd and Shields are bums and Hurme will be a career backup. It is also clear that Hurme didn't perform as well as he did in Ottawa. He struggled with Florida. It would take more games then that for a goaltender to get used to a new system.

 

[Jason MacIsaac]

you lost all your credibility when you said Roy was never able to win the world cup. I mean its pointless, and holds no value. There is very little in the way of international hockey that means squat. Actually strike that. There is absolutly nothing in the way of international hockey that can effectively measure a players career.

You had no credibility to begin with. I guess what Tretiak accomplished means nothing. Be gone with you. Go drive over the limit around a sharp turn Mario Andretti.

 

[PecaFan]

I have not seen this post, so I don't know exactly what arguments he made. However I have a database of every single shot, save and goal from 1952-present. There's less than a 1% correlation between shots against and save percentage. That tells me they're virtually independent of each other.

Don't worry about it, the whole Degroat thing was terrifically bad work. No sound theory behind it, no scientific method, no raw data presented, etc. Even us normal folks could see massive holes in it.

One obvious flaw I saw was that there was only emphasis on single games. Every goalie who ever got pulled after one or two periods "supported" his theory that low shots meant low save percentages. In fact, in those games, it's the opposite, the low save percentage caused the goalie to get pulled, leading to the low shots. Similarly, teams with a large lead play tight defensively, don't go on offense, and often generate very few shots in the third period, leading to low shot totals. Again, the low shots is a result of the bad goalie performance, not the other way around.

 

[gr8haluschak]

It measures goals per shots. If there was a goals per scoring chance stat that would be one step in the right direction. Some teams system allows all the shots from the outside but nothing lethal from the inside. That is going to change the stats in a large way.

What is the point of your first sentence ? as I said before it is the ONLY stat a goalie on his own controls.

As for your point about shots from the outside: the goalie still has to make saves - it is that simple, a shot on goal is one that is worthy of being a score and what is the goalies job ? to prevent that. As for your argument that scoring chances are more important you just proved yet another stat that a goalie cannot control on his own, thus further backing up what I said: the most important stat for a goalie is the save %

 

[Jason MacIsaac]

Don't worry about it, the whole Degroat thing was terrifically bad work. No sound theory behind it, no scientific method, no raw data presented, etc. Even us normal folks could see massive holes in it.

One obvious flaw I saw was that there was only emphasis on single games. Every goalie who ever got pulled after one or two periods "supported" his theory that low shots meant low save percentages. In fact, in those games, it's the opposite, the low save percentage caused the goalie to get pulled, leading to the low shots. Similarly, teams with a large lead play tight defensively, don't go on offense, and often generate very few shots in the third period, leading to low shot totals. Again, the low shots is a result of the bad goalie performance, not the other way around.

It wasn't done through single games. It was done over a year. Honestly, how many times would a goaltender be pulled over the course of a season and how much would that affect the stats.

 

[Jason MacIsaac]

What is the point of your first sentence ? as I said before it is the ONLY stat a goalie on his own controls.

As for your point about shots from the outside: the goalie still has to make saves - it is that simple, a shot on goal is one that is worthy of being a score and what is the goalies job ? to prevent that. As for your argument that scoring chances are more important you just proved yet another stat that a goalie cannot control on his own, thus further backing up what I said: the most important stat for a goalie is the save %

So a goaltender that can make let in 2 goals on 35 outside shots and 5 danerous shots should be considered better then a goaltender who has let in 2 goals on 10 outside shots and 10 dangeroud shots.

I really don't agree with that theory. Especially when the first goaltender gets to be more in the game and alert then the goaltender who has to stay focused with little action.

 

[trentmccleary]

Did I say Luongo was a bad goaltender. He is miles ahead of those goaltender. Kidd and Shields are bums and Hurme will be a career backup. It is also clear that Hurme didn't perform as well as he did in Ottawa. He struggled with Florida. It would take more games then that for a goaltender to get used to a new system.

I'm not sure that Hurme will ever play again to prove you right or wrong (severe back injury).
But the SV%'s of those 3 goalies didn't change when they moved from/to better systems to Florida. More shots didn't help them look better at all.