Sportsnet.ca 'Shot Quality Project' (EDIT: Now with results!!)

[deckercky]

Very interesting work. So far the results don't seem to be anything surprising - definitely cross-ice passes and deflections are how a lot of goals get scored, while pretty much any NHL goalie can consistently stop the unscreened shot. But once this study is complete, we can start figuring out how to quantify shot quality, and that is a big step forward for hockey analytics.

I think it will be interesting when you get details on goalies (eg, which goalies do well against which types of situations), as well as team data (which teams allow a lot of cross-ice passes? which teams get a lot of cross-ice passes). That will give information to separate great goalies from great defensive systems, and isolate shot quality for teams.

I wish they had defensive and offensive tips separated, since that could help with analysis on whether a team is better to get in the lane to block shots or clear the lane as much as possible.

 

[Freudian]

Great work. Very interesting stuff that confirms what most of us intuitively understood and illustrates the weakness of the method of weighing every shot attempt the same.

"I plan to use this extra layer of data to isolate why a player like Steven Stamkos can maintain a higher shooting percentage than the norm; contrast where Chris Kunitz receives the puck playing with Crosby compared to without; what type of opportunities Chara can and can’t prevent; if Tuukka Rask sees more clean shots than Ondrej Pavelec, and whether it matters; and, if the Maple Leafs have figured out a cheat code for the Fenwick close success model."

Can't wait for this.

 

[hatterson]

Great work. Very interesting stuff that confirms what most of us intuitively understood and illustrates the weakness of the method of weighing every shot attempt the same.

It doesn't necessarily show a weakness of the method on a large scale.

It doesn't matter if certain shots are harder or easier than others if all goalies face a similar ratio of 'hard' and 'easy' shots.

The only way to show a weakness would be to show that there's a way for a team to sustain higher levels of 'easy' shots (and thus lower levels of 'hard' shots)

 

[Freudian]

It doesn't necessarily show a weakness of the method on a large scale.

It doesn't matter if certain shots are harder or easier than others if all goalies face a similar ratio of 'hard' and 'easy' shots.

The only way to show a weakness would be to show that there's a way for a team to sustain higher levels of 'easy' shots (and thus lower levels of 'hard' shots)

Since the save percentage for clean shots is so stable, wouldn't the fact that we have such differing save percentages for goalies over a season suggest that all goalies doesn't face the same ratio of clean shots?

 

[Bear of Bad News]

Since the save percentage for clean shots is so stable

Is it really that stable, though? I understand that the article's author said as much, but the link:

Doesn't really have a lot of gradation in it. Why did he make the graph go all the way down to a save percentage of 0.500? All that does is make the useful range of information entirely obfuscated.

Anyhow, if I had to eyeball it (and it appears that I'm forced to) then the range of save percentages on "clean shots" varies between 0.940 and 0.980. That doesn't seem like particularly stable to me.

 

[billybudd]

if Tuukka Rask sees more clean shots than Ondrej Pavelec

This is what I'm really interested in seeing his results for. Heat maps using distance as a proxy for shot quality already very probably show that Rask does, but I'd like to see what a better model of quality shots has to say about it.

 

[billybudd]

Is it really that stable, though? I understand that the article's author said as much, but the link:

Doesn't really have a lot of gradation in it. Why did he make the graph go all the way down to a save percentage of 0.500? All that does is make the useful range of information entirely obfuscated.

Anyhow, if I had to eyeball it (and it appears that I'm forced to) then the range of save percentages on "clean shots" varies between 0.940 and 0.980. That doesn't seem like particularly stable to me.

One of the datapoints for deflections was that low.

Guess is he wanted to standardize the various graphs to show how much more variance there was for (shot type X) in comparison to "clean shots."

 

[Bear of Bad News]

Ah, that makes more sense - thanks!

If anyone's got better eyes than I do (even for a goalie, my eyesight's pretty bad) and wants to eyeball the range of "clean shot" save percentages, I'll defer to you.

 

[Delicious Dangles*]

Ah, that makes more sense - thanks!

If anyone's got better eyes than I do (even for a goalie, my eyesight's pretty bad) and wants to eyeball the range of "clean shot" save percentages, I'll defer to you.

Looks like 23 dots fall in the 0.945 to 0.965 range, with two outliers at 0.970 and 0.980, assuming the points are the centers of the squares.

Which is a very stable range, especially comparatively.

 

[Bear of Bad News]

Looks like 23 dots fall in the 0.945 to 0.965 range, with two outliers at 0.970 and 0.980, assuming the points are the centers of the squares.

Which is a very stable range, especially comparatively.

Obviously comparatively (and I would expect a larger range of save percentages on tougher shots). By itself, I don't agree.

Taking your numbers at face value, that's an error range of 3.5% to 5.5%, with the worse goaltenders making 57% more errors than the best goaltenders.

Considering that they're all purportedly "easy" shots, I think that the difference between goaltenders here is meaningful.

 

[hatterson]

Since the save percentage for clean shots is so stable, wouldn't the fact that we have such differing save percentages for goalies over a season suggest that all goalies doesn't face the same ratio of clean shots?

I'll address Taco's response below, but just a couple notes here:

On ratio of clean/dirty shots:
We know (based on trivial observation) that all goalies don't face the same ratio of easy and hard shots in a single game, so it's at least possible that they don't on a season level. Either way, the real question when we're looking at projecting isn't whether goalies *did* face more/less clean shots, it's whether teams have a control over this in the future.

The question of whether they did or not is of interest if we're comparing past seasons, or perhaps comparing different goalies for awards in a given season (IE, is a .930 with 95% clean shots more impressive than a .925 with 85% clean shots?), but it doesn't tell us much about the future. Just because Bernier faced a high number of clean shots in one season doesn't mean he will in the next, *UNLESS* we can show that a team has real control over shot quality against. That's the real question that needs to be addressed for this to have predictive power.

On the differing save percentage:
The article states the best-worst difference is 38 goals over 1,000 shots or .038 save percentage. Even assuming that all goalies face an identical ration of 85% clean shots that .038 difference can still result in .0323 save percentage difference over a season. That's the difference between .905 (guy everyone wants to run out of town) and .9373 (vezina favorite).

The variations we see in current goaltender performance seem to be well within the ranges even if shot quality turn out to not be a thing on the aggregate level.

Ah, that makes more sense - thanks!

If anyone's got better eyes than I do (even for a goalie, my eyesight's pretty bad) and wants to eyeball the range of "clean shot" save percentages, I'll defer to you.

Based on zooming in to the pixels it appears that each pixel is worth .00227(27) in save percentage. Unfortunately the dots are 4-6 pixels high so we can't get exact, but I took a decent guess.

It appears the numbers fluctuate from a low of .9409 to a high of .9795 so it appears your eyes are pretty good. Although that would mean the average was .954228 rather than the stated .951 in the article. However the difference between high and low is .0368 or 36.8 goals per 1000 shots, which is very close to the 36 stated in the article.

Either way, the numbers we can extrapolate from the graph seem close enough to reality to draw some conclusions.

First, I completely agree that a variation between .9409 and .9795 doesn't seem very stable. The goalie at .9795 is nearly three times as good as the goalie at .9409. 38 goals per 1,000 shots may not be a big difference if the range is 580-618 out of 1,000, but it's pretty significant when the range is 941-979 out of 1,000.

To extrapolate that difference into something a little more familiar lets assume 30 shots per game with 25 being clean, 2 being transition 2 being rebound and 1 being being deflection (That's a rough average of what 30 shots would turn out to based on the percentages in his article, it's not ideal, but it gets the point across)

Goalie A is at the top end with a .9795 save percentage on those and Goalie B is at the bottom end with a .9409. For all other goals the goalies are perfectly identical, each stopping an exactly average amount of shots. This means a .693 save percentage in transition, .760 on rebounds and .736 on deflections.

Now each goalie will let in an average of 2 * (1-.693) = .614 transition goals, an average of 2 * (1-.760) = .48 rebound goals and 1 * (1-.736) = .264 deflection goals for a total of 1.358 goals against per game.

Now for clean shots. Goalie A will allow an average of 25 * (1-.9795) = .5125 goals. Goalie B will allow an average of 25 * (1-.9409) = 1.4775 goals.

That would bring their total stats to

Goalie A: .938 save percentage, 1.8705 GAA
Goalie B: .905 save percentage, 2.8344 GAA

That's a *massive* difference which means I'd strongly disagree with the idea that the clean save percentage numbers aren't that big a deal.

Edit: And to further show the importance. Being at the low end vs. high end of either of the other 3 categories only makes the GAA move +/- .25 or so. Certainly not minuscule, but if you could choose between Best at clean worst at everything else, and worst at clean best at everything else, you're better off with the first option by a significant margin (roughly .924/2.28 vs .910/2.70)

 

[Freudian]

Just because Bernier faced a high number of clean shots in one season doesn't mean he will in the next, *UNLESS* we can show that a team has real control over shot quality against. That's the real question that needs to be addressed for this to have predictive power.

Of course. We shouldn't assume anything either way. There seems to be a blanket assumption in the advanced stat world that a shot attempt against is worth the same no matter what.

If it's shown that teams have shot type distribution that differs from other teams (both for and/or against) and it is shown that different shot types have varying probability of resulting in goals (which this study seems to show), the basic corsi/fenwick assumption that all shot attempts have equal worth should be thrown to the side. This is even if it's only with one seasons worth of empiric data to show this because the evidence I've seen from the advanced stat crowd that shot quality doesn't matter is much weaker than that.

 

[hatterson]

If it's shown that teams have shot type distribution that differs from other teams (both for and/or against) and it is shown that different shot types have varying probability of resulting in goals (which this study seems to show), the basic corsi/fenwick assumption that all shot attempts have equal worth should be thrown to the side. This is even if it's only with one seasons worth of empiric data to show this because the evidence I've seen from the advanced stat crowd that shot quality doesn't matter is much weaker than that.

I would add an additional criteria.

As you said we must show:

1.) Teams have different shot quality on a season by season basis
2.) Shot quality matters (seems trivially obvious)

But, if 1 is true, we must also show:

3.) Teams can *control* their shot quality for or against.

The reason that I add 3 is because if teams cannot control their shot quality for or against, then their past results do not predict their future results and thus we should treat all teams the same when moving forward.

If 1 is true but 3 is not, shot quality is only useful to explain *why* teams might be doing better than but not predict if they can keep doing it.

 

[Freudian]

I would add an additional criteria.

As you said we must show:

1.) Teams have different shot quality on a season by season basis
2.) Shot quality matters (seems trivially obvious)

But, if 1 is true, we must also show:

3.) Teams can *control* their shot quality for or against.

The reason that I add 3 is because if teams cannot control their shot quality for or against, then their past results do not predict their future results and thus we should treat all teams the same when moving forward.

If 1 is true but 3 is not, shot quality is only useful to explain *why* teams might be doing better than but not predict if they can keep doing it.

It might be hard to isolate the effects of player quality/performance from the effects of the system. Does Oilers allow lots of chances against in transition because of their system or because their players make mistakes (turnovers, ill timed pinches).

All hockey analysis seem to have a big "it depends" element to it. I don't mind. I like "it depends".

 

[Jyrki]

next article on this has been posted

http://www.sportsnet.ca/hockey/introducing-the-shot-quality-project-part-ii/

I'm surprised at how successful transition plays appear to be - given it results in a shot on goal, of course. He didn't include no-pass plays though, which I imagine would end up mangled with the "clean shots" sample.

 

[hatterson]

It might be hard to isolate the effects of player quality/performance from the effects of the system. Does Oilers allow lots of chances against in transition because of their system or because their players make mistakes (turnovers, ill timed pinches).

All hockey analysis seem to have a big "it depends" element to it. I don't mind. I like "it depends".

I think you'd be able to compare team-wide data with individual player on/off ice data to see if it's a great player (is Chara really amazing at preventing hard shots?) or if it's a team system (did the Leafs really "break" fenwick close?)

However, you're right, there's always going to be grey area. We're never going to be able to perfectly project what happens when two sets of 20 hockey players go up against each other, either in a single game or a series. At least until we stop having hockey played by humans and start having it played by robots

I'm surprised at how successful transition plays appear to be - given it results in a shot on goal, of course. He didn't include no-pass plays though, which I imagine would end up mangled with the "clean shots" sample.

It also doesn't include plays where the pass is broken up or missed by the receiver.

Cross ice passes in the slot are incredibly successful, but also fairly hard to get. Probably their danger is one of the reasons they're hard to get, defenses know that you should let a pass float across the crease to Stamkos if you don't enjoy slowly skating back to center-ice for a faceoff.

 

[sharks9]

Cool graph showing the difference in shot location between Subban and Markov

 

[Thesensation19*]

It looks awesome as i skipped through it. Plan on reading more about it but I have been discussing this for years even on here...

Ive mentioned it a bunch of times as people were doubting Lundqvist and talking about AV system.

All I proclaimed was that its easy to score goals and diminish a goalies stats when the opponents are given a lot of High Quality Shots (HQS)

And that so far, AV and the Rangers have allowed plenty of those. The worst part so far is that they lose most games due to a high difference in quality shots.

Same applies when Fluery is being made fun or Byrs in the playoffs... why do they suck then or suck consistently. The team in front of them is allowing high quality of shots. A goalie is bound to let one in on that

 

[billybudd]

It looks awesome as i skipped through it. Plan on reading more about it but I have been discussing this for years even on here...

Ive mentioned it a bunch of times as people were doubting Lundqvist and talking about AV system.

All I proclaimed was that its easy to score goals and diminish a goalies stats when the opponents are given a lot of High Quality Shots (HQS)

And that so far, AV and the Rangers have allowed plenty of those. The worst part so far is that they lose most games due to a high difference in quality shots.

Same applies when Fluery is being made fun or Byrs in the playoffs... why do they suck then or suck consistently. The team in front of them is allowing high quality of shots. A goalie is bound to let one in on that

I'd rewatch those Pittsburgh series. Fleury was not letting in "high quality shots," unless those include weak wristers from the wall and shots from below the icing line in the corner. His playoff reputation is well, well earned.

 

[glucker]

There's no such thing as shot quality. All shots are equal.
-corsicans

 

[Bear of Bad News]

There's no such thing as shot quality. All shots are equal.
-corsicans

That's cute to give a name to a group of people that you disagree with.

No one who looks at CORSI in any serious fashion looks at it in isolation.

 

[hatterson]

The latest article was posted yesterday afternoon

http://www.sportsnet.ca/hockey/shot-quality-reveals-rasks-true-value/

The article compares Price and Rask and their workload.

One of the key things I see should be a surprise to no one. Boston's defense is really good at limiting quality scoring chances.

The difference is exemplified in the next graphic where all common shots are removed, leaving only the difference in workload

Obviously the quality of shots Carey Price has faced is very significantly higher than the quality of shots Tukka Rask as faced over the timeframe measured.

Also an interesting bit at the end

If it is just noise, we must acknowledge that Tuukka Rask and his career .929 save percentage are on track to become—by that statistic—the greatest goaltender in the history of the game. If Rask can maintain even a league average save percentage his next 100 games, he will bump Dominik Hasek from the top spot. Is he better than Dominik Hasek?

Maybe. Or maybe shot quality actually exists.

The project continues.

 

[Hammer Time]

The Bruins have long been a low shot-quality against team. I think I remember from 2010 there was a study on shot distance, and it concluded that Tim Thomas was overrated (i.e. he wasn't the clear-cut best goalie in the league, just one of the best).

The Canucks outshot the Bruins in the 2011 finals, but scoring chance data ("scoring chance" being defined as within the slot area) shows the Bruins significantly outchancing the Canucks. At the time I thought the shot quality difference was mainly due to score effects, but I always suspected the Bruins were inherently giving up lower-quality shots.

This just provides more evidence to confirm there is a trend. I mean, Bergeron, Chara, and Julien on the same team would make you suspect low shot quality, no?

There is a bit of a false argument at the end though. The increase in save percentage (Rask>Hasek!) isn't just due to shot quality, it's also due to a drastic decline in power plays in recent years.

 

[PensFreak]

It would be awesome if we could see the development of a goaltender's quality rating based on the shots they save and don't. Transition shots don't hurt the rating much, as they're difficult to stop. Clean shots, on the other hand, would be very detrimental to it. Similar to a QB rating type of stat. Interesting idea I think.

 

[Wesleyy]

I'm not sure how the author of the article is coming up with the rebound and transition shot data. The league, as far as I know (and I've researched this quite a bit), does not record anything like that. The only way to mine that data is to watch all of the footage and chart that data on your own (or have the money to outsource it), which I doubt the author did. The analysis and conclusion makes sense but without information on how the data is collected I'm reluctant to believe the conclusions drawn in the series of article.