Scouting, Statistics, and St. Louis

Iain Fyffe

Hockey fact-checker
The last thing I'll say about the "eyes" thing is this: if eyes capture everything, why was Martin St. Louis not drafted in 1994 or 1995? Trained eyes, professional eyes, saw him play and decided he was simply too small to play in the NHL. The numbers at least should have said "you really ought to consider this young fellow, what else are you going to use your 8th-round pick on?". The numbers ain't perfect, but the eyes sure ain't either.
 

Canadiens1958

Registered User
Nov 30, 2007
20,020
2,773
Lake Memphremagog, QC.
Incomplete

The last thing I'll say about the "eyes" thing is this: if eyes capture everything, why was Martin St. Louis not drafted in 1994 or 1995? Trained eyes, professional eyes, saw him play and decided he was simply too small to play in the NHL. The numbers at least should have said "you really ought to consider this young fellow, what else are you going to use your 8th-round pick on?". The numbers ain't perfect, but the eyes sure ain't either.

Your analogy is incomplete and does not reflect how scouts recommend players or how teams draft.

Basically scouts fill out an evaluation form for each draft eligible player they see, rating all the skils the team deems important as well as all the other attributes the team wishes to consider. Some players are seen by multiple scouts from a team, some are seen by only one. Each team weighs size differently but it still causes upward or downward movement in the ratings.

Then a master ranking list is prepared which includes team needs. Simply a range from Best Player Available to positional need in the organization.

In closing Martin St.Louis was not drafted because of the numbers which ranked him below the number of draft rounds. He would have had a better chance of being drafted under the old visual system supported by a scout with a solid track record. Like Bobby Lalonde app 25 years earlier.

Edit the NHLCentral Scouting Bureau with evaluation criteria goes back to 1975:

http://en.wikipedia.org/wiki/NHL_Central_Scouting_Bureau
 
Last edited:

Iain Fyffe

Hockey fact-checker
Your analogy is incomplete and does not reflect how scouts recommend players or how teams draft.
I know all this. Pay attention to the claim I was responding to: that eyes are all one needs to properly evaluate players. That claim is clearly false. A purely numerical analysis suggests St. Louis was one of the very best players available in the draft, one which a team should have a very good reason for not drafting at all. Are you saying that it was reasonable that St. Louis was not considered one of the 286 best players available in 1994, and not considered one of the best 234 players available in 1995?

I'm not, of course, suggesting that anyone should have known that there was a scoring title and a Hart trophy in his future. Indeed, the same numerical analysis suggests Eric Perrin would have been nearly as good a pick as St. Louis, and though he had a solid career he was hardly a star. But the fact that 520 players were considered better bets than St. Louis points to one of the glaring flaws in relying on your eyes, even when those eyes belong to professional hockey talent evaluators, especially when dealing with a player that's probably only been seen a few times.

And he's far from an isolated example. Eyes are not the perfect analytical tool that has been suggested.
 

Canadiens1958

Registered User
Nov 30, 2007
20,020
2,773
Lake Memphremagog, QC.
Not Really

I know all this. Pay attention to the claim I was responding to: that eyes are all one needs to properly evaluate players. That claim is clearly false. A purely numerical analysis suggests St. Louis was one of the very best players available in the draft, one which a team should have a very good reason for not drafting at all. Are you saying that it was reasonable that St. Louis was not considered one of the 286 best players available in 1994, and not considered one of the best 234 players available in 1995?

I'm not, of course, suggesting that anyone should have known that there was a scoring title and a Hart trophy in his future. Indeed, the same numerical analysis suggests Eric Perrin would have been nearly as good a pick as St. Louis, and though he had a solid career he was hardly a star. But the fact that 520 players were considered better bets than St. Louis points to one of the glaring flaws in relying on your eyes, even when those eyes belong to professional hockey talent evaluators, especially when dealing with a player that's probably only been seen a few times.

And he's far from an isolated example. Eyes are not the perfect analytical tool that has been suggested.

I would love to see the purely numerical analysis that supports your claim about St. Louis and Perrin in 1994 and 1995 given their pre draft eligible hockey backgrounds.
 

OrrNumber4

Registered User
Jul 25, 2002
15,715
4,975
In defense of advanced statistics, what makes them so useful is the context they provide. There are so many different ways to look at a player statistically; a picture really gets painted.

What I think further gives them support, is the fact that a ranking of best players-of-all-time by Corsi makes a lot of sense. If fits....
 

overpass

Registered User
Jun 7, 2007
5,254
2,736
It's not post hoc in the sense that it uses only St. Louis' numbers from before the 1994 draft in order to determine his ranking. His 94/95 stats are not used, nor are any of his professional stats.

Is the method derived using any post-1994 numbers, or could it have been replicated in 1994?
 

Canadiens1958

Registered User
Nov 30, 2007
20,020
2,773
Lake Memphremagog, QC.
Regardless

It's not post hoc in the sense that it uses only St. Louis' numbers from before the 1994 draft in order to determine his ranking. His 94/95 stats are not used, nor are any of his professional stats.

Regardless the PGVT+ metric was developed well after the the 1994 and 1995 drafts with some of the contributing data generated after 1995 and by the players drafted in 1994 and 1995. The scouts and GMs you criticize did not have the benefit of the metric so it is still up to you to provide a numerical analysis supporting your claim with what was available to the scouts and GMs going into the 1994 and 1995 drafts.
 
Last edited:

alanschu

Registered User
Aug 12, 2005
8,657
912
Edmonton, Alberta
This is not empiricism; there are no experiments. The scientific method cannot apply, because we cannot test hypotheses with repeatable experiments. Instead, it's logical analysis of the available information, combined with an understanding of the limitations of each source of information.

EDIT: I wrote up a huge response but decided to not incinerate the board due to how infuriated this post made me.
 
Last edited:

Canadiens1958

Registered User
Nov 30, 2007
20,020
2,773
Lake Memphremagog, QC.
Watching

You are right but may underestimate the amount of material one needs to watch. It's easy to see that Gretzky was dominant against his peers but accurately estimating the quality of goaltending, lines, pairings, strategies he was facing and doing the same for every player you want to compare him to really requires a lot of effort.


False dilemma. Of course you can get a better idea of a players ability by watching a single game than looking at a single box score. The problem of course is that neither gives you much context. Luckily we usually have more than a a single box score to go by and I'd say that by looking at a players career statistics you can estimate his performance over several games better than by watching a single game. It's very rare to find a game where a player for example scores 0.43 goals and has 0.68 assists.


EDIT: Using both would be preferable. The stats available and what you can get from them give only a rough picture of what happened. Only watching games isn't nearly a perfect way either since most people are really bad at estimating rates at which specific events occur. Probably no one here can tell a 0.43 gpg player from a 0.47 gpg one but the latter will end up scoring more goals in the same amount of games.

EDIT2: If for example we're trying to rank the top 50 goal scorers all time based on peak production (while taking the scoring environment into account) most of them are probably going to be tightly packed together and even small differences can be crucial.

You overestimate the amount of time one has to watch to evaluate a player.The skating ability and hockey sense is evident within 10 shifts.

The flaws that preclude any type of serious pro career surface at most within a game.

You are then left with determining how good the qualified players are, what the obstacles are to their development.

No one cares whether a player will be a 0.43 or 0.47 or x.abc PPG player at the next level. The actual performance will be determined by league and team circumstances. This is also true when evaluating players within a level for potential acquisition. All that matters is the upgrade potential for a specific team circunstances.
 

Canadiens1958

Registered User
Nov 30, 2007
20,020
2,773
Lake Memphremagog, QC.
False

This is not empiricism; there are no experiments. The scientific method cannot apply, because we cannot test hypotheses with repeatable experiments. Instead, it's logical analysis of the available information, combined with an understanding of the limitations of each source of information.

This is simply false.

Scouting is scientific method, testing hypotheses with repeatable experiments.

A player is identified as having NHL potential - the hypotheses. Then he evaluated on the required skills. The the scouts goes to the laboratory to evaluate the player in various repeatable situations = experiments.

More specifically a 17 year old center will be watched and evaluated against different systems within his league, outside his league, internationally. Then the process is repeated later in the season to see how he progressed, if he made the necessary changes in his game, approach. Likewise the evaluations are made on an individual bases. How he plays against older players,same draft year, younger players. How he plays against drafted or non-drafted skaters, goalies, how he plays in various situations,PP,PK, end of game,etc. How he plays against opponents. in specific match-ups - vs LHS/RHS dmen or checks, etc.

On going laboratory experiment situations that are repeatable, recorded, evaluated and applied to the hypotheses.
 
Last edited:

ssh

Registered User
May 22, 2008
94
0
You overestimate the amount of time one has to watch to evaluate a player.The skating ability and hockey sense is evident within 10 shifts.

The flaws that preclude any type of serious pro career surface at most within a game.

You are then left with determining how good the qualified players are, what the obstacles are to their development.

No one cares whether a player will be a 0.43 or 0.47 or x.abc PPG player at the next level. The actual performance will be determined by league and team circumstances. This is also true when evaluating players within a level for potential acquisition. All that matters is the upgrade potential for a specific team circunstances.
I wasn't replying to part of the discussion about projecting prospects abilities. The post was only about comparing NHL players in different circumstances. I'm sure that if a prospect has serious deficiencies in his game a capable observer can probably notice them very quickly.

When we're looking at a large group of top level NHL players there may be some that separate themselves from the pack but as you go down the list you get increasing numbers of players grouped tightly together and the differences can indeed be very marginal. In order to get a better picture you need to watch more than a few games.
 

Canadiens1958

Registered User
Nov 30, 2007
20,020
2,773
Lake Memphremagog, QC.
Not So

I wasn't replying to part of the discussion about projecting prospects abilities. The post was only about comparing NHL players in different circumstances. I'm sure that if a prospect has serious deficiencies in his game a capable observer can probably notice them very quickly.

When we're looking at a large group of top level NHL players there may be some that separate themselves from the pack but as you go down the list you get increasing numbers of players grouped tightly together and the differences can indeed be very marginal. In order to get a better picture you need to watch more than a few games.

Not so. You have to appreciate the depth of the data bank available on each player. An outsider with very limited access to the data bank will require a greater amount of time than an insider who is part of the evaluation process.

Also the focus of the insider is completely different. Yesterday cannot be changed. Use the data available today to optimize the future for the team. It may be an interesting recreational exercise to review yesterday in terms of stats but the only benefit is if something may be learned going forward.
 

Iain Fyffe

Hockey fact-checker
Is the method derived using any post-1994 numbers, or could it have been replicated in 1994?
The method is calibrated using 1989-1999 data. The same method, with presumably slightly different factors, could be developed from data from any set of seasons. Part of the reason I presented this research using seasons before 1989 was for this purpose, to demonstrate that it works outside of those seasons.

Which is to say, St. Louis would likely not have exactly the same rating if different seasons were used. However, a suggestion that he would drop from the very top of the list, to completely off it, is absurd.
 

ssh

Registered User
May 22, 2008
94
0
Not so. You have to appreciate the depth of the data bank available on each player. An outsider with very limited access to the data bank will require a greater amount of time than an insider who is part of the evaluation process.

Also the focus of the insider is completely different. Yesterday cannot be changed. Use the data available today to optimize the future for the team. It may be an interesting recreational exercise to review yesterday in terms of stats but the only benefit is if something may be learned going forward.
OK. We're talking about different things. Scouts and GMs have (had) much more data (both statistics and the amount of games watched) available to them than what we have to work with.

I was only offering thoughts on evaluating players strictly from an outsiders POV, based on the information we have available. Even that may vary quite a bit, especially regarding access to old game material, newspaper articles etc.
 

Iain Fyffe

Hockey fact-checker
Regardless the PGVT+ metric was developed well after the the 1994 and 1995 drafts with some of the contributing data generated after 1995 and by the players drafted in 1994 and 1995.
Apparently I should have used an example from a different season. I picked St. Louis because he's the most obvious example. Build the calculations on any season-set of data, and you'll get pretty similar results. You won't get one set telling you St. Louis is one of the very best players available, and another set telling you there are 520 players better than him.

Another straw man Iain. you don't need to watch every game, you just need to watch a decent sample against top level opponents. One of us is missing the point, its not me.
What is a "decent" sample? Is if the sample you want to gain a "decent" understanding of the player?

If you're interested in every single player, in several different leagues and want to compare their performances during the same or different seasons you really need to watch a lot of hockey.
Precisely. What do you do if you need to know about players in different leagues, or different continents even?

Or you could watch the player you are interested in evaluating with your own two eyes in some game situations
I'm interested in evaluating every player. How do I do that, and avoid gaining only a superficial understanding, or even a mistaken understanding because they few games I watch happen to be among the player's worst of the season?

EDIT: Using both would be preferable. The stats available and what you can get from them give only a rough picture of what happened. Only watching games isn't nearly a perfect way either since most people are really bad at estimating rates at which specific events occur.
Exactly. Use both. I'd never claim that numbers are all you need. But I was responding instead to a claim that eyes are all you need.

EDIT: I wrote up a huge response but decided to not incinerate the board due to how infuriated this post made me.
How so? The scientific method doesn't absolutely require experimentation, I don't think, but it is generally seen as an important part. I mean, this isn't a philosophy of science messageboard here, it's about hockey.

Scouting is scientific method, testing hypotheses with repeatable experiments.
No, it's not. A repeatable experiment is something that someone else can repeat at a later time in another place to duplicate your results. In the lab you isolate what you're trying to study, in an effort to exclude outside factors as much as possible. There is no excluding outside factors in a hockey game.
 

Canadiens1958

Registered User
Nov 30, 2007
20,020
2,773
Lake Memphremagog, QC.
Comparisons

Apparently I should have used an example from a different season. I picked St. Louis because he's the most obvious example. Build the calculations on any season-set of data, and you'll get pretty similar results. You won't get one set telling you St. Louis is one of the very best players available, and another set telling you there are 520 players better than him.



How so? The scientific method doesn't absolutely require experimentation, I don't think, but it is generally seen as an important part. I mean, this isn't a philosophy of science messageboard here, it's about hockey.


No, it's not. A repeatable experiment is something that someone else can repeat at a later time in another place to duplicate your results. In the lab you isolate what you're trying to study, in an effort to exclude outside factors as much as possible. There is no excluding outside factors in a hockey game.

Wouldn't matter since you fail to recognize that using post draft data invalidates your efforts. The NHL entry draft is based on pre draft data. So the valid comparables would be Dale Derkatch, Stan Drulia, Steve Tsijiura, Kevin Reeves, Patrick Lebeau who had similar physical attributes, hockey skills and results leading to their draft year. All five were drafted and had a career history by 1994. Stephan Lebeau and Patrice Lefebvre who were not drafted but had similar physical attributes,hockey skils, leading to their draft year, and results I could provide you with other names.

Sure it is repeatable. Repeats every year for different players and circumstances. The method just gets reinforced even though the subjects change.
 

Iain Fyffe

Hockey fact-checker
Wouldn't matter since you fail to recognize that using post draft data invalidates your efforts. The NHL entry draft is based on pre draft data. So the valid comparables would be Dale Derkatch, Stan Drulia, Steve Tsijiura, Kevin Reeves, Patrick Lebeau who had similar physical attributes, hockey skills and results leading to their draft year.
Say what? Post-draft data is used to determine what the relationship between amateur stats and success at the NHL level is. It's called building a model. The model is intended to be applied to later draft years. Naming a few players (some of whom the system rates highly, and some that it doesn't) isn't helpful to the discussion. The system is built on the numbers of all players from a particular time, not just cherry-picked ones.

All five were drafted and had a career history by 1994. Stephan Lebeau and Patrice Lefebvre who were not drafted but had similar physical attributes,hockey skils, leading to their draft year, and results I could provide you with other names.
Thanks, I have a data set of every draft-eligible player from 1989 to 1999, and then most draft-eligible players from 1983 to 1988, and 2000 to the present (players that would obviously be rated very low were not bothered with).

Are you trying to point out that not all smallish scoring forwards in junior don't pan out at the NHL level? No ****. Shall I list all the big players drafted in the early first round that failed to pan out as well? You'll get nowhere by looking at a few players. You need to look at all of them.

Sure it is repeatable. Repeats every year for different players and circumstances. The method just gets reinforced even though the subjects change.
The fact that it's different players and different circumstances means that it is not a repeatable experiment, in the meaning of the scientific method.

Aside from the obvious straw man here, that is some impressive logic.. that somehow not having shot quality data to give context to your claim based completely on save percentage somehow strengthens it.
Based completely on save percentage...what were you saying about straw men?
 

Canadiens1958

Registered User
Nov 30, 2007
20,020
2,773
Lake Memphremagog, QC.
Quite the Contrary

Say what? Post-draft data is used to determine what the relationship between amateur stats and success at the NHL level is. It's called building a model. The model is intended to be applied to later draft years. Naming a few players (some of whom the system rates highly, and some that it doesn't) isn't helpful to the discussion. The system is built on the numbers of all players from a particular time, not just cherry-picked ones.


Thanks, I have a data set of every draft-eligible player from 1989 to 1999, and then most draft-eligible players from 1983 to 1988, and 2000 to the present (players that would obviously be rated very low were not bothered with).

But we are not discussing building a model. We are discussing the model that was in place in 1994. It seems that you do not have that model or any numerical analysis from 1983 leading to 1994. That smaller players were drafted pre 1994 contributes to the 1994 model. How does the ranking for the listed players compare to Martin St.Louis? Their pre 18 year old numbers were equal to or more impressive than those posted by Martin St. Louis.
 

Iain Fyffe

Hockey fact-checker
But we are not discussing building a model. We are discussing the model that was in place in 1994. It seems that you do not have that model or any numerical analysis from 1983 leading to 1994. That smaller players were drafted pre 1994 contributes to the 1994 model. How does the ranking for the listed players compare to Martin St.Louis?
I'd have to look them up, but I don't think any of them were quite as high as St. Louis. Several of them were quite highly-rated though. And if out of a half-dozen players you get one MVP, another solid NHLer and a few duds, that's a group of players you need to paying a great deal of attention to.

We are most certainly discussing model building, because that's what the system is. It takes certain inputs and runs them through the model to translate them into expected future results.

I suspect you haven't actually read the entirety of what I've written on the subject at Hockey Prospectus. I'm not going to repeat it all here. Your idea that there is no numerical analysis from 1983 to 1994 is just bizarre, considering what I've written over there and in this thread.

Their pre 18 year old numbers were equal to or more impressive than those posted by Martin St. Louis.
Are you sure about that? On a superficial level, perhaps. But the system I'm talking about is essentially a very sophisticated interpretation of how impressive a player's scoring numbers are, at least when discussing forwards. It takes into account a large number of factors in making that interpretation. You of all people should know that a high point total in the QMJHL is not necessarily impressive.
 

Canadiens1958

Registered User
Nov 30, 2007
20,020
2,773
Lake Memphremagog, QC.
Not Quite

We are most certainly discussing model building, because that's what the system is. It takes certain inputs and runs them through the model to translate them into expected future results.

I suspect you haven't actually read the entirety of what I've written on the subject at Hockey Prospectus. I'm not going to repeat it all here. Your idea that there is no numerical analysis from 1983 to 1994 is just bizarre, considering what I've written over there and in this thread.


Are you sure about that? On a superficial level, perhaps. But the system I'm talking about is essentially a very sophisticated interpretation of how impressive a player's scoring numbers are, at least when discussing forwards. It takes into account a large number of factors in making that interpretation. You of all people should know that a high point total in the QMJHL is not necessarily impressive.

No we are discussing the application of a model not the building of an alternative to the system Your model is an attempt at a parallel scouting/drafting tool that is not forward looking.

No ,the point is that you have to produce the data. So that it is strictly your definition of which numerical analysis that you generated is applicable. I want to be dealing with a pure Iain Fyffe
numerical analysis and explanation, not a third or alternative party.

But Martin St.Louis and Eric Perrin never played in the QMJHL. Their midget AAA numbers are interesting but equaled or surpassed by others. Then you have the unique issue that from youth hockey thru the University of Vermont they were together as 2/3rds of a trio. Likewise for a good part of their first IHL season in Cleveland.
 

Bear of Bad News

Your Third or Fourth Favorite HFBoards Admin
Sep 27, 2005
13,341
26,509
No we are discussing the application of a model not the building of an alternative to the system Your model is an attempt at a parallel scouting/drafting tool that is not forward looking.

So is your point that traditional methods at the time didn't identify St. Louis as a future star? (And isn't that obvious?)

Iain's point is that the right things existed to be looked at (they just weren't being used at the time).
 

Ad

Upcoming events

Ad

Ad

-->