Scouting, Statistics, and St. Louis

[Hardyvan123]

Yes I think he could, look at the top scorers on the 86-87 season and look at the first season post-lockout...

I see what you did there picking the absolute advantage in the 2 example seasons.

Okay, just for fun how many points does the 18 year Crosby 06 do in 87?

How about a 20 year old AO?

just a range would be okay too.

 

[Iain Fyffe]

Your model is an attempt at a parallel scouting/drafting tool that is not forward looking.

It is? I thought I was using past data, for which we have before and after, to predict the after for current players, for which we only have the before.

Yes, actually I'm quite sure that's what I was doing, because that's what I was doing.

But Martin St.Louis and Eric Perrin never played in the QMJHL.

So you cannot compare players from different leagues? How do scouts do it then?

(Hint: You can do it with the numbers as well. It's actually one of the easiest factors involved in my analysis.)

Their midget AAA numbers are interesting but equaled or surpassed by others.

I don't use midget AAA numbers, for two reasons: the only ones I have access to are from Quebec, and the further away a season is from the player's pro years, the less useful it is.

Iain's point is that the right things existed to be looked at (they just weren't being used at the time).

Exactly. There's nothing I'm doing that couldn't have been done at the time. Perhaps not with precisely the same results, but with similar results. And given the very rough nature of these results, similar is good enough. It could have served as a very useful second opinion to scouts, saying "are you really really sure we shouldn't even draft this guy in the 11th round, because it's quite possible he's going to be really good?"

There's no sure bet, there would be many misses if you used only the numbers. Which isn't a thing, because there are also many misses when you use only the scouts. Using both, using all information available to you, would surely be the best strategy. Get someone who knows how to interpret junior numbers, someone who can see that Patrice Lefebvre's 1985 numbers are not impressive at all, and his 1986 are only mildly so. I checked the results, the system would not have suggested drafting him in 1985 at all, and for 1986 and 1987 would peg him as a very late-round pick, which seems pretty reasonable for a player of his type. There are illusions in his stats, which you can largely see through if you understand them.

 

[Canadiens1958]

Details

So is your point that traditional methods at the time didn't identify St. Louis as a future star? (And isn't that obvious?)

Iain's point is that the right things existed to be looked at (they just weren't being used at the time).

My point is that both St. Louis and Perrin chose paths in their hockey careers that were outside the the normal hockey path of the era. The scouting and drafting in 1994 reflected the skepticism of the day.

Specifically both players were very clear that if they chose the university path they would be a duo.Likewise if they chose the Q.

As a result they took a step back during their pre draft careers to retain eligibility for the American university option. Effectively both suffered from some incompletes - no significant international competition, short season schedules, limited systems experience both in terms of playing and playing against, in their hockey resumee going into their draft year. Then there was the question about how they would perform as individuals as opposed to a duo, needed some answers.

As a result they were not well perceived from a risk standpoint by the NHL scouting/GM establishment given the incompletes and questions. The same issues were considerations for other Quebec players, regardless of size, who missed a year of mainstream hockey development to retain their US university eligibility.This is unique and necessary to Quebec since high school ends at grade eleven.

As for Iain's point. Hard to comment without seeing his data and appreciating how the incompletes are viewed or accounted for in the rankings.

 

[Canadiens1958]

Scouts

So you cannot compare players from different leagues? How do scouts do it then?

(Hint: You can do it with the numbers as well. It's actually one of the easiest factors involved in my analysis.)


I don't use midget AAA numbers, for two reasons: the only ones I have access to are from Quebec, and the further away a season is from the player's pro years, the less useful it is.


Exactly. There's nothing I'm doing that couldn't have been done at the time. Perhaps not with precisely the same results, but with similar results. And given the very rough nature of these results, similar is good enough. It could have served as a very useful second opinion to scouts, saying "are you really really sure we shouldn't even draft this guy in the 11th round, because it's quite possible he's going to be really good?"

There's no sure bet, there would be many misses if you used only the numbers. Which isn't a thing, because there are also many misses when you use only the scouts. Using both, using all information available to you, would surely be the best strategy. Get someone who knows how to interpret junior numbers, someone who can see that Patrice Lefebvre's 1985 numbers are not impressive at all, and his 1986 are only mildly so. I checked the results, the system would not have suggested drafting him in 1985 at all, and for 1986 and 1987 would peg him as a very late-round pick, which seems pretty reasonable for a player of his type. There are illusions in his stats, which you can largely see through if you understand them.

Never claimed that scouts cannot do not compare players from different leagues. The Sunday afternoon games in Ottawa/OHL, Sunday evening game in Gatineau/QMJHL are the most scouted games.

Scouts not only use elite midget numbers but the Midget espoir numbers and numbers from equivalent leagues across Canada and the world plus the various summer leagues and international competitions.They make sure they get them. Leave them out and your developmental analysis will be incomplete.

You are in the ballpark for Patrice Lefebvre. Shawinigan drafted him seriously doubting that he was an NHL draft pick. Likewise Stephan and Patrick Lebeau. Shawinigan had a perspective that they wanted their top Midget AAA players around for five seasons as franchise cornerstones.Their scouts nailed their Midget AAA picks in these situations, optimizing the team benefits..

That said, Patrice Lefebvre illustrates the reluctance to draft Martin St. Louis and Eric Perrin. Lefebvre had fewer incompletes and unanswered questions.

 

[Iain Fyffe]

My point is that both St. Louis and Perrin chose paths in their hockey careers that were outside the the normal hockey path of the era. The scouting and drafting in 1994 reflected the skepticism of the day.

No kidding. But that skepticism was misplaced, that's the point. The purpose of a system such as this is to point out where such problems might lie. It looks at what the player actually did at the level he played and at the age he played, and compares him to other players who did the same, and what they did in their later careers.

As a result they took a step back during their pre draft careers to retain eligibility for the American university option. Effectively both suffered from some incompletes - no significant international competition, short season schedules, limited systems experience both in terms of playing and playing against, in their hockey resumee going into their draft year.

So the scouts had their reasons to overlook them, that much is obvious. The point is, were their reasons valid ones, were they strong enough to overcome the very impressive results the players did put up in the hockey they did play? That the league they played in was not exactly some backwater circuit, that they played against older players and still excelled, that players a year or two older than they were usually didn't put up such numbers, etc, etc.

As a result they were not well perceived from a risk standpoint by the NHL scouting/GM establishment given the incompletes and questions.

The system doesn't care what reasons you think you have for not seriously considering a player. It just picks the players it thinks you should seriously consider. If you fail to consider them, and they later win a Hart trophy, you should perhaps rethink something.

As for Iain's point. Hard to comment without seeing his data and appreciating how the incompletes are viewed or accounted for in the rankings.

They're not. The system only considers what the player actually did, not what he might have otherwise done. It can serve as a check to see if you might be putting way too much emphasis on what the player hasn't done, when what he actually has done is outstanding.

 

[Iain Fyffe]

You are in the ballpark for Patrice Lefebvre. Shawinigan drafted him seriously doubting that he was an NHL draft pick. Likewise Stephan and Patrick Lebeau. Shawinigan had a perspective that they wanted their top Midget AAA players around for five seasons as franchise cornerstones.Their scouts nailed their Midget AAA picks in these situations, optimizing the team benefits..

That said, Patrice Lefebvre illustrates the reluctance to draft Martin St. Louis and Eric Perrin. Lefebvre had fewer incompletes and unanswered questions.

The system sees the Lebeaus as good selections, maybe 3rd-rounders or so. Whereas Lefebvre was seen as likely not worth a pick, and St. Louis as being worth an early pick. I daresay all of these particular ratings are pretty fair given how these players' careers panned out.

I understand that there was reluctance to draft these players. The entire point is that reluctance can be misplaced and exaggerated. It's the classic forest for the trees problem. You lock on one possible issue and allow it to overshadow all other factors, even if said issue is not important enough to do so.

 

[Canadiens1958]

After the Fact

No kidding. But that skepticism was misplaced, that's the point. The purpose of a system such as this is to point out where such problems might lie. It looks at what the player actually did at the level he played and at the age he played, and compares him to other players who did the same, and what they did in their later careers.


So the scouts had their reasons to overlook them, that much is obvious. The point is, were their reasons valid ones, were they strong enough to overcome the very impressive results the players did put up in the hockey they did play? That the league they played in was not exactly some backwater circuit, that they played against older players and still excelled, that players a year or two older than they were usually didn't put up such numbers, etc, etc.


The system doesn't care what reasons you think you have for not seriously considering a player. It just picks the players it thinks you should seriously consider. If you fail to consider them, and they later win a Hart trophy, you should perhaps rethink something.


They're not. The system only considers what the player actually did, not what he might have otherwise done. It can serve as a check to see if you might be putting way too much emphasis on what the player hasn't done, when what he actually has done is outstanding.

But you and the system were not there in 1994 advocating for the selection of of Martin St.Louis and Eric Perrin. You are just Monday morning quarterbacking the old drafts.

If the system works, time capsule today, the eligible non-drafted 2012 players and predict their career paths in the NHL. Five. ten years from now we will have an answer.

 

[Hardyvan123]

Your analogy is incomplete and does not reflect how scouts recommend players or how teams draft.

Basically scouts fill out an evaluation form for each draft eligible player they see, rating all the skils the team deems important as well as all the other attributes the team wishes to consider. Some players are seen by multiple scouts from a team, some are seen by only one. Each team weighs size differently but it still causes upward or downward movement in the ratings.

Then a master ranking list is prepared which includes team needs. Simply a range from Best Player Available to positional need in the organization.

In closing Martin St.Louis was not drafted because of the numbers which ranked him below the number of draft rounds. He would have had a better chance of being drafted under the old visual system supported by a scout with a solid track record. Like Bobby Lalonde app 25 years earlier.

Edit the NHLCentral Scouting Bureau with evaluation criteria goes back to 1975:

http://en.wikipedia.org/wiki/NHL_Central_Scouting_Bureau

Pretty sure that all teams scout the ECAC quite heavily and would have had a chance to see him in Laval-Laurentides (QAAA)at 16 and in Hawkesbury Hawks CJHL at 17.

teams quite simply missed on him.

 

[Hardyvan123]

What do you think would give you a more superficial concept of how a player would perform in a future game, actually watching a game or reading the box score the next day?

So how does one exactly evaluate the differences in the level of play and the degree of difficulty between say watching Wayne 81-87 and watching games between 06-12?

Looking at as many statistical models and numbers is going to tell us a lot more about the differences in the two eras than our eyes will.

I watched a lot of hockey, and Wayne in the 80's and I watch a lot today.

The level of play and pace in the 80's overall isn't that much greater than the WHL games I attend today. (That's the eye test and I'm talking level of play, time and space and teams abilities to defend ect.. not the age and strength of players).

To think that Wayne would only dip 40-50 points in his prime, and quite a few ahve suggested quite a bit less, doesn't stand up to the numbers in todays NHL game, in terms of talent and how hard it is to score.

 

[Hardyvan123]

This is simply false.

Scouting is scientific method, testing hypotheses with repeatable experiments.

A player is identified as having NHL potential - the hypotheses. Then he evaluated on the required skills. The the scouts goes to the laboratory to evaluate the player in various repeatable situations = experiments.

More specifically a 17 year old center will be watched and evaluated against different systems within his league, outside his league, internationally. Then the process is repeated later in the season to see how he progressed, if he made the necessary changes in his game, approach. Likewise the evaluations are made on an individual bases. How he plays against older players,same draft year, younger players. How he plays against drafted or non-drafted skaters, goalies, how he plays in various situations,PP,PK, end of game,etc. How he plays against opponents. in specific match-ups - vs LHS/RHS dmen or checks, etc.

On going laboratory experiment situations that are repeatable, recorded, evaluated and applied to the hypotheses.

you do not understand what the scientific method is do you?

I don't know exactly what it is either but it is conducting experiments under controlled circumstances to see if similar, or exact results can be achieved.

Observing players in any sports game is not controlling or accounting for any of those factors.


Shift by shift, game by game season by season man so many different variables.

 

[overpass]

Iain, here is your original post on St Louis.

The last thing I'll say about the "eyes" thing is this: if eyes capture everything, why was Martin St. Louis not drafted in 1994 or 1995? Trained eyes, professional eyes, saw him play and decided he was simply too small to play in the NHL. The numbers at least should have said "you really ought to consider this young fellow, what else are you going to use your 8th-round pick on?". The numbers ain't perfect, but the eyes sure ain't either.

If you are going to compare the numbers against the eyes for St Louis in 1994, you can't use your system that you have published in Hockey Prospectus. That was developed using post-1994 data, including St Louis himself, so it isn't a fair comparison to the information that scouts had in 1994. A fair comparison would use a system developed using solely available data from 1994.

 

[Hardyvan123]

Observing from an end, up high.

The greats, 1 or 2 shifts. The skating,execution, hockey sense. Wow meter worthy.

Excellent to average 3 to 6 shifts. Below average 6 to 10 shifts. You want to see them coming and going.

Goalies are trickier. The forget abouts are rather easy. Poor - skating, mobility, movement, technique.

The remaining minority is a question of adversity and fall apart moments. Happen to every goalie so it comes down to learning, adapting and changing.

Okay prove it, go to a game or watch it on tv when it starts again and let us know what type of career each 1st year player is going to have.

Even people who do this type of evaluation for a living get it wrong in the sense that you can look at most 1st rounders and throw them in a pot over and over instead of looking where they will rank from where they are picked1-30.

Here are the top players by points from the 06 draft

http://www.hockeydb.com/ihdb/draft/nhl2006e.html

The drafting is getting a bit better than it used to be but it isn't science in any sense of the word.

 

[Iain Fyffe]

But you and the system were not there in 1994 advocating for the selection of of Martin St.Louis and Eric Perrin. You are just Monday morning quarterbacking the old drafts.

If I weren't in high school at the time, maybe I would have been. But you're dancing right 'round the point, which is that any team could have produced a very similar system using data available at that time. There's nothing magical about the present day that allows this type of calculation to be done. You'd have just needed the knowledge and effort to do it.

If the system works, time capsule today, the eligible non-drafted 2012 players and predict their career paths in the NHL. Five. ten years from now we will have an answer.

Yeah, maybe I should publish a year-by-year list of the best picks according to the system at Hockey Prospectus, with the series to go up to the current day in the end. Oh, it looks like my past self already though of that (but was I actually there to suggest it to myself?) and has already put almost 20 years worth of info up. Only a dozen more to go before we hit the present day.

It's like you think this is some new thing, and not something that has many years of work put into it, with much detail available online.

teams quite simply missed on him.

Yeah, this seems a pretty non-controversial point to me.

 

[Iain Fyffe]

If you are going to compare the numbers against the eyes for St Louis in 1994, you can't use your system that you have published in Hockey Prospectus. That was developed using post-1994 data, including St Louis himself, so it isn't a fair comparison to the information that scouts had in 1994. A fair comparison would use a system developed using solely available data from 1994.

I've already addressed this. Using information available at that time, you wouldn't arrive at exactly the same results. But they would be close. Seriously, I've been studying this issue for a decade now. You're assuming the results are unique to the specific data set used, and that's not true. Any set of years you use will result in results of about the same shape, which does not include St. Louis being behind 520 other players. Any system built on scoring stats simply will not allow this, unless you drastically underestimate the quality of the ECAC or irrationally punish players under 5-10.

As I said earlier, I should have used another example, because St. Louis allows this red herring of a response. He's just the most visible example, because he exemplifies the extreme bias small players once faced from scouts using their eyes. This bias no longer exists to nearly the same extent as it once did.

The articles on Hockey Prospectus to date are largely for fun, as is spelled out often enough in them.

Okay prove it, go to a game or watch it on tv when it starts again and let us know what type of career each 1st year player is going to have.

I'd like to see a rating for every draft-eligible player, at least for the leagues my system covers. I can produce my ratings for a season in a few hours of collecting data and processing results.

The drafting is getting a bit better than it used to be but it isn't science in any sense of the word.

I'd say it's gotten more than a bit better, but there's still room for improvement.

 

[Canadiens1958]

Yes

Pretty sure that all teams scout the ECAC quite heavily and would have had a chance to see him in Laval-Laurentides (QAAA)at 16 and in Hawkesbury Hawks CJHL at 17.

teams quite simply missed on him.

Yes but they did not get to see them play a 60+ game season, any significant international games or within and against varied or complex systems.

 

[Canadiens1958]

They Did

If I weren't in high school at the time, maybe I would have been. But you're dancing right 'round the point, which is that any team could have produced a very similar system using data available at that time. There's nothing magical about the present day that allows this type of calculation to be done. You'd have just needed the knowledge and effort to do it.


Yeah, maybe I should publish a year-by-year list of the best picks according to the system at Hockey Prospectus, with the series to go up to the current day in the end. Oh, it looks like my past self already though of that (but was I actually there to suggest it to myself?) and has already put almost 20 years worth of info up. Only a dozen more to go before we hit the present day.

It's like you think this is some new thing, and not something that has many years of work put into it, with much detail available online.

Teams did produce such numerical models but the models did not allow for players to take seven seasons post draft to make a team as a regular as was the case for Martin St. Louis.

Go beyond the size variable and you have similar situations with Adam Oates, Alex Burrows, Josh Gorges, Brian Rafalski and others who were never drafted. In fact there is a net advantage to not being drafted as opposed to being drafted late, since the player can then choose the best development path for him with independent teams, enjoying the benefits of free agency not limited by the 50 contract limit that NHL teams have.

Basically the old model and your efforts show one thing, that developmental time post draft should be stretched well beyond the previously accepted limits.

 

[ot92s]

Three people who try to use statistics to predict human action went hunting...

they came across a deer. The first shot but missed by a yard left. The second shot and missed by a yard right. The third threw down his rifle and yelled, "We got him."

 

[Theokritos]

Three people who try to use statistics to predict human action went hunting...

they came across a deer. The first shot but missed by a yard left. The second shot and missed by a yard right. The third threw down his rifle and yelled, "We got him."

That's not how it works. The statisticians don't go hunting, they observe hunters going on a hunt.

I reject your standard scientific approach of relying upon empirical observation in the study of sports, and instead, favor the use of logical analysis, a logic which is influenced by Immanuel Kant's analytic–synthetic distinction.

I'm not sure 1) how your "logical analysis" looks like and 2) how it is supposed to be inspired by the distinction between analytic and synthetic. Care to explain?

 

[Johnny Engine]

Three people who try to use statistics to predict human action went hunting...

they came across a deer. The first shot but missed by a yard left. The second shot and missed by a yard right. The third threw down his rifle and yelled, "We got him."

That sounds like a 0% kill rate and a 3' average distance from the deer. Are you saying that would be impossible to distinguish from success using only numbers?

 

[ot92s]

"try to use statistics to predict human action"

That's not how it works. The statisticians don't go hunting, they observe hunters going on a hunt.

think about this part some more...

 

[Canadiens1958]

Tim Thomas

A couple of thoughts after reading the past few pages, which have been highly enjoyable. Debates like this are why I keep coming back to the History board.

1) After all that has been said about St. Louis as a prospect, only in the last few posts was there a passing reference to his strange post-draft development.

Not one, but TWO organizations offered him the opportunity to prove himself on NHL ice, and sent him packing to free agency. That's TWO coaching staffs who considered that he might have NHL potential, put him on the ice against his peers, and were so unimpressed that they didn't even expect compensation for his rights.

That has to mean something, doesn't it? It's not like St. Louis was overlooked in the draft, arrived immediately in Tampa and took a normal development curve to stardom. He was a 20-30 point type of player in Calgary, which anyone would consider inadequate for a scoring winger.

The other example that gets overlooked is Tim Thomas who played with Martin St.Louis and Eric Perrin at the University of Vermont - goalie. Thomas was a late round 1994 Draft pick of the Quebec Nordiques. Took much longer than Martin St.Louis to make the NHL. But there is no "size bias" to hook onto in the case of Thomas.

Same scouts saw all three and the evaluations were accurate within the accepted norms of a drafted players expected development curve.

Eighteen years post draft it is very easy to Monday morning quarterback the draft based on delayed results.

 

[Theokritos]

"try to use statistics to predict human action"
...
think about this part some more...

The prediction part? Well, they observe hunters and, based on their observations, form predictions. But the question is whether through induction or not. Which leads us to the second part of my post:
I'm not sure 1) how your "logical analysis" looks like and 2) how it is supposed to be inspired by the distinction between analytic and synthetic. Care to explain?

 

[Bear of Bad News]

Three people who try to use statistics to predict human action went hunting...

they came across a deer. The first shot but missed by a yard left. The second shot and missed by a yard right. The third threw down his rifle and yelled, "We got him."

This tired old joke exhibits that you aren't keeping up with what Iain's doing.

 

[Iain Fyffe]

Projectinating is not difficult

When I first brought up the Martin St. Louis example, I said that numerical analysis would suggest his future value. I did not say that only my current system would do so, and although the discussion evolved into one about that system in particular, it is not only about that system.

As much as I like to believe the Projectinator has some clever insights built into it, such a system is not rocket science. So I took an hour earlier today and built another projection system for the QMJHL, based on players who played in that league in the 1970/71 season, and their subsequent pro career to 1980/81. I then used this very crude system to project the careers of QMJHL players after that season. The output in this case is peak (best 5 of 10 seasons) points-per-game production at the NHL level, assuming a scoring environment of 3.4 goals per game. Results for some interesting players (including all the highest ratings):

Player|Season|Est PPG
Crosby S|2006|1.62
Lemieux M|1984|1.55
LaFontaine P|1983|1.40
Daigle A|1993|1.32
Briere D|1996|1.31
Lecavalier V|1998|1.26
Turgeon P|1987|1.20
Bouchard P|2002|1.15
Hawerchuk D|1981|1.09
Carson J|1986|0.74
Ribeiro M|1998|0.74
Richards B|1998|0.73
Perreault Y|1991|0.71
Damphousse V|1986|0.70
Lebeau S|1988|0.69
Lebeau P|1989|0.67
Turgeon S|1983|0.62
Lefebvre P|1987|0.62

That literally took me an hour, including the time it took me to collect the data in the first place (I used hockeyDB, needing the SIHR site only for a few birthdays). If I can put something together that produces fairly convincing results in that little time, you can imagine what can be done with extensive research. Just like any numerical ranking, some will be projected better by the scouts, some by the numbers, and many (typically including the very best players) will be more or less equally rated by the two.

And please note: none of the above players are included in the data set used to develop the estimator.

 

[Iain Fyffe]

Yes but they did not get to see them play a 60+ game season, any significant international games or within and against varied or complex systems.

If that's why they wrote him off, they put too much emphasis on those things.

Three people who try to use statistics to predict human action went hunting...

they came across a deer. The first shot but missed by a yard left. The second shot and missed by a yard right. The third threw down his rifle and yelled, "We got him."

You're supposed to say "on average, we got him". But even that's a ridiculous misrepresentation/misunderstanding.

If a player shoots a foot wide of the net on the right, then a foot wide on the left, we don't say that on average, he scored a goal. On average, he got bupkis. And he might need to work on his shooting.