Combining Scouting Reports And Numbers

[R S]

I've been working on coming up with my own formula that will combine information from a player's scouting report and the player's on-ice production. Ideally the information is compiled and generated to provide a ranking player-by-player and team-by-team for use by that team's hockey operations department.

I've posted my early findings on my blog, with the first focus of the project being on junior hockey. Feel free to give it a read.

http://whl-from-above.blogspot.ca/2014/08/combining-scouting-and-stats.html

As hockey fans know, it’s been a big summer for analytics. While some NHL teams have been using advanced stats behind closed doors for years, the scene was given greater exposure this off-season thanks to organizations plucking various bloggers from the internet and moving them into front offices around the league.

For the past 18 months or so I’ve taken an interest in the topic, looking to learn as much as I could in an effort to see the game from an another angle and expand my knowledge base.

Over that time one thing has become very clear to me. Statistics are a great tool to have, but they are not the perfect solution. The game of hockey is simply too fast, too random and too complex when compared to baseball, for example, to really be able to put too much stock purely into statistics (at least for the time being).

With that in mind, it occurred to me that it would be ideal to come up with a metric to combine the statistical side of the game with the scouting part of the game. This would end up being a number that would roll a player’s skillset and in-game production into one comfortable package. If done thoroughly across the board, it can give you a ranking player-by-player and team-by-team to show you which players are the best combinations of production and style of play.

Click to read more...

 

[Doctor No]

Very interesting stuff - thanks for putting it out there. Bill James was talking about this sort of thing in the 1980s (I believe as part of his Brock-2 treatise in the 1985 Baseball Abstract), about the potential for incorporating intangible data (such as "speedy", or "contract hassle") into the numbers. Of course, at the time we were all running on 64k machines, and his Brock-2 would barely run as is.

It would require contemporary scouting reports, but I think that it would be important to validate the process on past data.

 

[Mathletic]

The approach is interesting. It compares to what the A's and Patriots do in other sports. It's also similar to what I did this past year using the CSS scouting sheets. Although it's still subject to subjective assessment, if you work on the grading you'll come to something fairly objective. At least, more so than gut-feel decisions. Then again, I wouldn't use this on its own. I'd incorporate this into a bigger model, especially if you had a bit more data than only for this one year. Problem with using this on its own is you'll tend to put a figure on a subjective inerpretation of a player's value. You may end up putting something like hockey IQ and skating worth the same when in reality one could be more important than the other. Other models like adjusted +/-, point shares, CORSI and so on don't describe reality very well but they capture the macro effect much better.

 

[Aceboogie]

Very cool stuff!

I will defiantly be following this.

I truely believe that drafting in rounds 3-7 is a crap shoot and there is a huge opporunity to use stats and scouting reports to drastically improve your chances of finding a player.

 

[R S]

Very interesting stuff - thanks for putting it out there. Bill James was talking about this sort of thing in the 1980s (I believe as part of his Brock-2 treatise in the 1985 Baseball Abstract), about the potential for incorporating intangible data (such as "speedy", or "contract hassle") into the numbers. Of course, at the time we were all running on 64k machines, and his Brock-2 would barely run as is.

It would require contemporary scouting reports, but I think that it would be important to validate the process on past data.

Thanks for the feedback.

In doing some further work on the subject it's become clear just how important the accuracy of the reports is. Along with that, the transferring of the report to a number based system NEEDS to be consistent from player to player for it to have any validity. That likely would make the process hard for an entire team to employ just due to the difference in opinion/style from scout to scout, for example.

I also agree that looking back at past drafts would be necessary to help validate it. Unfortunately I only have my personal set of reports from the 2012, 2013 and 2014 NHL Drafts (on WHL players only), so I will be looking to use that information to see if it helps turn anything else up.

The approach is interesting. It compares to what the A's and Patriots do in other sports. It's also similar to what I did this past year using the CSS scouting sheets. Although it's still subject to subjective assessment, if you work on the grading you'll come to something fairly objective. At least, more so than gut-feel decisions. Then again, I wouldn't use this on its own. I'd incorporate this into a bigger model, especially if you had a bit more data than only for this one year. Problem with using this on its own is you'll tend to put a figure on a subjective inerpretation of a player's value. You may end up putting something like hockey IQ and skating worth the same when in reality one could be more important than the other. Other models like adjusted +/-, point shares, CORSI and so on don't describe reality very well but they capture the macro effect much better.

Thanks for the thoughts, I know you have done some similar work in areas like this and respect your opinion on the matter.

As for the value on the categories in my reports, I essentially broke it down into 5 "categories" more so than traits. Looking back, traits is definitely the wrong word. I won't divulge which traits I include in my categories, but I feel like I've done a good job keeping them balanced, value-wise in relation to each other.

And I 100% agree that the more data, the better. Not only can the scouting categories likely be expanded on, but eventually being able to use advanced stat figures in a model like this would drastically improve it. Unfortunately with this model currently being only compared to junior hockey my hands are tied in that aspect.

Very cool stuff!

I will defiantly be following this.

I truely believe that drafting in rounds 3-7 is a crap shoot and there is a huge opporunity to use stats and scouting reports to drastically improve your chances of finding a player.

Thanks for taking the time to read it.

 

[BeardyCanuck03]

Very cool stuff!

I will defiantly be following this.

I truely believe that drafting in rounds 3-7 is a crap shoot and there is a huge opporunity to use stats and scouting reports to drastically improve your chances of finding a player.

I'm not the biggest fan of using analytics that much for drafting. There are too many variables that can't be taken into account. How does one league compare to another? Will the player be able to put up these numbers in the NHL?

I think using analytics in this way is a good start, a good way to "pre-scout" but you can't rely on them to make your decisions. The dreaded eye ball test needs to be a larger part at this point. Analytics shows a result, and in a sport that is as fast and complicated as hockey is, there are many reasons as to why a player may have a certain result, and he may not have the same result in a different situation (role/team/linemates).

Rounds 3-7 will always be a crap shoot and there will be multiple ways to find diamonds in the rough. This could be one, but this will not be able to find late bloomers, players who take longer to either grow or figure out how to take their game to the next level.

 

[Aceboogie]

I'm not the biggest fan of using analytics that much for drafting. There are too many variables that can't be taken into account. How does one league compare to another? Will the player be able to put up these numbers in the NHL?

I think using analytics in this way is a good start, a good way to "pre-scout" but you can't rely on them to make your decisions. The dreaded eye ball test needs to be a larger part at this point. Analytics shows a result, and in a sport that is as fast and complicated as hockey is, there are many reasons as to why a player may have a certain result, and he may not have the same result in a different situation (role/team/linemates).

Rounds 3-7 will always be a crap shoot and there will be multiple ways to find diamonds in the rough. This could be one, but this will not be able to find late bloomers, players who take longer to either grow or figure out how to take their game to the next level.

I think no matter how good a scout is, the success rate for these guys will still be extremely low. Simply because they are drafted at 18, so this is a very volitale time for development. Some 17 YOs developed at an early age and thats the reason for looking good, but will fizzle out in the coming years. Some players develop later in life, so go undetected in the draft

The reason the NFL has less busts (and less rounds) is that they draft the guys at 21, as opposed to 18. So those guys have a chance to develop and show their true ability. Imagine being able to pick the 2010 draft today.

So that is true what you said. But thats the nature of the best in picking 18 yos. No advanced stat or scout can overcome this difficutly

Another thing is that 3-7th round prospects are not scouted as heavily as 1st rounders. They are probaly scouted 4-5 times by teams, and even then arent zoned in on. So theres a risk a scout catches that guy during a rough patch and chooses not to draft him, similarly goes for a guy on a hot streak. Advanced stats, and rating from team scouts who watch every game can show a better picture over a whole year.

I have pretty extreme views when it comes to the draft and later round picks. I think these picks are such a crap shoot with a 4th rounder and 7th rounder having a pretty similar shot at making it. I personally think a team could employ one of the following methods and come out better than normal

a) Weigh all the scouting agencies based on how good you think they are (mckeens, redline, NHL central scouting). Then apply this weighting to each player and their rankings from each different scouting agency and combine a prospects different ranking to get one overall rankings score and draft the player with the best rankings score

b) Pick the forward from the CHL with the highest point totals who is available when you draft. Use NHLe to compare to other high scoring forwards in other leagues (take into account TOI for SEL and KHL)

c) There was advanced stats for 17 YOs that was available for this draft, so if that is available next year, pick the player with the best analytics number (employ weighting for the different categories such as corsirel, quality of comp). This can help you pick defense, or two way forwards

Ofcourse these should only be done for 3rd-7th rounders, as they only have a roughly 8% chance of working out anyway. So its not like you are throwing a pick away

This is purely hypothetical and cant see any team doing this in the future. But when I have time I would like to do work for scenerios A and B for different teams drafting from previous years to determine who they would have picked if they had used this strategy and if they would have been better off.

 

[R S]

Just posted a new piece that looks back at the 2014 NHL Draft while using my new stats/scouting report formula.

http://whl-from-above.blogspot.ca/2014/08/combining-scouting-and-stats-20-2014.html

I’ve spent the last few days and weeks digging into numbers at the junior level in hopes of finding a way to combine scouting reports with on-ice production numbers. As someone who scouts the game at the junior level and someone who wants to one day work in the hockey operations department of an NHL team, this particular branch of the game is extremely interesting to me.

Following the work on the Blades roster in that first post, I started to think how the information might be applicable to the process of drafting players. While the system seemed to quite accurately sum up how the players on Saskatoon’s 2013-14 roster played the game, would the system help predict their future growth and success?

With that question in mind, I turned to the data I had compiled for Eastern Conference based WHL prospects ahead of the 2014 NHL Draft. Given I am located in the WHL’s Eastern Conference, I really focused on these player’s only when it came to providing a draft ranking in 2014.

 

[Hardyvan123]

I'm not the biggest fan of using analytics that much for drafting. There are too many variables that can't be taken into account. How does one league compare to another? Will the player be able to put up these numbers in the NHL?

I think using analytics in this way is a good start, a good way to "pre-scout" but you can't rely on them to make your decisions. The dreaded eye ball test needs to be a larger part at this point. Analytics shows a result, and in a sport that is as fast and complicated as hockey is, there are many reasons as to why a player may have a certain result, and he may not have the same result in a different situation (role/team/linemates).

Rounds 3-7 will always be a crap shoot and there will be multiple ways to find diamonds in the rough. This could be one, but this will not be able to find late bloomers, players who take longer to either grow or figure out how to take their game to the next level.

I agree with this, especially for draft aged players who are still developing their games and growing into their bodies, the amount of variance for them and projections based on stats from junior teams probably never will be as reliable as say for older mature players already in the pros.

that being said it's interesting stuff and worth pursuing to be sure.

 

[NHLPaul]

Awesome idea. I think most people have been thinking a combination is the best way to analyze a player. It's just a hard thing to do. This is a good step and I will definitely be following it. Good work.

 

[R S]

New piece up and it involves combining statistical draft rankings with "eye-test" draft rankings to try and find that perfect hybrid ranking.

http://whl-from-above.blogspot.ca/2014/09/combining-scouting-and-stats-30-perfect.html

My last piece started to explore what would happen when you used a combination of scouting reports and statistical production to determine future pro success. And while it was very preliminary, only looking back to the very recent 2014 NHL Draft, it did seem to turn up some interesting results.

In looking back at that piece it seemed to help indicate which players may have the best potential offensive ceiling, longterm, ranking them accordingly on a very basic level.

And while the system is not perfect, or proven mind you, it got me thinking about finding a hybrid system to rank the players even more accurately. Because at the end of the day, that’s what scouting is all about, ranking the players as close to “perfect” as possible.

There were three players on my list (ranked by their NR, essentially a made up category which uses a formula to combine their scouting report and statistical output) that really got me thinking about a way to refine the ranking.

The players were Brayden Point, Jayce Hawryluk and Jaedon Descheneau.

 

[NikF]

I'll be following this.

about the potential for incorporating intangible data (such as "speedy", or "contract hassle") into the numbers.

I've tried doing something similar last year then gave up due to a couple of things. Mostly, the biggest issue when trying to score a player numerically is which categories to even use and how many. I've settled on some but then inevitably started falling in the "yeah, but..." arguments with myself because each prospect utilizes his own unique skillsets in a way that produces an unique player. Not only are skillsets unique to each player but the combination and the interaction between different skillsets are unique as well. It would be like judging very different art paintings by applying very crude basic drawing techniques to all of them and scoring them off that.

The only two attributes that I would be comfortable applying to all are the ability to read the play and compete level (or hockey IQ and compete level, but again this is very subjective), and even then there are things that are "excusable" when there are certain skillsets backing them up. In short, maybe Virtanen can afford to have a bit of a hole at the hockey IQ side of it because of his athletic ability. Then there's the issue to say Virtanen lacks hockey IQ to begin with - I think he has a tendency to be a one and done offensive player and sometimes doesn't know how to sustain the play, but then again he is also quite smart at reading and finding himself in scoring positions. It gets really complex, so where and how are you going to draw the line exactly? What's the next step? Apply different models of ratings to different types of players? For example a quick-strike fast winger gets a differently weighed (or different altogether?) focus on attributes than a playmaking center? I don't know what the ideal combination would be.

 

[R S]

Bumped with a new piece.

After working to develop a system to grade players based on a combination of scouting reports and statistical output, I’ve been attempting to try and double-check the system using some of my old scouting reports.

In my last piece I looked back at the 2014 NHL Draft and introduced the idea of a Hybrid Ranking. While that data brought up some interesting notes, it was hard to really dig too much into it because that draft is still so recent, having just happened this past summer.

With that in mind, I decided to go back and do some digging into the 2013 and 2012 NHL Drafts. Looking back at these draft years should give me a bit more of a window to see how some of those players in question have developed since their big draft day.

I decided to start with looking back at the WHL forwards from 2013.

http://whl-from-above.blogspot.ca/20...-stats-40.html

 

[R S]

Been a while since I added a new piece. But I put one up today in regards to how things look if I re-ranked the 2013 NHL Draft using on-ice production data mixed with scouting reports.

In my last piece I looked back at the WHL forwards from the 2013 NHL Draft and re-ranked them using my new formula, including my new “Hybrid Ranking” idea. It brought about some interesting observations, including the idea that guys like Nic Petan and Greg Chase offered amazing value in 2013. You can read more about that here.

I did a similar thing for the 2014 NHL Draft, although it’s harder to truly gauge that one yet because that draft was only this past summer. You can check that piece out here.

With that in mind, I decided to again dig back into the 2013 NHL Draft. While the group of WHL forwards in that draft was pretty solid, the blueline seemed to be a step above. The WHL defender group seemed to have better top-end talent and better depth. This recent digging seems to confirm that, based on how the players have developed since.

- The first main observation is just how good Seth Jones was compared to the rest the WHL class on the blueline. His Player Rating was the highest of any 2013 defenceman from the WHL, not surprisingly, while his NR soared 5 points higher than any other D. The only defender to come close to his NR mark from the past two NHL Draft’s was Julius Honka with an 82.

- There exists a clear indication here that there was a “top 3” in 2013 for WHL blueliners based on this ranking. Interesting to note that inside that battle, Morrissey was picked ahead of Pulock, despite Pulock sitting second on both my Actual Draft Rank and my NR Ranking list. Either way, both are top end prospects who should make decent NHL impacts next season. Mirco Mueller is a guy who will be the 2nd WHL defender from 2013 to jump to the NHL, although this list is about future top-end potential, not who plays first. Either way it’s worth noting that he’s well on his way to being an NHL player already.

The rest --> http://whl-from-above.blogspot.ca/2014/10/combining-scouting-and-stats-50.html

 

[R S]

Posted what will potentially be my final idea in this series as I have gone back and used it to re-rank the 2012 NHL Draft from a WHL perspective.

Once again, it shows evidence that essentially teams should always lean to the more offensive guy in the draft, taking the guy who has produced more offensively.

Feel free to check it out and let me know if anyone has any questions or comments. Feedback is greatly appreciated.

http://whl-from-above.blogspot.ca/2014/12/cnrs-60-using-stats-to-help-re-rank.html