Objective Assessment of Each Amateur Scout Versus AI

[Izzy Goodenough]

After watching these clips from the Canucks Amateur scouting meeting, it seems the whole group is susceptible to group dynamics that result in subjective ranking based on the "loudest voice in the room syndrome".

This is what Allvin is implying he is probably trying to avoid. Thus far the Cross-over scouting approach they adopted several years ago does not appear to be working as, likely because the input involves too few validated voices.

It would be interesting to know if any professional sports franchise has methods to assess the TALENT of the individual scouts by allowing all of them to project their own comprehensive lists before the meetings and then to retrospectively assess which ones got it right in previous drafts based on games played in the NHL or some other metric.

It would also be interesting to determine if machine learning, trained on the performance of prospects from earlier drafts, could better rank the current prospects compared to the scouting department.

At least then, Allvin would know who to lean on even if they aren't the alpha in the room or whether he should lean on AI.

In the absence of this sort of assessment, the Canucks scouting will likely continue to be abysmal.

Click to read more...

 

[rypper]

I don't think you can really read anything into these short clips that say nothing.

 

[Izzy Goodenough]

I don't think you can really read anything into these short clips that say nothing.

Then how do you explain this summary of the long history of Canucks' scouting and drafting:

 

[MarkusNaslund19]

I don't think you can really read anything into these short clips that say nothing.

No kidding, Jesus Christ the levels of extrapolation on these boards sometimes are incredible.

 

[Petey But Really Jim]

No kidding, Jesus Christ the levels of extrapolation on these boards sometimes are incredible.

Oh I’m sorry, are we posting wrong again?

 

[MarkusNaslund19]

Oh I’m sorry, are we posting wrong again?

I'm sorry, who's 'we'? Did I address you?

 

[David Bruce Banner]

As far as I'm concerned, our amateur scouting, at least under Allvin and Rutherford, is largely unproven. Letterkenny had an up and down year, but that's pretty much what you'd expect from a 15th OA. At least he finished the season strong. Re-Petey seems like a win, so far. The rest are late round longshots anyway.
And why do we need AI, when we have @Melvin 's Potato GM?

 

[Izzy Goodenough]

AI trained on previous draft successes can assess draft lists prepared by humans currently employed as scouts.

So, in summary, Luddites, AI is your friend until it isn't.

 

[Play]

As far as I'm concerned, our amateur scouting, at least under Allvin and Rutherford, is largely unproven. Letterkenny had an up and down year, but that's pretty much what you'd expect from a 15th OA. At least he finished the season strong. Re-Petey seems like a win, so far. The rest are late round longshots anyway.
And why do we need AI, when we have @Melvin 's Potato GM?

Melvs potato is superior for only 2-3 years I’m guessing. Then AI will incredibly top over it

 

[Nucker101]

Would be cool if the team hired an independent(not affiliated with hockey operations) analyst or company to keep a long term record and evaluations of individual scouts. Because we all know scouts essentially get a reset button anytime a new President/GM is brought in.

 

[MS]

Would be cool if the team hired an independent(not affiliated with hockey operations) analyst or company to keep a long term record and evaluations of individual scouts. Because we all know scouts essentially get a reset button anytime a new President/GM is brought in.

I mean, it shouldn’t even be that complicated.

Surely they have ~20 years of stuff saved somewhere and if I (or many other posters here) had a look at I could tell you in probably 2 hours which scouts knew what they were doing and which were in outer space.

 

[andora]

I mean, it shouldn’t even be that complicated.

Surely they have ~20 years of stuff saved somewhere and if I (or many other posters here) had a look at I could tell you in probably 2 hours which scouts knew what they were doing and which were in outer space.

Fot something like this you would need criteria.. or columns in orser to measure

What would be the columns we are trying to measure their ability to draft skills.. goals .. leadership etc...

Just having nhl games as criteria is empty imo as so much goes into a player getting there.. maybe getting to the ahl as an established player - identifying a player good enough to get to a point of where coaching and internal development take over to help or stagnate the player

 

[MS]

Fot something like this you would need criteria.. or columns in orser to measure

What would be the columns we are trying to measure their ability to draft skills.. goals .. leadership etc...

Just having nhl games as criteria is empty imo as so much goes into a player getting there.. maybe getting to the ahl as an established player - identifying a player good enough to get to a point of where coaching and internal development take over to help or stagnate the player

I wouldn’t be using a GP type criteria. I’d just be looking for smart and looking for stupid.

On the leaked 2010 draft list, our WHL scouts put a double-overage with 3 points in Teigan Zahn into the 3rd round and didn’t get any of Mark Stone/Brendan Hallagher/Radko Gudas (the eventual 3 best WHL players from that draft) rated anywhere on our draft list. Doesn’t take a complicated formula or external review to see that the WHL scouts had no idea what they were doing.

 

[andora]

I wouldn’t be using a GP type criteria. I’d just be looking for smart and looking for stupid.

On the leaked 2010 draft list, our WHL scouts put a double-overage with 3 points in Teigan Zahn into the 3rd round and didn’t get any of Mark Stone/Brendan Hallagher/Radko Gudas (the eventual 3 best WHL players from that draft) rated anywhere on our draft list. Doesn’t take a complicated formula or external review to see that the WHL scouts had no idea what they were doing.

Beyond sad canucks couldnt be bother to take gallagher.. that one always bothered me.. took some bum with there 5th that was a few before gallagher

I remember seeing giants games that year.. gallagher made sense with the way he played.. it bugged me

 

[Tables of Stats]

The way to do a drafting AI would be to set a range of dates - I'd probably pick a 20-year range to try to get as large a sample size as possible - and then take every player who was drafted as well as any undrafted players who made the NHL or AHL and assign them a score. You might set a single overall score or try to rate categories like offense, defense, skating, and physical play. Then you design an AI to see if it can identify trends in the highest-graded players and try to ascertain if there are any traits that are chronically overlooked.

Then you can take your scores for the incoming class and use them to inform your drafting choices. The trick is figuring out if the system actually works, so you might want to train a set on the oldest 10 and 15 years and have them predict drafts that have already happened to see if the AI can identify talent.

 

[credulous]

there's not enough data on prospects to train any kind of useful ml model. all you really have access to is their counting stats and the leagues they played in and their height/weights for their draft year maybe. you'll basically just find that players that score a lot and are big get drafted high. adding scouting reports from various scouts you might do better but i'm skeptical you could get a cohesive set of labels out of scouting reports

any model would probably do better than "traditional" scouting but i doubt it would be appreciably better than the potato model

 

[VanJack]

Forget AI....over the years posters on these boards have done a better job drafting players than the guys who get paid to do it for the Canucks.

Hopefully Allvin has swept away the vestiges of the scouting disasters that pock-marked the Benning regime.....but have to admit I really don't have any idea of all the changes they've made.

All I can say is that hopefully the drafting of Woo; and signing of guys like Bains, Wouters, Neilsen and Kannok Liepert signals that the Canucks are finally doing something different in the WHL.

 

[mossey3535]

there's not enough data on prospects to train any kind of useful ml model. all you really have access to is their counting stats and the leagues they played in and their height/weights for their draft year maybe. you'll basically just find that players that score a lot and are big get drafted high. adding scouting reports from various scouts you might do better but i'm skeptical you could get a cohesive set of labels out of scouting reports

any model would probably do better than "traditional" scouting but i doubt it would be appreciably better than the potato model

This. People seriously underestimate the skill and amount of data you need to make a ML work. Like anything else ML/AI has it's strengths and weaknesses, it's still garbage in=garbage out if you have low quality data.

 

[Izzy Goodenough]

Considering the input data, probably the most meaningful metric is simply minutes played (in NHL and/or AHL). I think this is likely the catchall for the performance of draft choices rather than scoring. A nuance could be minutes /games played or minutes on the power play or the penalty kill.

This metric would train AI on all positions and roles on a team.

Other possible training metrics around this stat are possible.

 

[bossram]

there's not enough data on prospects to train any kind of useful ml model. all you really have access to is their counting stats and the leagues they played in and their height/weights for their draft year maybe. you'll basically just find that players that score a lot and are big get drafted high. adding scouting reports from various scouts you might do better but i'm skeptical you could get a cohesive set of labels out of scouting reports

any model would probably do better than "traditional" scouting but i doubt it would be appreciably better than the potato model

Yep. This is it. For most junior leagues, all a model would have to go off are counting stats and listed height/weight. I highly doubt an "AI" model would be produce anything meaningfully different than the models that already exist, like the "potato" model.

 

[StrictlyCommercial]

Two part draft system:

Quantitative: Does he put up points?
Qualitative: Can he skate?

Guarantee better success than the Canucks.

 

[F A N]

It would be interesting to know if any professional sports franchise has methods to assess the TALENT of the individual scouts by allowing all of them to project their own comprehensive lists before the meetings and then to retrospectively assess which ones got it right in previous drafts based on games played in the NHL or some other metric.

The issue with that is that the list would be based on incomplete information. If you were a talented regional scout you don't really know much about players outside of your region. You may be right most of the time with your assessment of the players you have viewed. And sometimes rankings are judgement call based on individual biases. The player you think could turn out to be the best player in the draft might not be the guy you would draft #1 overall for reasons.

Gillis attempted to draft using past data. They were prioritizing regions based on the number of NHL players that came out of that league.

 

[God]

I don't think it would be that good, but I wonder how much information out of public scouting reports you could get with a natural language processor and whether it would perform adequately with just points. I suspect the answer is not much... because all of the scouting reports read pretty much the same to me: skates well, great shot, reads the play, etc.

 

[Vector]

I don't think it would be that good, but I wonder how much information out of public scouting reports you could get with a natural language processor and whether it would perform adequately with just points. I suspect the answer is not much... because all of the scouting reports read pretty much the same to me: skates well, great shot, reads the play, etc.

If Sam Cosentino was replaced with an AI, his lists would actually be improved. So there's that.

 

[God]

If Sam Cosentino was replaced with an AI, his lists would actually be improved. So there's that.

If I used Sam Consentino's scouting reports as training data for a natural language model, it would probably be better to simply trade all of your picks for last year's 15th overall.