Pre-draft lists are a terrible way to benchmark IMO. What makes Bob Mackenzie's polled scouts better than ours?
We see it every year, Jordan Schroeder is a great example of a player who "fell" to 22, way down from the pre-draft rankings, and 4 years later it looks like the pre-drafts were way off.
Pre-draft helps "fans" analyze their teams picks. Teams have enough data, and enough resources to track the players down the line. The old adage is still true, it takes about 5 years to rate a draft class.
I'd personally look to try and poach some more good area scouts from good drafting teams like Ottawa.
Do you know what benchmarking is?
You compare your results to another person in a similar situation.
If you do worse, you need to improve, if you do better, then you are better.
So by comparing our list to other pre-draft list, we know who we would pick and who they would pick...then we see how things turn out over time by comparing the players we would end up with if we used their list to the guys we actually eneded up with. If the guys we would pick turn out better, we benchmark well, if the guys they would've picked do better, then we benchmark poorly.
To make any conclusions about a draft immediately after the draft is basically just a guess.
--------------------------
Here's a comparison:
Actual - Kesler, Schneider, Bourdon, Grabner, White, Hodgson, Schroeder, Jensen, Gaunce
ISS - ???, Graham, Koipitar, Vasyunov, Mayorov, Hodgson, Schroeder, Grimaldi, Gaune
TSN - Glazachev, Bolland, Kopitar, Sanguinetti, Petrecki, Hodgson, Schroeder, Saad, Gaunce
CSS - Fridsche, Bolland, Kopitar, Stewart, Perron, Myers, Schroeder, Rattie, Gaunce (Note, CSS only does NA and Intl. so I picked the NA or Intl. based on who was higher on TSN's list to try and make it fair)
Redline - A. Stewart, Green, Skille, Williams, Katic, Petrov, Schroeder, Khokhlachev, Finn
Based on this comparison, I'd probably rank the lists as follows:
1. CSS
2. TSN
3. Canucks
4. ISS (really depends on who they get in 2003...they didn't exist I don't think)
5. Redline
I think that our paid scouts should be able to beat free services and public lists and it shouldn't be close which means we do need to improve, that being said, I think we stayed pretty close to the board on most of our recent picks except with Horvat last year (the wide majority would have picked Nichuskin) so a GM/scout should either make their name, or be fired based on that pick.
It should be noted that we have gone to the OHL or US 9 times in the last 11 years for our 1st pick (including the above + Horvat and McNally).
P.S. Doing this has gotten me reading some scouting reports...a lot of reports compare Jeff Carter to Joe Thornton (because both played in the Soo)...their games are really nothing alike at all.