New and Improved CIS Ranking System

[kovace11]

Hey guys,

First post. Yay. So in my opinion the CIS ranking system that is currently in place is really bad. Only 10 teams, no real transparent measure for how teams move up and down, and anyone outside of the top 10 doesn't have a good gauge of how they rank compared to the rest of the country.

I made a website which ranks CIS Men's Hockey teams according to their Elo rating. This rating is basically used to rank Chess players and has been adopted by FIFA to rank national teams in soccer. The rating takes into account the status of the match (regular season, nationals, provincials), the goal differential, and the relative difference in strength of the two teams. I ran it for every CIS game since 2009 and plan on updating it weekly. If there is interest, I will keep rolling with it.

Anyways, thought you guys might find some use to it. Let me know what you think.

www.kovapowerankings.com

Click to read more...

 

[Hollywood3]

The problem with formulaic rankings is that you have to pre-suppose that at some point in the past there existed a bona fide, unassailable ranking of the teams involved.

Another problem when you have teams from different leagues is that the leagues are not necessarily of the same calibre. It was not clear from your site, but I did not see if the tables included non-conference games. And, if so, what weight was given to those games relative to regular season games.

In the OHT36 rankings, I give almost no weight to non-conference games after the regular season is in full swing. However, on a conference basis I do use them as a bit of a gauge to rank where the conference mid-point should be.

This year, there were not any CW v AUS games. The OUA was 1-3 v CW and 4-9 v AUS. This is fairly consistent with past results and tends to place them just below the level of the other conferences.

I also do not carry forward the rankings from one season to the next. In IRB and FIFA rankings they do not rank teams for a specific season, since seasons vary from place to place. Their rankings are a moving target with no start nor end point.

There is also the problem of "loser points". My approach is that (whether I like it or not) this is the point system adopted. Some games are worth 2 points, and others worth 3. So a system which is based on wins rather than points will have an inherent skew inconsistent with the league's rules of competition.

FIFA rankings have also been criticized in that teams which played "friendly" games against low-ranked countries had their ranking dropped even with wins in those games.

IRB rankings are criticized because what one team gains, the other loses. So if Canada loses by 1 point to New Zealand (which would be a fabulous result for them) they would drop. Meanwhile, if Samoa beat Jamaica by 1 point (which would be a terrible result for them) they would gain.

The IIHF and IBAF have rankings which solely use final standings in international tournaments, which gradually lose value over time. This means they do not enter individual games. They simply say "Canada got 1st, Sweden 2nd, etc."

 

[kovace11]

Hey,

Thanks for your feedback.

I didn't consider any non-conference games. Provincial play-off games were given a value of 50, nationals were given a weighting of 60 and regular season games were weighted at 40.

I should note that this system is separate from the one FIFA (as an organization) uses, but rather a formula that has been adapted which people feel does a better job than FIFA of ranking member teams.

I started with the 2009-2010 season, and ran the program for every game since then up until Dec 1. 2013. The system isn't perfect, there is no such thing as a perfect system. But I feel like it does a really good job of ranking the teams, better than the CIS Top 10 anyways.

The teams that go to nationals and provincials play against each other and since these games are more heavily weighted their Elo ratings are adjusted as such depending on how they do. Their is a lot of potential for the really good teams to improve their Elo rating substantially. Once the next regular season begins lower ranked teams that might not have made it to nationals or provincials in previous years have an opportunity to move up in the standings by proving their merit by winning or drawing against higher ranked opposition. Similarly, higher ranked opposition will quickly slip down the rankings if they start off slow and begin the lose against poorly ranked opposition.

How each conference champ does at nationals while representing it's conference kind of acts as a pseudo-rater for the strength of the conference. If Team A wins the CIS, their Elo rating shoots up, and when the next season begins and they return to regular season conference play. Now, Team B (who is in their conference but didn't even make provincials) will be rewarded and now receive a higher Elo boost if they beat Team A as opposed to if they were playing in a division whose conference champion lost in the quarters at nationals.

 

[Hollywood3]

OK, this shows another problem. In early season games the ranking points are based on where your opponent was last year. So if a team is weaker this year than last, the teams that beat them early (i.e. before they fall in the rankings) get more points than the teams that beat them late. The reverse is also true.

This is another way of saying that the accuracy (or lack thereof) of a team's historical ranking has undue impact on further rankings.

I also usually stop the rankings sometime after the playoffs start. IMO at some point the rankings become moot.

I also believe that there exists a stronger correlation between this year's non-conference and this year's standings than between last year's standings and this year's standings.

 

[Big Bad Jon]

I love these stats! Any team that ranks UNB number 1 is aok in my books. Probably does need a bit of tweeking though given that Acadia has knocked off UNB twice (albeit once in a shootout) in the last month they should probably be ahead of UNB.

Always interesting to see different ranking methods. Might need a bit of tweeking, but keep it up!

 

[kovace11]

OK, this shows another problem. In early season games the ranking points are based on where your opponent was last year. So if a team is weaker this year than last, the teams that beat them early (i.e. before they fall in the rankings) get more points than the teams that beat them late. The reverse is also true.

This is another way of saying that the accuracy (or lack thereof) of a team's historical ranking has undue impact on further rankings.

I also usually stop the rankings sometime after the playoffs start. IMO at some point the rankings become moot.

I also believe that there exists a stronger correlation between this year's non-conference and this year's standings than between last year's standings and this year's standings.

Good point. I think the main strength of the rankings is that it is purely objective. You have some inputs (score, goal differential, importance of match, difference of strength between teams), and you get your output. Combine all the outputs from the different teams and you have your final ranking. Using this system there is no biased opinions involved and very little to argue about; every team's fan is obviously going to cite specific cases as to why their team should be ranked higher than other teams (Acadia beat UNB this year).

The power rankings are arguing that UNB has just won the past two national championships and is sitting just about at the top of the standings (1 point behind) in the AUS with the best goal differential. That's the beauty of the system, it has a memory. It's essentially remembering how UNB did in the recent past and given how they have done recently, it's making an assumption that UNB is still the top team. The rankings are also saying that Acadia is closing the gap (their Elo rating difference is much less now). The point you made was fair, it does work off the assumption that the teams quality remain the same from end of one year to the beginning of next, but if you're building any sort of objective ranking system there is inherently going to be assumptions that have to be made.

I think I'll keep them going throughout the playoffs just to see how things play out.

Cheers again, I really appreciate the feedback. I'll look to add more features but really want to maintain the simplicity.

 

[kovace11]

I love these stats! Any team that ranks UNB number 1 is aok in my books. Probably does need a bit of tweeking though given that Acadia has knocked off UNB twice (albeit once in a shootout) in the last month they should probably be ahead of UNB.

Always interesting to see different ranking methods. Might need a bit of tweeking, but keep it up!

Agree that there might be some truth to that but like I mentioned above this can be seen as a strength of the rankings. It's giving the top teams more protection from fluctuating down in the standings just because they find themselves in a slump. You and hollywood both see the rankings as having to be an instantaneous reflection of the league. Meanwhile, the rankings I'm trying to implement make contender teams have "earn" their right to be on top, even though it might take a full season. I guess I'm ranking the strength of a hockey program over a month-2 month window while you guys are referring to the strength of a hockey team within a given 1-2 week period. The rankings can only do one or the other, not both.

I'll use an example. Let's assume, Canada is ranked 1st and US is ranked 5th. Russia is say 4th. If United States beats Canada in a tournament twice this year (World Championships? a friendly game?) are we going to right away jump to the conclusion that the US has the best hockey team in the world?? Unlikely. We might rethink how good we think Canada is relative to the second ranked team, or improve our opinion of the US moving them a couple spots up above Russia to 4th or 3rd position, but we would give Canada some credit for what they've done in the past. Now if the US, is consistently beating top competition and defeats Canada in a very important match like the Olympic Final, then we would have a very good reason to have them on top, and I think that would naturally be reflected in the standings. This is what I mean by ranking a program vs ranking a team. Might be a bit more helpful for recruits as well. They can see which teams are consistently performing well as opposed to which teams are hot right now.

Again, thanks for taking the time to view the website and provide the critique...really do appreciate it.

 

[MiamiHockey]

Two quick thoughts ... one on the stats, and one on the value of the rankings.

Stats ... Obviously, when you are dealing with small sample sizes, you're going to face challenges, because good teams might lose 2 or 3 games due to bad luck. Over the course of the season, you expect those to even out. So, in addition to GF / GA, you might also consider SOG as a "better" metric of a team's quality. By the end of the season, the GF / GA will be fine, but early in the season, when you need more data to help discern quality, it's likely to help.

Value ... arguably, the Top 10 only matters once each year ... for the University Cup seedings. It has no bearing on qualification for the University Cup, but you could argue that it impacts whether the AUS / CWUAA / OUA champion gets the #1 seed, and who gets the #4, 5, and 6 seeds, which is critical. So, the important issue is getting the Top 10 to be more accurate by the end of the season, not at the beginning. If that is your goal, then the previous season's results should not be factored into the ranking at all, because by the end of the current season that information provides no value.

 

[kovace11]

Rankings released current as of January 11, 2014.

www.kovapowerankings.com

 

[Hollywood3]

Rankings released current as of January 11, 2011.

www.kovapowerankings.com

The link to 2013-14 hockey is not working, for me at least.

 

[Canadiens1958]

Sad But True

The link to 2013-14 hockey is not working, for me at least.

Sad but true. Point was PMed to the originator without response leaving no choice but to close the thread. It will be deleted if there is no reply to my previous PM.

 

[Canadiens1958]

Re-opened.

The thread has been re-opened. Worthwhile effort and read once we get by the initial bumps in the road with dates being out of sync and link questions raised previously.

Trust things will run smoothly going forward once the necessary edits are made.

 

[kovace11]

Thanks.

Yeah, sorry about the hiccup. Not really an expert in running a website so I'm kind of learning on the fly. I'll try and doing a better job of checking my PMs and making necessary edits. If you see a bug just send me an email via the "Contact" tab on the website that way it'll go directly to me and I can change it ASAP.

With that, the rankings current as of January 19th are live.

UNB back to number 1, but only 2 points ahead of Alberta, so this should make for an interesting week.

Windsor and UPEI were the biggest positive movers, moving up by 5 and 4 spots to 11th and 12th.

StFx and Ottawa both dropped by 5 spots to 16 and 17.

 

[Hollywood3]

The 2013-14 link is not working again.

 

[kovace11]

Works fine for me and everyone I've spoken to. Thanks though! Also, I've updated the standings making them current up to the last day of the regular season.

 

[kovace11]

Just updated the rankings current as of the morning of February 27. Playoff games are now weighted more. I also included a column to show how much the actual ELO score has raised or dropped based on the week's performances.

Lakehead jumped up 7 spots to 12. As did UBC to 16. Sask dropped by 6 to 14 and UofT by 4 to 22.

1. UNB
2. Alberta
3. Carleton
4. Acadia
5. McGill
6. UQTR
7. Western
8. Saint Mary's
9. Windsor
10. Calgary

Check out the full stats at: http://www.kovapowerankings.com