Testing Whether We Need to Spend to Win

[StefanW]

As you all know, Eugene Melnyk has been making the claim that you do not have to spend cash like a drnuken sailor in order to win. The counter argument is that teams at the top of the spending scale are the ones who win more often. So who is right?

I'm not one to just take someone's word for it on important questions like this, so I decided to statistically test the relationship between spending in relation to other teams and winning. To do this, I compiled imformation about spending and success for each of the 30 teams over a 6 season period (07-08 to 12-13), which gave me 180 team cases (6 seasons X 30 teams) to work with. If you are not into stats you may want to stop reading now and just skip to the tables. For those who stick it out, I will try to keep things at a basic level so those who are only casually interested, and how little or no background in numbers, can still easily follow.

The variables I used were:
Season (08-09, 09-10, etc)
Team (Sens, Pens, Sharks, etc)
Spending (in relation to other teams, so the top spending team is coded 1, second is coded 2, etc)
Spending category (rank order spending divided into three groups: top third, middle third, and bottom third teams)
Result (0=missed playoffs, 1=out in first round, 2=out in second round, 3=out in third round, 4=lost cup final, 5=won the cup)

After checking to make sure the data is clean and good to go, the first thing I check the correlation between spending and result. Correlation basically means that when one of these variables changes, the other variable changes as well. The result was a Pearson correlation of .37, which means that when spending goes up the result moderately goes up. From a stats point of view this corrrelation is ok but nothing to write home about. It is important to note that correlaiton is not the same thing as causation. To use a non-hockey example, people getting into car accidents is strongly correlated with people shovelling snow. This does not mean shovelling snow causes car accidents.

The next thing I did was check how spending category relates to result. The results can be seen in the following table, which shows how often top third (first through 10th in spending), middle third (11th through 20th), and bottom third (21st through 30th) spending teams have had different level of success:

Result | Top Third | Middle Third | Bottom third
Missed Playoffs | 13 | 34 | 37
Out 1st Rnd | 20 | 13 | 15
Out 2nd Rnd | 12 | 8 | 4
Out 3rd Rnd | 8 | 2 | 2
Lost in Final | 4 | 1 | 1
Won the Cup | 3 | 2 | 1

When you eyeball these numbers it appears that teams at the top of spending are less likely to miss the playoffs and more likely to make it deep into the playoffs. However, in stats eyeballing something is not enough. What we do is figure out how likely the set of results charted in the above table is to occur completely by chance. To do this I ran a test called an analysis of variance (ANOVA for short). This set of results is highly unlikely to occur completely by chance, and are thus what we refer to as "statistically significant." For people who are into stats the results are F(29,150)=1.90, p<0.001. For those that are not, sorry if I made your eyes burn there.

The trouble with ANOVA with three groups is that the test only tells you if the overall model is significant. In other words, there is no real detail. E.g. top spending teams may be more likely to succeed than bottom spending teams, but there may be no statistical difference between top and middle spending teams. To drill down and compare results of the the three categories of teams I used what is referred to as a post hoc analysis. People who are not into stats can skip the rest of this paragraph because it is not important to you. For those who are into stats, I selected a Games-Howell post hoc test due to the unequal variances in the three groups.

The following table summarizes the comparison between the three groups. If the "significance" value is below 0.01 the difference between the two groups is unlikely to be a fluke.

Cap Category| Compared with | Standard Error | Significance
Top Third | Middle Third | 0.240 | 0.002
- | Bottom Third | 0.226 | <0.001
Middle Third | Top Third | 0.240 | 0.002
- | Bottom Third | 0.209 | 0.656
Bottom Third | Top Third | 0.226 | <0.001
- | Middle Third | 0.209 | 0.656

Based on this table, the top third of spending teams have significant greater success than middle or bottom spending teams, but there is no statisically significant difference between results for middle third and bottom third teams. So Mr Melnyk is right, sort of. If the goal is to do well, you can be frugal and still meet the objective. If the goal is to go into the third round of playoff and beyond, and to eventually win the cup, you are significantly more likely to succeed if your spending is in the top third of the league.

Limitations:
1) I was only able to find good cap data for a 6 years stretch. I would like to expand this to include the entire cap era.
2) The limited sample size lead to a pretty high standard error. I'm not happy with that.
3) Limited sample size meant I had to collapse teams into three categories. I would have preferred six groupings for a more refined analysis.
4) Rank ordering teams in this way is not ideal because the gap, for example, between first and second place spending teams may be greater than the gap between the 29th and 30 th place spending teams. This leads to pretty wonky variances.

Future Possible Analysis:
1) If I input the actual team spendings I can test Melynk's "what is important is dollar per point" theory that he trumpets in interviews. I'd like to do that when I have some more time.
2) Once I collect more seasons of data I would like to look at trends, and see if the gap between rich and poor teams is widening, closing, or staying the same over time.
3) I'd love to do more of a team by team analysis if I can get enough seasons of data.


Thanks for having a look, and feel free to make comments, criticisms, and suggestions. I slapped this together fairly quickly, and accept (and even expect) that I may have made embarassing mistakes along the way.

Click to read more...

 

[krapsik]

Nothing to criticise. Very good reading. Waiting for future more deep analyse.

 

[Quo]

Wow Stef.

At a glance, this is great work you've done here. I can't speak to the numbers specifically as I'm about the furthest thing from a statistician but I love looking at things like this.

Results so far are not surprising to me personally.

Thank you for the effort and I hope you can expand on this in the future.

 

[83DIZ65]

Informative.....I enjoy seeing this kind of analysis.....To be honest I'm pretty confident the best way to go is to be a middle of the pack spender as it gives a good chance to win and allows for saving some money.

 

[Goat Boy]

Whoa! Awesome analysis Stef! It definitely makes sense, as I think intuitively we know that most teams are fairly close in ability, with only a small number of teams being truly ahead of the curve (all of which usually spend to the cap).

 

[Scrub*]

nice post.

 

[Neiler]

Rare posts like this are the one of the reasons I still read these boards after several years. Excellent work.

 

[Back in Black]

Too many words! You lost me at Eugene Melnyk

 

[Micklebot]

Informative.....I enjoy seeing this kind of analysis.....To be honest I'm pretty confident the best way to go is to be a middle of the pack spender as it gives a good chance to win and allows for saving some money.

The reality is, until your core talent is ready, there is no benefit to bringing in those final pieces that push your salary into the top third.

A team like Buffalo doesn't benefit from going crazy at UFA time or trading for big contracts, because their core isn't there yet, but when a team like LA went from 17th in Payroll to 7th the next year and won the cup, it was more representative of them realizing their core players were ready, and spending the extra money to push them over the edge. Spending money didn't make LA a contender, developing their talent did, and spending more money pushed them over the top.

 

[StefanW]

Thank you all for all of the kind words. I was not sure how this would be received, and the positive responses motivate me to go into more depth and into tougher areas of analysis.

The reality is, until your core talent is ready, there is no benefit to bringing in those final pieces that push your salary into the top third.

A team like Buffalo doesn't benefit from going crazy at UFA time or trading for big contracts, because their core isn't there yet, but when a team like LA went from 17th in Payroll to 7th the next year and won the cup, it was more representative of them realizing their core players were ready, and spending the extra money to push them over the edge. Spending money didn't make LA a contender, developing their talent did, and spending more money pushed them over the top.

I can actually test this theory once I have more cases. I can separate out teams that make big payroll moves up or down versus staying relatively pat in relation to other teams, and see if that increases their level of success.

 

[Erik Alfredsson]

*Sigh* No he didn't say he's not going to spend, he said he is not going to spend right now, just for the sake of spending, which in that case, I'm totally on board being patient and not spending just for the sake of spending. Like holy crap, some people think that if we spend all our problems will magically go away. That's not how you build a hockey team, you don't spend to spend, you spend to build a good team, an Eug and Murray will do that when the time or opportunity is right.

 

[mat_sens]

Thanks, that was a great read.


It makes me remember the good old days of quantitative research in University. Ohhh SPSS.

 

[StefanW]

Thanks, that was a great read.


It makes me remember the good old days of quantitative research in University. Ohhh SPSS.

Thanks

I actually put all of the numbers into SPSS to do the analysis. When I first learned it the program was DOS based and there was no windows interface, so I had to do everything using syntax. Things are so much easier now.

 

[booger]

Thanks, that was a great read.


It makes me remember the good old days of quantitative research in University. Ohhh SPSS.

I prefer SAS. Great post Stef!

 

[Micklebot]

Thank you all for all of the kind words. I was not sure how this would be received, and the positive responses motivate me to go into more depth and into tougher areas of analysis.




I can actually test this theory once I have more cases. I can separate out teams that make big payroll moves up or down versus staying relatively pat in relation to other teams, and see if that increases their level of success.

I'm not sure you can come to anything conclusive, because you still need to spend that extra money wisely. Had LA just signed Clarkson at 5 mil, and Wiess at another 5mil, they wouldn't have been any more successful.

I guess with enough data, you can see the forest through the trees and bad signings will get lost in the data though.

 

[thinkwild]

Nice stuff. I have read other tests to measure the correlation coefficient for payrolls and performance using winning percentage for performance. It was done first for baseball but has applied across all sports, though mostly pre-cap. And it had been found that there is a moderate correlation. However a moderate correlation would be expected.

Another test was the Granger Causality Test, which rather than measure causality, attempts to measure precedence. Does one one time series variable consistently and predictably change before the other. If A does occur before B, it still doesnt prove A causes B, however it would seem safe to rule out B causing A.


So they tested the hypothesis, does payroll predictably rise before performance. Three leading sports economists made the measurements of team points regressed against lagged team points and payrolls, and then the other way around.

What they found was that winning percentages Granger caused higher payrolls. However higher payrolls didnt Granger cause higher winning percentages. By Granger caused, is meant one delta consistently, predictably and statistically significantly precedes the other.

In other words,
- winning always precedes increases in payroll.
- but increasing payroll doesnt always precede winning


Their conclusion was (prior to the salary cap) that high payrolls are not a necessity in the NHL. The trend is to more competitive balance, and the current competitive balance encourages the bidding up of players because of the playoff profits which show no sign of imbalance because of money. High payrolls did not mean you would do well in the playoffs, nor did it mean you would make money. Team performance Granger causes relative Team payroll, but the opposite hypothesis can be rejected.


Payroll disparity has never been the cause of an uncompetitive league; its the effect of a healthy one.

If we are to be a contender, some of our young players will have to earn top dollar contracts, or we havent been built right. You'll never be able to buy or trade for enough of those players at once. That is why development is so important. There is no shortcut. No quick rebuilds. Just patience and great drafting. And when this team proves they are good enough to go against the best teams as a pick'em, then the time for spending will be right.

But we cant spend to try and get to the top, only over it once we're at the top. Otherwise, i think we are trying to take shortcuts.

 

[DrEasy]

4) Rank ordering teams in this way is not ideal because the gap, for example, between first and second place spending teams may be greater than the gap between the 29th and 30 th place spending teams. This leads to pretty wonky variances.

Instead maybe you could, for each team, calculate what percentage of the top spending team (for that year) they've spent, and plot that relative to (or example) the points total in the regular season?

 

[BankStreetParade]

Thanks for having a look, and feel free to make comments, criticisms, and suggestions. I slapped this together fairly quickly, and accept (and even expect) that I may have made embarassing mistakes along the way.

I really enjoyed your analysis. Here's my feedback:

If you wanted to measure the cost per point theory why didn't you compile a list that included the total dollars spent by each team for x number of years as well as the number of points they accrued in that same time and come up with a $/pt figure that could have been sorted from highest to lowest? And then charting the accomplishments of that team with regard to playoff, playoff wins, Cups, etc.

Another thing I didn't think was efficient was grouping the tiers by 10s. Although you addressed it, the variance between the first and last team in each tier is too great I think to account for an actual result that shows spending correlates with winning. For instance, last year, the Flyers spent ~$72.5 million (the highest spending team) while the 10th highest spending team was the Lightning at ~$64.1 million. The difference of ~$8.4 million is too high in my opinion to just lump the 10 highest teams together and present their results. As another example, that same gap in spending covers the jump from the 5th highest spending team (Chicago) to the 17th highest spending team (New Jersey).

 

[Icelevel]

Melnyk is quoted somewhere recently (in the last yr?) as saying it is no secret that the teams that spend are the ones likely to go far. he said that the stats don't lie. And he said he was well aware of this.

Given that, there is an optimal time to spend. A rebuilding team may not want to spend because of low expectations/ability/maturity AND a rebuilding team may not want to succeed(or get in at 8th seed) so that they continue to pick high.

Once the rebuild is over i guess this is where the analysis becomes relevant and what you are focusing on.

We are assuming Ottawa's rebuild is over now i guess. Now is when we should see spending start to climb.

 

[Karl Eriksson]

As the great biggie smalls found following his statistical analysis: "mo money mo problems"

 

[Benjamin]

Is this cap numbers from pre-trade deadline or after/playoffs?

Because the numbers could be a bit skewed by sellers and buyers.

A team thats top 10 in spending could sell off and get to the bottom 10. Or vice versa.

 

[dingbatz]

For those who haven't seen it, this is a nice visual:

http://hockeyimpact.darkhorseanalytics.com/

 

[Holdurbreathe]

The reality is, until your core talent is ready, there is no benefit to bringing in those final pieces that push your salary into the top third.

A team like Buffalo doesn't benefit from going crazy at UFA time or trading for big contracts, because their core isn't there yet, but when a team like LA went from 17th in Payroll to 7th the next year and won the cup, it was more representative of them realizing their core players were ready, and spending the extra money to push them over the edge. Spending money didn't make LA a contender, developing their talent did, and spending more money pushed them over the top.

100% agree.

While I enjoyed the OP's analysis what it fails to show is the deviation in the cost of the core groups of the teams in the comparison.

I believe what is missed in the cost to winning discussion is player quality.

IMO the core groups of Pit, LA, Chi, etc are considerable more expensive than Ottawa's and for the most part younger and better.

People tend to forget how many lean years many of these teams suffered through before drafting early and often over several years building their current core groups.

Once that process is complete, players like Malkin, Crosby, Toews, Kane, etc aren't cheap to retain.

 

[Nac Mac Feegle]

Spending wisely is the key.

One or two pieces away from being a legit contender and a reasonably affordable piece is out there for the taking? Spend away.

Need to re-sign the developing youngsters whoa re the core? Give 'em a reasonable raise.

Throwing a pile of money at the feet of any UFA that will sign with us so we can go to the media and show what big spenders we are? Forget it!

 

[StefanW]

There are many great replies and comments, and so I am snipping away anything I am not directly responding to.

I'm not sure you can come to anything conclusive, because you still need to spend that extra money wisely.... (snip)...I guess with enough data, you can see the forest through the trees and bad signings will get lost in the data though.

No team goes into a contract thinking that it is bad, and no team offers up a deal that is well in excess of what a player can get in the open market. I get what you are saying and it is correct. I would just add that whether spending is wise has a random component to it. Thank you very much for this food for thought.

Another test was the Granger Causality Test, which rather than measure causality, attempts to measure precedence.
(snip)
What they found was that winning percentages Granger caused higher payrolls. However higher payrolls didnt Granger cause higher winning percentages. By Granger caused, is meant one delta consistently, predictably and statistically significantly precedes the other..

Thanks for this. I was not familiar with the Granger test or how it has been used in sports. I'm more familiar with Cox Regression analysis as a way of using time sequence data to build up a predictive model. I wish I could comment more, but I'm going to have to do a bunch of reading first Very interesting.

Instead maybe you could, for each team, calculate what percentage of the top spending team (for that year) they've spent, and plot that relative to (or example) the points total in the regular season?

Yes, that is a good way to approach it. I did a rush job, and I definitely will explore this option as a way of fine tuning the analysis. Very good point.

If you wanted to measure the cost per point theory why didn't you compile a list that included the total dollars spent by each team for x number of years as well as the number of points they accrued in that same time and come up with a $/pt figure that could have been sorted from highest to lowest? And then charting the accomplishments of that team with regard to playoff, playoff wins, Cups, etc.

Another thing I didn't think was efficient was grouping the tiers by 10s. (snip).

Yeah, your first comment exactly describes the approach I was going to use first when doing the cost per point analysis. Great minds think alike

Your comment about the tiers of ten is completely correct and I could not agree more. I was stuck with it this time around because I only had 6 years of data and I did not want my cell sizes to get too small for the analysis. I saw small cell as the greater evil because my SE was already too big for my liking. You are bang on correct to criticize this decision though.

Given that, there is an optimal time to spend. A rebuilding team may not want to spend because of low expectations/ability/maturity AND a rebuilding team may not want to succeed(or get in at 8th seed) so that they continue to pick high.

I would dispute that any team wants to intentionally finish low. However, the comment I bolded about an optimal time to spend is definitely something that I can figure out in the next stage of analysis. My assumption, and from what I see in your comment yours as well, is that there is an optimal time to spend. If there is we should see it in the relationship between a dramatic increase in payroll and the level of success of that team.

As the great biggie smalls found following his statistical analysis: "mo money mo problems"

He is a wise, wise man.

Is this cap numbers from pre-trade deadline or after/playoffs?

It is total yearly payroll, and no distinction is made for when cash is spent. I did think of the exact point you are making, and I came to the conclusion that increases in payroll at the trade deadline are usually pretty small because you only pay about 20% of full contract value. I admit up front though that if there is an issue here it is unlikely that I will be able to adequately fix it, because the sample size is too small to fine tune in this way. Very insightful comment.

For those who haven't seen it, this is a nice visual:

http://hockeyimpact.darkhorseanalytics.com/

That is a great visual! I love how you can change your anchor points. I was toying with the idea of building a macro for this in Excel, but that is low priority because I wouldn't be able to share it online anyway.

While I enjoyed the OP's analysis what it fails to show is the deviation in the cost of the core groups of the teams in the analysis.

I believe what is missed in the cost to winning discussion is player quality.

IMO the core groups of Pit, LA, Chi, etc are considerable more expensive than Ottawa's and for the most part younger and better.

People tend to forget how many lean years many of these teams suffered through before drafting early and often over several years building their current core group.

Once that process is complete, players like Malkin, Crosby, Toews, Kane, etc aren't cheap to retain.

As a general point, the SD of salary was not included because I did not input the exact salaries prior to running these early tests. But your first comments is more specific. Defining core players is subjective, and as such it is impossible to accurately include this type of variable for 30 teams over the course of six years. For instance, is Philllips in our core group? I would say yes, another researcher trying to replicate my results could easily say no.

The other factor to consider with core group is that even teams who have no had a lot of success can have expensive core groups. Winnipeg extended a pile of player to long expensive contracts this past summer even though they did not win much and did not make it into the playoffs. I would argue that all teams have core guys, and their cost is just rolled into the larger team salary.

Your point about core groups becoming expensive to retain is well made and I accept that logic. When a team has playoff success it is logical to assume that a bunch of players will ask for raises. This can lead to teams dumping players (like Chicago did a few years ago with Buff and Ladd), so the line between winning and payroll may not be all that clear. It is definitely something that would be fun to test and play with if I can get several more years worth of data. Thanks for taking the time to put together a well thought out response, and I really appreciate your insightful critique.