Adjusted stats - how valuable?

[Iain Fyffe]

Which is exactly what happens when you try to answer the simple question of who scored more points in the 90's with Adjusted Stat totals.

That is a simple question. It's also a pretty uninteresting one, because it ignores so much context. That is, the answer is easy but not very meaningful. If you want a better answer - a more meaningful answer - you can use adjusted stats.

1991 was quite different from 1999, and if you want to pretend that they're not different, that a point scored in 1991 means exactly the same with respect to a player's offensive ability as a point scored in 1999, you're free to do so. Just don't expect rational analysis to agree with you.

 

[Czech Your Math]

When you compare Crosby to OV since the LO, you don't convert it to AS's.
When you compare Gretzky to everyone else in the 80's, you don't convert it.

I do.

Why would you, you already have it EXACTLY HOW IT HAPPENED!!!
There's no need to guess or estimate or adjust, no need what so ever!!!

You seemed really concerned about factors that could potentially cause top tier players to have an advantage against lesser players, compared to during other seasons. Yet when simple math tells us that the same % advantage will yield a higher margin in raw points in seasons where scoring is substantially higher, you completely ignore it.

Just admit you were getting a little silly there, that sometimes you have this need to do more with the numbers than needs to be and move on.

Hey, I like to get silly... somebody, stop me!... but I have been objective.

See, once again, here you are implying that you can only use AS's or Raw stats and not both. That's the difference between you and I, I don't replace raw stats with AS's or vice-verse. I USE IT ALL to find a conclusion.

AS gives us the same info as raw stats about the season, except it gives us more information about the relative value (even in comparison to other season). You can use scissors to mow a large lawn, but those who understand how a lawnmower works probably don't think about which to do the same work in less time.

No, what you do is take AS's and hold to it like glue, no matter what any other info is saying including out right throwing away any info that isn't "in line" with your original AS's conclusion.

Simply this...the value or weight you assign to AS's(or any info for that matter) in any given situation should be determined by how it fits with all the other information.
If AS's is conflicting with all the other info, then its weight has to be reduced. Same if it was the raw point total in conflict.

You don't seem to consider the possibility that conflicts between raw and/or adjusted stats and other means of evaluation may be due to the other means being in error. Why would one have to reduce the importance of AS for evaluation at those times that one deems the other data in error? If the other means have no adjustment for context, the potential for error when comparing to other seasons is very substantial (in some cases, quite likely IMO). I don't feel obligated to weight Morenz's awards or finishes on the same level as Stan Mikita's. I'm not preventing anyone else from doing so, but shouldn't prevent me from stating my reasons for evaluating the context differently than others.

There's other factors at work besides just the numbers.

I agree, but until further research allows analysis of most/all of the most important potential factors, it's difficult to arrive at a better estimate of the quality/difficulty of different player-seasons.

Like during the height of the DPE, you had Jagr finishing ahead by large margins because he was so big and strong, all the clutching and grabbing had a lot less effect on him.
Take away the clutching and grabbing from that time, scoring goes up and suddenly those margins of Jagr become much smaller.
Jagr is not going to increase his points by the same amount that say a Kariya will because the C&G going on was a much bigger factor on Kariya's point totals than it was for Jagr's.
The value that is being assigned to Jagr in one of those heavy DPE is accurate for that season and anyone you bring into that season should have to deal with that value of Jagr's.
However, that value shouldn't carry him to 160 or 170 points in 1990 because the advantage he has on most other players in those DPE years doesn't translate to simply an increase in overall league scoring.
He was already ahead because the reason why those years were lower scoring had a lot less affect on him to begin with.
You have to temper those AS numbers with reality sometimes and you know as well as I do that some of those DPE number values are just whack. Not completely whack but there's some major inflation going on.

You're making a large assumption based on one potential factor. While I can see why players like Kariya may have been hurt more in % terms by the lack of rules enforcement, it's by no means proven that he did. First, Jagr was very skilled as well, so any decrease in scoring due to negation of skill would hurt him as much or more than anyone. Second, using the beloved "eye test", I would guess that no one drew as many uncalled penalties as Jagr during his prime, in part due to his size and strength (and that he never dove). Third, if we're estimating how he would have done in the 80s, he would be hurt less (even in % terms) than other stars by the fact that the 80s had a larger % of goals scored at even strength, where Jagr was especially effective. This generally hurt him in an era with more PPs, but would be to his advantage in the 80s, when teams scored a lower % of goals on special teams.

I have seen data that supports your conclusion, but until this conclusion can be verified and the individual factors can be better ascertained, further adjustment for quality/difficulty is not known to be necessary, nor can it be fairly applied if it is. I've mentioned a number of factors ITT that could be partially responsible, some of which would require no adjustment and some of which would help/hurt some players much more than others. Until these and other factors can be explore further, further adjustment may cause as much or more additional distortion as it does clarify things.

 

[Rhiessan71]

Whose was more valuable offensively during that time? Jagr

Just to be straight, that value is measured by said players contribution to their team winning right?

So if I were to tell you that Gretzky's contribution to his team winning in 95/96 was almost equal to Jagr's contribution in 95/96.
Even though Gretzky only scored 102 points while Jagr scored 149, would you agree or disagree with that?

Or if I told you that Gretzky's 90 points in 97/98 was more valuable to his team winning than Jagr's 102 points were to his team the same year?

I mean as long as we are strictly talking value and only what the numbers say right?

Gretzky had more "value" to his team winning in 4 out of the 7 years (90/91, 91/92, 93/94 and 97/98) that they each played a full season (Gretz only played 44 in 92/93 and Jagr only played 63 in 96/97).

In fact out of all 9 years, only once did Jagr contribute substantially more value to his team winning than Gretzky and that was Gretzky's final year in the league.

Points/Total Team Goals
90/91 Jagr:57/342 16.67% Gretz:163/340 47.94%
91/92 Jagr:69/343 20.12% Gretz:121/287 42.16%
92/93 Jagr:94/367 25.61% Gretz:65/338 19.23%
93/94 Jagr:99/299 33.11% Gretz:130/294 44.22%
94/95 Jagr:70/181 38.67% Gretz:48/142 33.80%
95/96 Jagr:149/362 41.16% Gretz:102/256 39.84%
96/97 Jagr:95/285 33.33% Gretz:97/258 37.60%
97/98 Jagr:102/228 44.74% Gretz:90/197 45.69%
98/99 Jagr:127/242 52.48% Gretz:62/217 28.57%

I can play the "value" game too.

I have seen data that supports your conclusion, but until this conclusion can be verified and the individual factors can be better ascertained, further adjustment for quality/difficulty is not known to be necessary, nor can it be fairly applied if it is. I've mentioned a number of factors ITT that could be partially responsible, some of which would require no adjustment and some of which would help/hurt some players much more than others. Until these and other factors can be explore further, further adjustment may cause as much or more additional distortion as it does clarify things.

Wait, wait, wait.
Can you show me where Adjusted Stats results have been verified?
Show me the same level of proof and verification for Adjusted Stats that you're asking me to provide and verify here.
Are you actually asking me to provide 100% proof when you can't prove Adjusted Stats are 100%?

 

[Czech Your Math]

Just to be straight, that value is measured by said players contribution to their team winning right?

So if I were to tell you that Gretzky's contribution to his team winning in 95/96 was almost equal to Jagr's contribution in 95/96.
Even though Gretzky only scored 102 points while Jagr scored 149, would you agree or disagree with that?

Or if I told you that Gretzky's 90 points in 97/98 was more valuable to his team winning than Jagr's 102 points were to his team the same year?

I mean as long as we are strictly talking value and only what the numbers say right?

Gretzky had more "value" to his team winning in 4 out of the 7 years (90/91, 91/92, 93/94 and 97/98) that they each played a full season (Gretz only played 44 in 92/93 and Jagr only played 63 in 96/97).

Once you reduce it to a team context, a whole host of other factors enter the picture. For instance, in '98 Gretzky's team had 428 GF/GA while Jagr's team had 416. So Jagr's team played in a slightly lower scoring context than Gretzky's... despite Jagr increasing the scoring context more than Gretzky did. That also doesn't figure the quality of the lower lines, the quality of the defense, etc. Really, ES goals and special teams goals should be separated as well.

 

[Iain Fyffe]

In fact out of all 9 years, only once did Jagr contribute substantially more value to his team winning than Gretzky and that was Gretzky's final year in the league.

Points/Total Team Goals
90/91 Jagr:57/342 16.67% Gretz:163/340 47.94%
91/92 Jagr:69/343 20.12% Gretz:121/287 42.16%
92/93 Jagr:94/367 25.61% Gretz:65/338 19.23%
93/94 Jagr:99/299 33.11% Gretz:130/294 44.22%
94/95 Jagr:70/181 38.67% Gretz:48/142 33.80%
95/96 Jagr:149/362 41.16% Gretz:102/256 39.84%
96/97 Jagr:95/285 33.33% Gretz:97/258 37.60%
97/98 Jagr:102/228 44.74% Gretz:90/197 45.69%
98/99 Jagr:127/242 52.48% Gretz:62/217 28.57%

You misunderstand. It refers to absolute value, not value relative to teammates.

I can play the "value" game too.

Hey, how about you quit the games and bring some discussion?

 

[Czech Your Math]

Wait, wait, wait.
Can you show me where Adjusted Stats results have been verified?
Show me the same level of proof and verification for Adjusted Stats that you're asking me to provide and verify here.
Are you actually asking me to provide 100% proof when you can't prove Adjusted Stats are 100%?

I didn't ask you to prove or verify anything in that post, I was talking about how AS may possible advance for comparison purposes.

Listen, most likely neither of us is going to substantially change how we evaluate players. Discussion such as these are often most beneficial to others with perhaps a more open mind on these matters. Maybe some are unsure how to proceed... use raw stats and try to make numerous mental adjustments themselves... or use adjusted stats and make few (or no) further adjustments,

It's like if someone has a very vague idea of where they're going, but really don't know how to get there. Which is going to help them more?

Your method is like telling them "go down several blocks, make a right at a coffee shop, go about a half mile, make a left a block past the convenience store, go a couple miles, make a right across from the gas station with the guy holding a sign out front... etc., etc... then make a quick right at the burger stand, and it's three blocks down on the right." It may be a great route for some people, if they don't miss the important landmarks, otherwise they could get lost and have no idea where they're going.

I think the method which I and others use is a lot more explicit. It's like giving someone a map with the route highlighted, so that if they know where they're going and can see how to adjust their route if they make a wrong turn.

Perhaps in each case, we can only narrow it down to a block or two, and that person will have to take it from there.

 

[Rhiessan71]

I didn't ask you to prove or verify anything in that post, I was talking about how AS may possible advance for comparison purposes.

Listen, most likely neither of us is going to substantially change how we evaluate players. Discussion such as these are often most beneficial to others with perhaps a more open mind on these matters. Maybe some are unsure how to proceed... use raw stats and try to make numerous mental adjustments themselves... or use adjusted stats and make few (or no) further adjustments,

It's like if someone has a very vague idea of where they're going, but really don't know how to get there. Which is going to help them more?

Your method is like telling them "go down several blocks, make a right at a coffee shop, go about a half mile, make a left a block past the convenience store, go a couple miles, make a right across from the gas station with the guy holding a sign out front... etc., etc... then make a quick right at the burger stand, and it's three blocks down on the right." It may be a great route for some people, if they don't miss the important landmarks, otherwise they could get lost and have no idea where they're going.

I think the method which I and others use is a lot more explicit. It's like giving someone a map with the route highlighted, so that if they know where they're going and can see how to adjust their route if they make a wrong turn.

Perhaps in each case, we can only narrow it down to a block or two, and that person will have to take it from there.

No, no, you said you have seen data that supports my conclusion but that it needs to be verified.
So all I'm asking is for you to show me the level Adjusted Stats have been verified so I know what I need to shoot for.

All through this thread, every time I brought up a reasonable point, you have responded by telling me that while it sounds reasonable, it would have to be verified first.
Show me how well Adjusted Stats have been verified so I can attempt to do the same.

 

[Czech Your Math]

No, no, you said you have seen data that supports my conclusion but that it needs to be verified.
So all I'm asking is for you to show me the level Adjusted Stats have been verified so I know what I need to shoot for.

I meant that I've seen data suggesting adjusted PPG may have been easier for top tiers than lower tiers in DPE compared to 80s. However, if this is due to disproportionate influx of high quality Euros, then it may not require further adjustment (i.e. to further adjust when unjustly penalize DPE players). If it's generally easier, but it was also easier to stay healthy in the 80s, then these factors both would require adjustment (and may offset in many cases). If it's due to increased PP opportunities, then it would be fairest to adjust based on how much each player relied on PP points vs. ES (or something similar to that). These are just some of the potential factors and they can combine or offset as they change.

If I've asked anything of you, it's that you understand, if not agree with the perspective that adjusted stats are an improvement over raw data, because they provide more information while maintaining the same essential in-season rankings and proportions (the latter esp. if done "simply" and properly).

 

[Czech Your Math]

All through this thread, every time I brought up a reasonable point, you have responded by telling me that while it sounds reasonable, it would have to be verified first.
Show me how well Adjusted Stats have been verified so I can attempt to do the same.

There is really no verification needed for the valuation part of AS, because it's axiomatic... it's already inherently built into the formula.

That's like saying if X = 2, that we need to verify that 4*X = 8.

Yes you can verify that, but it's automatically true.

What needs to be studied more and verified/quantified IMO is how difficult it was to reach a certain valuation in different seasons for different quality players. Much of the confusion is also due to the apparent misconception that NHL seasons do not vary in terms of quality, total talent, per-team talent, etc. If all the potentially influential factors remained constant from season to season, then determining which player had the more valuable season would be the same as which player had the "better" or "higher quality" season in terms of difficulty. However, they don't remain constant, so it's not the same, and it's translating value into quality. When the effective total talent pool was smaller and/or the per team quality is lower, then it's like a "clearance sale" where a fixed level of adj. production (i.e. value) buys a lot of value (the good players are comparatively wealthier). When the total talent pool increases and/or the per team quality increases, then it's tough to find any bargains, because a fixed level of adj. production (value) buys less than during sales.

 

[Iain Fyffe]

Show me how well Adjusted Stats have been verified so I can attempt to do the same.

"Verifying" adjusted stats makes no sense. It's not a hypothesis or a theory, it's a straightforward mathematical adjustment.

Saying that players of a certain tier are affected differently by scoring changes than others needs to be verified. Dividing by 6.5 and then multiplying by 6.2 does not.

 

[Rhiessan71]

If I've asked anything of you, it's that you understand, if not agree with the perspective that adjusted stats are an improvement over raw data, because they provide more information while maintaining the same essential in-season rankings and proportions (the latter esp. if done "simply" and properly).

And...as I have informed you countless times now, I do NOT accept AS's as a replacement for raw stats and never will.
I use them BOTH and will continue to use them BOTH.
I don't think the Raw numbers are right but I also don't think the AS's numbers are right either. I believe the real answer is somewhere between the two, it's just a matter of which way to lean, which stat gets more weight in each comparison.
The weights do not remain constant.

"Verifying" adjusted stats makes no sense. It's not a hypothesis or a theory, it's a straightforward mathematical adjustment.

Ahh yes, the math isn't flawed, just the assumption that all players are equal and all players are affected equally in different scoring environments that is.

Saying that players of a certain tier are affected differently by scoring changes than others needs to be verified. Dividing by 6.5 and then multiplying by 6.2 does not.

No, sorry, it has already been presented, proven and verified in the link I supplied earlier to Overpass's study.
That's not the issue, the issue is that no one has figured out exactly how to incorporate it globally yet.
Just like what I said earlier about DPE inflation of value. Most rational people know it's there and there is info to back it up. It has been verified. It's not a matter of whether there is weight to it or not, there is. It's a matter of how much weight it should have.

All of which is missing(among other factors) when someone decides they are going to spout Adjusted Stats at face value as a final answer.

 

[Iain Fyffe]

Ahh yes, the math isn't flawed, just the assumption that all players are equal and all players are affected equally in different scoring environments that is.

That assumption is not made. Adjusted stats does not make any assumption about who would score what if they actually played in another scoring environment. It's just a scaling of the actual results in the actual games they played, a transformation of the curve to make it comparable to other seasons in a systematic way. It does not say "this is how someone would have scored has they played at another time" it says "this is how impressive this guy's totals are compared to other players in other times."

No, sorry, it has already been presented, proven and verified in the link I supplied earlier to Overpass's study.

You misunderstand again. I didn't say it needs to be verified in the sense that it hasn't been yet. I'm saying it is something that needs to be verified before being used. Until something like this is verified it's a conjecture.

All of which is missing(among other factors) when someone decides they are going to spout Adjusted Stats at face value as a final answer.

This tends to get lost in all the inaccurate statements made about what adjusted scoring does in this thread, including some made by you.

 

[Dalton]

I already have, despite the fact that the onus of proof is on you. You're making the claim that adjusted scoring normalizes player scoring results. This is your assertion. Prove your assertion right.

I have already provided mathetical, logical and graphic evidence that adjusted scoring does not normalize player scoring results.

Prove your assertion right.

I can save you some time: you cannot prove it right, because it is wrong. But so long as you continue to make this unfounded assertion, I will continue to point out how wrong it is.

Why is the onus on me? You haven't proved anything. AS fans as a whole just keep repeating the same assertions over and over again, ignoring arguments to the contrary.

You are averaging seasons. That ignores the impact of outliers. This is an error. Prove it isn't. The onus is on you. I've proven my point just by pointing to the study.

 

[Dalton]

Your error is in believing that this process results in normalization. It does not.

You seem to think that by using any kind of average in a calculation, you are normalizing the results. This is simply not true. Applying a flat multiplier, as adjusted scoring does, does not change the essential shape of the curve.

No your error is that the process includes normalization at a very fundamental level. You average seasons. You seem to think that averaging seasons is a loophole. It is not.

Outliers have a big effect on averages. Outliers drive the averages. You don't seem to grasp the meaning of that. I wrote a very long post describing this effect with illustrations. Of course I got no response. Cherry picking points and posts is disingenuous. I know it's hard work and we have lives but I don't see this debate being taken seriously at times.

 

[Dalton]

Take your own advice. The study to which you continue to refer makes two essential points:

1. That outliers should not be automatically be dimissed when analyzing human performance.

2. That human performance generally does not follow a bell curve, but instead a power law curve.

Neither of these points are relevant to adjusted scoring, because adjusted scoring neither ignores outliers nor assumes a bell curve. But of course, I've said this over and over and over again. Perhaps this time you'll take note of it?

If you want to show that this study is relevant to the discussion, you have to demonstrate that adjusted scoring ignores outliers (it doesn't, players are adjusted based on their actual stats regardless of whether they might appear to be outliers or not) and that it assumes a bell curve (it doesn't, since the same essentially flat multiplier is applied to all players in a season). Until you demonstrate these two things (which you cannot, since they are not true), this study is utterly irrelevant to this discussion, and you should stop trying to use it to support your position.

It's not about a literal bell curve. Its about a thinking pattern that assumes it. When you average seasons you are assuming normalcy. You are ignoring the impact of outliers. Why do you think averaging seasons is exempt?

Your argument suggests that you haven't read, didn't understand or read selectively the study in question. Here are the answers you seek (in part)-

"Regarding performance measurement and management, the current zeitgeist is that the median worker should be at the mean level of performance and thus should be placed in the middle of the performance appraisal instrument. If most of those rated are in the lowest category, then the rater, measurement instrument, or both are seen as biased (i.e., affected by severity bias; Cascio & Aguinis, 2011 chapter 5). Performance appraisal instruments that place most employees in the lowest category are seen as psychometrically unsound. These basic tenets have spawned decades of research related to performance appraisal that might “improve” the measurement of performance because such measurement would result in normally distributed scores given that a deviation from a normal distribution is supposedly indicative of rater bias (cf. Landy & Farr, 1980; Smither & London, 2009a). Our results suggest that the distribution of individual performance is such that most performers are in the lowest category. Based on Study 1, we discovered that nearly two thirds (65.8%) of researchers fall below the mean number of publications. Based on the Emmy-nominated entertainers in Study 2, 83.3% fall below the mean in terms of number of nominations. Based on Study 3, for U.S. representatives, 67.9% fall below the mean in terms of times elected. Based on Study 4, for NBA players, 71.1% are below the mean in terms of points scored. Based on Study 5, for MLB players, 66.3% of performers are below the mean in terms of career errors. Moving from a Gaussian to a Paretian perspective, future research regarding performance measurement would benefit from the development of measurement instruments that, contrary to past efforts, allow for the identification of those top performers who account for the majority of results. Moreover, such improved measurement instruments should not focus on distinguishing between slight performance differences of non-elite workers. Instead, more effort should be placed on creating performance measurement instruments that are able to identify the small cohort of top performers."

You also benefit from reading this which explicitly refers to analyzing whole industries with respect to individual performance-averaging seasons-

"There are important differences between Gaussian and Paretian distributions. First, Gaussian distributions underpredict the likelihood of extreme events. For instance, when stock market performance is predicted using the normal curve, a single-day 10% drop in the financial markets should occur once every 500 years (Buchanan, 2004). In reality, it occurs about once every 5 years (Mandelbrot, Hudson, & Grunwald, 2005). Second, Gaussian distributions assume that the mean and standard deviation, so central to tests of statistical significance and computation of effect sizes, are stable. However, if the underlying distribution is Paretian instead of normal, means and standard deviations are not stable and Gaussian-based point estimates as well as confidence intervals are biased (Andriani & McKelvey, 2009). Third, a key difference between normal and Paretian distributions is scale invariance. In OBHRM, scale invariance usually refers to the extent to which a measurement instrument generalizes across different cultures or populations. A less common operationalization of the concept of scale invariance refers to isomorphism in the shape of score distributions regardless of whether one is examining an individual, a small work group, a department, an organization, or all organizations (Fiol, O’Connor, & Aguinis, 2001). Scale invariance also refers to the distribution remaining constant whether one is looking at the whole distribution or only the top performers. For example, the shape of the wealth distribution is the same whether examining the entire population or just the top 10% of wealthy individuals (Gabaix, 1999). Related to the issue of scale invariance, Gabaix, Gopikrishnan, Plerou, and Stanley (2003) investigated financial market fluctuations across multiple time points and markets and found that data conformed to a power law distribution. The same distribution shape was found in both United States (U.S.) and French markets, and the power law correctly predicted both the crashes of 1929 and 1987.

Germane to OBHRM in particular is that if performance operates under power laws, then the distribution should be the same regardless of the level of analysis. That is, the distribution of individual performance should closely mimic the distribution of firm performance. Researchers who study performance at the firm level of analysis do not necessarily assume that the underlying distribution is normal (e.g., Stanley et al., 1995). However, as noted earlier, researchers who study performance at the individual level of analysis do follow the norm of normality in their theoretical development, research design, and choices regarding data analysis. These conflicting views, which may be indicative of a micro–macro divide in OBHRM and related fields (e.g., Aguinis, Boyd, Pierce, & Short, 2011), could be reconciled if individual performance is found to also follow a power law distribution, as it is the case for firm performance (Bonardi, 2004; Powell, 2003; "

Its like a fractal. Averaging seasons is not a free pass or a loop hole. Its the same error. This error is fundamental to AS. If you don't average seasons you don't have AS as it is now constructed.

There is a section on reccommended math technique. Linear regression is not among them. Bayesian is suggested as most promissing.

This doesn't even take into account the impact of outliers on the rest as I explained. Any student of the history of anything can clearly see the impact of outliers on peers. Without calculus there is no explosion of differential equations leading to questions that led to the study of complex numbers. Things build and grow on the past. You can't remove Gretzky or Keppler from history, you can't erase the impact of their achievements for the purpose of answering the question of how they'd fare today. You can only look at what they did and marvel at what they might do today.

What about eras without extreme outliers? How do you compare them to seasons or eras that have them? AS simply averages the extreme outliers downwards. We compare Gretzky , Orr and Howe to whoever is best at their job at the moment. Obvious flaw that doesn't need math to see. I discussed this issue yet it was ignored. The study proves that this thinking or methodology is flawed.

"In addition to the study of leadership, our results also affect research on work teams (e.g., group empowerment, shared efficacy, team diversity). Once again, our current understanding of the team and how groups influence performance is grounded in an assumption of normality. The common belief is that teamwork improves performance through increased creativity, synergies, and a variety of other processes (Mathieu, Maynard, Rapp, & Gilson, 2008). If performance follows a Paretian distribution, then these existing theories are insufficient because they fail to address how the presence of an elite worker influences group productivity. We may expect the group productivity to increase in the presence of an elite worker, but is the increase in group output negated by the loss of individual output of the elite worker being slowed by non-elites? It may also be that elites only develop in interactive, dynamic environments, and the isolation of elite workers or grouping multiple elites together could hamper their abnormal productivity. Once again, the finding of a Paretian distribution of performance requires new theory and research to address the elite nested within the group. Specifically, human performance research should adopt a new view regarding what human performance looks like at the tails. Researchers should address the social networks of superstars within groups in terms of identifying how the superstar emerges, communicates with others, interacts with other groups, and what role non-elites play in the facilitating of overall performance."

As things stand I have suggested that using fractions to look at raw data does a better job than AS. Others may have better suggestions.

 

[Dalton]

Your error was not realizing the other definiton of "means".



The idea Czech has behind using AS for direct comparison of seasons played by multiple players is predicated on the fact that average scoring fluctuates on a yearly basis.

Example:

Joe Smith scores 20 goals every year in three consecutive years.
Bob Jones scores 25, 25, and 10 in those same years.

They have the same overall totals, and we'll assume they had the same GP.

Adjusted stats provide us insight into whether it was harder to score goals in a given season, and might highlight one or the other a truly achieving more. In this example, if the third season has a higher league GPG than the first two, there's a good chance Bob Jones ends up with more AG.

And before you begin with the "See, I told you it was flawed!" this is no different than taking three random seasons out of Gretzky's career and comparing them to three random seasons out of Jagr's.

This would be fundamental error in thinking. This paragraph addresse that issue.

"As a second illustration of the implications of our results, consider the research domain of utility analysis in preemployment testing and training and development. Utility analysis is built upon the assumption of normality, most notably with regard to the standard deviation of individual performance (SDy), which is a key component of all utility analysis equations. In their seminal article, Schmidt et al. (1979) defined SDy as follows: “If job performance in dollar terms is normally distributed, then the difference between the value to the organization of the products and services produced by the average employee and those produced by an employee at the 85th percentile in performance is equal to SDy” (p. 619). The result was an estimate of $11,327. What difference would a Paretian distribution of job performance make in the calculation of SDy? Consider the distribution found across all 54 samples in Study 1 and the productivity levels in this group at (a) the median, (b) 84.13th percentile, (c) 97.73rd percentile, and (d) 99.86th percentile. Under a normal distribution, these values correspond to standardized scores (z) of 0, 1, 2, and 3. The difference in productivity between the 84.13th percentile and the median was two, thus a utility analysis assuming normality would use SDy= 2.0. A researcher at the 84th percentile should produce $11,327 more output than the median researcher (adjusted for inflation). Extending to the second standard deviation, the difference in productivity between the 97.73rd percentile and median researcher should be four, and this additional output is valued at $22,652. However, the difference between the two points is actually seven. Thus, if SDy is two, then the additional output of these workers is $39,645 more than the median worker. Even greater disparity is found at the 99.86th percentile. Productivity difference between the 99.86th percentile and median worker should be 6.0 according to the normal distribution; instead the difference is more than quadruple that (i.e., 25.0). With a normality assumption, productivity among these elite workers is estimated at $33,981 ($11,327 × 3) above the median, but the productivity of these workers is actually $141,588 above the median. We chose Study 1 because of its large overall sample size, but these same patterns of productivity are found across all five studies. In light of our results, the value-added created by new preemployment tests and the dollar value of training programs should be reinterpreted from a Paretian point of view that acknowledges that the differences between workers at the tails and workers at the median are considerably wider than previously thought. These are large and meaningful differences suggesting important implications of shifting from a normal to a Paretian distribution. In the future, utility analysis should be conducted using a Paretian point of view that acknowledges that differences between workers at the tails and workers at the median are considerably wider than previously thought."

 

[Dalton]

To summarize- averaging seasons assumes that there is an average performance that all players will be in range of. This is a proven methodological error when analyzing human performance. Outliers drive the averages. The number of outliers in a season varies, not to mention extreme outliers. The variance between groups of players varies season over season. The variance between players varies over seasons. Using a scalar does not account for this. Using a scalar dependant on averaging is a proven methodological error.

The value of a goal is dependant on its circumstances and certainly dependant on the team results. If a player scores 40% of his team's goals then obviously his goals have more value on the surface than a player who scores 10% of his team's goals. Using average goals per season doesn't address this at all. Let's talk game winning goals.

AS is so obviously just an attempt to predict what a player would score in a different season. Redefining it to combat criticism is inefffectual.

One variable in AS is average goals per season. That fails the smell test and it fails scientific scrutiny. AS may be a part of an argument about comparing players across eras if the parties agree on its value but it is in no way the final word. Its of very little value if the parties disagree on it value and probably takes the debate off topic.

Other variables used are also debatable.

I would conclude that AS has very limited value in any discussion.

 

[Bear of Bad News]

As things stand I have suggested that using fractions to look at raw data does a better job than AS. Others may have better suggestions.

SHOW IT. Exhibit it. Prove it.

Don't post pages and pages of academic text that may or may not have to do with your claim (keep in mind that I have a background in postgraduate mathematics, so if you've turned me off by this, that's a tremendous feat).

Show your claim. You're running out of chances.

 

[Iain Fyffe]

Why is the onus on me? You haven't proved anything. AS fans as a whole just keep repeating the same assertions over and over again, ignoring arguments to the contrary.

The irony is just dripping off of this paragraph.

Of course the onus is on you. You are making the specific claim that AS normalizes the stats. You need to prove that in order for it to be valid. (You cannot prove it, of course, because it is not true, as I have demonstrated logically, mathematically, and graphically.)

You are averaging seasons. That ignores the impact of outliers. This is an error. Prove it isn't. The onus is on you.

What does this even mean? How are seasons averaged? And even if seasons are averaged, that does not necessarily mean that seasons are normalized, which is what the study warns against. Averaging is not the same thing as normalization.

I've proven my point just by pointing to the study.

And as I've said over and over and over again, the main points of that study are irrelevant because AS does not do what it claims it does.

No your error is that the process includes normalization at a very fundamental level. You average seasons.

No you don't, and averaging is not normalization anyway. It's almost like you keep making the same assertions over and over again, ignoring arguments to the contrary. There's a large difference between ignoring an argument and rebutting it.

Outliers have a big effect on averages. Outliers drive the averages. You don't seem to grasp the meaning of that.

Presumably you're saying that outliers need to be considered when deriving an average, since the study admonishes not to automatically exclude outliers. Since AS does not do that, no problem?

Cherry picking points and posts is disingenuous. I know it's hard work and we have lives but I don't see this debate being taken seriously at times.

There's no debate. You make a point, it's rebutted. You make the same point, it's rebutted again. You make the same point again, and it's rebutted yet again. Etc. That's not a debate.

It's not about a literal bell curve. Its about a thinking pattern that assumes it.

That's a rather large shifting of the goalposts. So now when you say a bell curve, you don't actually mean a bell curve? If you don't actually mean a bell curve, then that study is going to be useless to you, since it discusses using actual bell curves.

When you average seasons you are assuming normalcy. You are ignoring the impact of outliers. Why do you think averaging seasons is exempt?

Seasons are not averaged with each other, and outliers are not excluded. So...?

Your argument suggests that you haven't read, didn't understand or read selectively the study in question. Here are the answers you seek (in part)-

Please. I've repeated the salient points of the study back to you many times now. For example:

"Regarding performance measurement and management, the current zeitgeist is that the median worker should be at the mean level of performance and thus should be placed in the middle of the performance appraisal instrument.

This is the first point. AS does not do this. As I demonstrated many pages ago, adjusted scoring results follow a power-law curve (they have to, because that's what the raw data does). This means that in adjusted scoring, the median perfomer is below the mean level of performance. Therefore AS follows along exactly with what the study says it should do.

You also benefit from reading this which explicitly refers to analyzing whole industries with respect to individual performance-averaging seasons-

"There are important differences between Gaussian and Paretian distributions.

No, see right here this part is talking about the difference betwen Gaussian (normal) and Paretian (power law) distributions. As such the comments only apply to a situation where a normal curve is assumed, which AS does not do. Therefore this is also irrelevant, as I have pointed out over and over and over....

So in summary, it's not that the evidence from the study is being ignored, it's that the evidence from the study has been responded to, rebutted, and demonstrated to be irrelevant to AS.

Averaging seasons is not a free pass or a loop hole. Its the same error. This error is fundamental to AS. If you don't average seasons you don't have AS as it is now constructed.

Nonsense (as far as I can tell). How does AS average seasons? I'll have to admit I'm not 100% sure I understand what you mean by "averaging seasons", as you've done a poor job explaining yourself.

You can't remove Gretzky or Keppler from history, you can't erase the impact of their achievements for the purpose of answering the question of how they'd fare today.

Ah, that's not what is being done. You're still stuck on the "magically transport to another time and place" interpretation of AS, which is of course incorrect. Another assertion that is made over and over, ignoring arguments to the contrary...

AS simply averages the extreme outliers downwards.

No it doesn't. In a season with low enough scoring, even an extreme outlier would be adjusted upwards. If Gretzky had scored 200 points in a 3.5 GPG league, and AS was using 4.0 GPG as a baseline, his totals would be adjusted upwards, despite his being an "outlier" (as you use the term, though he's not really an outlier since we have no sampling issues.)

In fact, you could set the arbitrary baseline number in AS so high that everyone in hockey history would be adjusted upwards, if you wanted to. Say you adjust everyone to a 20.0 goals-per-game standard. Outliers (if there really are any) would be adjusted upwards along with everyone else, because as I've said over and over again, everyone in the same season is adjusted in the same direction. And since everyone is adjusted in the same direction, there is no normalization.

 

[Iain Fyffe]

To summarize- averaging seasons assumes that there is an average performance that all players will be in range of.

AS does not do this. It assumes nothing about a range of performance, it merely uses the range of performance that actually occured.

The number of outliers in a season varies, not to mention extreme outliers.

I would say there are no outliers. Outliers are a sample that seems to be so high or so low as to be inaccurate. Since with AS we work with 100% of the data, we know that no one is an outlier, since we know it's what actually happened.

The variance between groups of players varies season over season. The variance between players varies over seasons. Using a scalar does not account for this.

Careful, you almost hit upon an actual flaw of AS there.

Notice how this has nothing to do with normalization or averaging. It has to do with the shape of the actual performance curve. The curve of AS results will always follow the curve of the raw results, because AS makes no adjustment to the shape of the curve.

But this "flaw" exists in the raw data as well, and therefore cannot be called a flaw of AS. It's an additional adjustment that can be made, and perhaps should be. But it does not mean that AS is fatally flawed, as you have been saying for some time.

AS is so obviously just an attempt to predict what a player would score in a different season. Redefining it to combat criticism is inefffectual.

Sorry, you're the one redefining it. I quoted the exact passage from the original source of AS. If you choose to ignore that, that's up to you, but don't expect anyone to take you seriously.

I would conclude that AS has very limited value in any discussion.

Sorry, that's your premise, not a conclusion.

 

[Dalton]

SHOW IT. Exhibit it. Prove it.

Don't post pages and pages of academic text that may or may not have to do with your claim (keep in mind that I have a background in postgraduate mathematics, so if you've turned me off by this, that's a tremendous feat).

Show your claim. You're running out of chances.

Love to. How about you start by showing me what I have to prove. AS supporters are wonderful at shouting prove it at late but woeful at stating precisely what must be proved. Its getting tiresome.

Keep in mind that I also have a background in post graduate math and a few years teaching it.

There is no doubt whatsoever that AS is doing exactly what this study is arguing against. I have stated that ad nauseum, giving multiple examples. I have also stepped outdide purely matheatical considerations in support of my argument.

How about composing your own argument and giving me something to debate?

 

[Iain Fyffe]

Love to. How about you start by showing me what I have to prove. AS supporters are wonderful at shouting prove it at late but woeful at stating precisely what must be proved. Its getting tiresome.

Prove that adjusted scoring normalizes. Don't pretend this is the first time this has been asked.

There is no doubt whatsoever that AS is doing exactly what this study is arguing against.

The study warns against assuming that human performance follows a bell curve. Adjusted scoring does not do this.

The study warns against automatically excluding outliers. Adjusted scoring does not do this.

I have stated that ad nauseum, giving multiple examples.

Agreed on the former, but not the latter. What examples have you given of normalization?

 

[Dalton]

The irony is just dripping off of this paragraph.

Of course the onus is on you. You are making the specific claim that AS normalizes the stats. You need to prove that in order for it to be valid. (You cannot prove it, of course, because it is not true, as I have demonstrated logically, mathematically, and graphically.)


What does this even mean? How are seasons averaged? And even if seasons are averaged, that does not necessarily mean that seasons are normalized, which is what the study warns against. Averaging is not the same thing as normalization.


And as I've said over and over and over again, the main points of that study are irrelevant because AS does not do what it claims it does.


No you don't, and averaging is not normalization anyway. It's almost like you keep making the same assertions over and over again, ignoring arguments to the contrary. There's a large difference between ignoring an argument and rebutting it.


Presumably you're saying that outliers need to be considered when deriving an average, since the study admonishes not to automatically exclude outliers. Since AS does not do that, no problem?


There's no debate. You make a point, it's rebutted. You make the same point, it's rebutted again. You make the same point again, and it's rebutted yet again. Etc. That's not a debate.


That's a rather large shifting of the goalposts. So now when you say a bell curve, you don't actually mean a bell curve? If you don't actually mean a bell curve, then that study is going to be useless to you, since it discusses using actual bell curves.


Seasons are not averaged with each other, and outliers are not excluded. So...?


Please. I've repeated the salient points of the study back to you many times now. For example:


This is the first point. AS does not do this. As I demonstrated many pages ago, adjusted scoring results follow a power-law curve (they have to, because that's what the raw data does). This means that in adjusted scoring, the median perfomer is below the mean level of performance. Therefore AS follows along exactly with what the study says it should do.


No, see right here this part is talking about the difference betwen Gaussian (normal) and Paretian (power law) distributions. As such the comments only apply to a situation where a normal curve is assumed, which AS does not do. Therefore this is also irrelevant, as I have pointed out over and over and over....

So in summary, it's not that the evidence from the study is being ignored, it's that the evidence from the study has been responded to, rebutted, and demonstrated to be irrelevant to AS.


Nonsense (as far as I can tell). How does AS average seasons? I'll have to admit I'm not 100% sure I understand what you mean by "averaging seasons", as you've done a poor job explaining yourself.


Ah, that's not what is being done. You're still stuck on the "magically transport to another time and place" interpretation of AS, which is of course incorrect. Another assertion that is made over and over, ignoring arguments to the contrary...


No it doesn't. In a season with low enough scoring, even an extreme outlier would be adjusted upwards. If Gretzky had scored 200 points in a 3.5 GPG league, and AS was using 4.0 GPG as a baseline, his totals would be adjusted upwards, despite his being an "outlier" (as you use the term, though he's not really an outlier since we have no sampling issues.)

In fact, you could set the arbitrary baseline number in AS so high that everyone in hockey history would be adjusted upwards, if you wanted to. Say you adjust everyone to a 20.0 goals-per-game standard. Outliers (if there really are any) would be adjusted upwards along with everyone else, because as I've said over and over again, everyone in the same season is adjusted in the same direction. And since everyone is adjusted in the same direction, there is no normalization.

Prove it? AS uses average goals per season in its formula. That is proof is it not?

I have stated this over and over ad nauseum and have never received a direct response. AS uses average goals per season, yes or no?

 

[Iain Fyffe]

I have stated this over and over ad nauseum and have never received a direct response. AS uses average goals per season, yes or no?

Yes. (Though I believe this is the first time you've explained what you mean by "averaging seasons"). Now, how does this result in normalization?

Please remember that the study does not warn against using averages in any way, it warns against normalization and assuming a normal distribution. Human performance will always have an average (mean), typically with the mode being below the median which is below the mean (which is in fact the pattern that AS results follow, since they take the same shape as the actual results).

If we're not going to use the actual distribution of performance, what distribution would you suggest?

 

[pdd]

Wait, wait, wait.
Can you show me where Adjusted Stats results have been verified?
Show me the same level of proof and verification for Adjusted Stats that you're asking me to provide and verify here.
Are you actually asking me to provide 100% proof when you can't prove Adjusted Stats are 100%?

You misunderstand the purpose of AS. While it does have its flaws, it is intended to show how the player's performance would have compared to the league in another year. The idea behind removing the player's own numbers is "His stats compare to this year as X; when adjusted to a common average they equal Y. Thus he can be compared against another season's performance, as the "everyone else" group is simply replaced with a different group of player/performances.

As for the Jagr/Gretzky comparison, the use of AS is a valid way of looking at things because the GPG is not the same in any given year and there was in fact wide variation across the 90s.