If we think randomness and luck are such huge factors then there's not much point in statistical analysis. Or any analysis other than "was that luck or skill?"
I don't agree with this. You want your team and players to push any edges they have, no matter how small. After all, that's how casinos make money. They know that in the short-run they can take a hit due to incredible amounts of random chance, but as long as they keep pushing their edge then there is no way they can lose in the long-run. Ultimately, you want the biggest edge, even if there is still a large amount of randomness obfuscating what that edge might be.
Regarding the bold, the questions are a bit circular. If you can't trust the numbers then how can you verify an eye test with them?
You only need to be able to trust the data collected in order to make a reasonable analysis. xGF is rendered useless if, for example, the shot location data is off by a large and biased degree. That actually happened last year for a short period of time, before the spreadsheet nerds noticed this and told the NHL to get their act together.
Other than that, I'm not sure what you mean by trusting the numbers. The data are what they are. xGF is just a calculation based on the data given to the public. The correlation is just another calculation based on this data. The predictive power based on X games is just another calculation. If the data provided is accurate, then the calculations are accurate.
The fundamental question about observational analysis vs statistical analysis is not going to be resolved by stats, unless those stats are completely objective and the observations are similarly objective and measured in a way that can be tested against those stats. If randomness is always there to bail out any given analysis it's sort of pointless.
Randomness doesn't bail out any given analysis if it truly is a factor. Again, I point to the casino example. If someone hits their number of roulette and the casino loses thousands of dollars, what's the explanation? Simple: random chance. It's a perfectly accurate description of what happened: the casino got unlucky, because on average with every spin of the roulette wheel they will make money. The player didn't "game the system" and the employee didn't mess up, it was just luck.
But I would agree with the first part of this paragraph. Essentially, you have to have a degree of trust in the data collected. There are always going to be inaccuracies in some of the shot data, sometimes wildly so, but by and large the data is reasonably accurate and any biases are adjusted for.
Of course, the same must be said of any qualitative analysis: are you sure you are accounting for all of your biases when you see what you see? How do you know what you just saw is helping the team, rather than just making you feel good? When someone is "mucking it up", how much do you think it adds to the team winning, and how do we know if that's accurate unless you test it?
I would say that when assessing how a player is doing, or their potential, a good deal of variation is going to exist because not everyone is equally skilled or perceptive when it comes to such evaluations. My "eye test" is not going to be the same as Lavi's.
But much of sport is art and flow as much as it is metrics and points. Can you quantify art, or flow states? Do you know art when you see it? Can you spot someone in the Zone?
A lot of this perception-based analysis hinges on the same sorts of unconscious information processing that's involved in creating or viewing art/music, or being in the zone. The creativity and proficiency needed for high level performance, and the decision making that happens in split seconds, will elude any chart or table because it comes from the same place. If it didn't then every junior or AHL player who puts up big numbers would just tear up the NHL, and we know that doesn't happen. There's something ELSE missing.
This drives stat people nuts because it can't be pinned down or bottled or fit into a spreadsheet, but it's at least as important as any statistic you can name.
This is what frustrates people about Kuzy. He's clearly got the skill and sense to produce and do the right things on the ice but for some reason he doesn't deliver with the consistency of the top stars.
I agree with most of this, but ultimately games are won and lost by who scores more goals than the other team. There aren't any style points in the standings. What you're describing is the "how", I'm describing the "what."
Kuznetsov has certainly changed as a player, there's no argument from me there. But he's controlling the things that he can control to help the team win.
Also, this kind of discussion is why we started that fancy stats thread that's collecting dust in the HF attic...
No thanks, that thread is for nerds.
Who'd a thunk it?
