I'm not challenging the notion that it's happening. I'm challenging the position that the fact that it is happening, to one degree or another, means "the truth is obscured beyond hope of ferreting it out."
We have lots of data that largely agrees. Some of it depends on self-reporting by parties with a vested financial interest, and some doesn't. Your experiences might cast some measure of doubt on the former, but it doesn't on the latter. The fact that they largely agree in spite of your experiences might not mean anything conclusively, but it does potentially add some context to your concerns.
More importantly, what we're ultimately talking about is valuable data that can be used to help make important decisions. Implying that there are no numbers out there that can be trusted because of your anecdotal/personal experiences, no matter how pervasive or valid, only works to undermine the very scientific process that you value. How can one expect policy makers to make data driven decisions if they are hearing from people of authority in the field that the truth is hopelessly obscured, and that there is no data that can be trusted? Even spreading that message among the general populace only works to undermine any information being put forth by those doing their best to gather good data and leverage it to generate better outcomes.
You know you have my respect, and my friendship, but I think that's a position that needs to be challenged.
You are definitely correct that meaningful & descriptive data can be gleaned, even in this current climate. If one ones to be technical - the truth can always be uncovered - it's just a question of how much resources need to be put into uncovering it - and resources include people, time, money, labor (to name a few). Plus, there is the added layer of how far into peoples personal information one could (or should) go, and whose authority that should be.
I am a biomedical engineer by education. The school I graduated from was ranked higher than MIT (at the time of graduation).
If I could be said to have an expertise, it would be in the realm of design of tests/experiments, number crunching, data analysis, and drawing conclusions (reports, technical memorandums, presentations).
Drawing conclusions, and presenting the data concisely - was always of more interest to me than even the research part of things.
And..I've also been in situations where politics (company politics) and business sense have put a finger on the scale.
Politics aside, what I've learned from the business side of things, is that situations arise where a conclusion is needed subito. From a researcher's perspective, there are (almost) never enough hours in the day, never enough technology, never enough manpower on a project to do a thorough enough job to get "the result". Researchers love to research, but sometimes, for the greater good, what's needed is a balanced perspective of other business units - empathy. When one understands the value of his/her research in the context of a bigger picture, it's a lot easier to have the humility to cut bait, if the time calls for it.
Even in situations when there are "lots of data that largely agrees".. what does that mean? There can be chimeras in numbers; it can also be very easy to get "in the weeds" with numbers, when, all along, a better answer lies in something more practical.
You are describing a situation where a policy maker must be on top of the numbers, all of the time, in order to make sound decisions - in real time. To me, that reminds me of the frustrations of attempting to determine a starting lineup for fantasy football (which is why I do not play fantasy sports, for that very reason).
People also need to live life, and that needs to be respected. For example, my family will pass on "virtual thanksgiving", regardless of what data or policy makers say - because you only receive so much time to spend with family, and that will take precedence - thank you very much. We are all well aware of the benefits of washing hands, wearing masks, and distancing, so we really don't need anyone to tell us how those should be configured - but there is no substitute for seeing someone in person.
For some researchers, particularly the ones that are in position to affect policy, I sometimes wonder if things are playing out like a real life episode of the twilight zone - always chasing and crunching numbers, never enough hours of the day... and without the good sense of knowing when to stop - the final reports might as well say "all work and no play make Jack a dull boy".