I’m grateful to Jessica Hullman for writing about this very wonderful and relatable essay by Rachael Meager on the gap between how we think about data and how we think we think about data. It’s a massively quotable piece, and Jessica pulls some great ones in her commentary; the one that rings through the years for me, though, is Meager’s quote or paraphrase of an anonymous grad student on learning that the chi-square test had shortcomings typically ignored by practitioners:
BUT MEDICAL PEOPLE USE THE CHI SQUARED TEST he moaned. It’s true. If we were right then, in our interpretation, it does seem bad that only a handful of people even knows that there should be either some check, or some adjustment. Nobody seems to care that there are probably many cases where this test is not functioning at all.
… and look, I have been around too long for this defect in chi-square to come as a surprise, but it is nonetheless news. I didn’t know about it, have never once adjusted for it, and wouldn’t (at present) know what to do if I wanted to adjust for it, or in what circumstances I should. I don’t even fully understand the problem. And it’s a common test! All over psychology for sure, and definitely present in reports I’ve written for work.
The thinking behind BUT MEDICAL PEOPLE USE THE CHI SQUARED TEST is what led me to my worst professional misstep ever: Joining an applied research group with an expectation of increased methodological rigor. It’s a forgivable thought, or at least I tell myself it was at the time: If your colleagues are using quantitative research to make decisions that affect real people, surely they must be more careful about their research than basic scientists publishing for purely theoretical interest.
IYKYK. For the rest of you, I regret to inform you this is flat backwards. For the most part, the people who are making big decisions based on data believe one of two things: Either they know better than the data, or they don’t need to worry about the subtleties of data analysis because the signals in the data are so clear. Given the noisiness of most data, option 2 usually amounts to believing you know better than the data.
And when I say “the subtleties of data analysis,” please bear in mind I’m the guy who didn’t know about the problems with chi-square. I’ve been a data scientist for some time, and I’ve developed a nose for data smells, but that’s very much not the same thing as a solid formal foundation in statistics. Sometimes it’s better… but not always, not even close. The things I think of as “subtleties” are table stakes for people like Jessica Hullman and Rachael Meager. Things like, “where you have more people, you have more of everything associated with people.”
… I have more thoughts in this general area, but I think I’ll close this post off for now. Part II in a bit, maybe.
Currently listening: A Court of Fey & Flowers S1E7, “The Masquerade Ball”