Against optimism about social science

Andrew Gelman:

I agree with Marcus and Rojas that attention to problems of replication is a good thing. It's bad that people are running incompetent analysis or faking data all over the place, but it's good that they're getting caught. And, to the extent that scientific practices are improving to help detect error and fraud, and to reduce the incentives for publishing erroneous and fradulent results in the first place, that's good too.

But I worry about a sense of complacency. I think we should be careful not to overstate the importance of our first steps. We may be going in the right direction but we have a lot further to go. Here are some examples:

1. Marcus writes of the new culture of publishing replications. I assume he'd support the ready publications of corrections, too. But we're not there yet, as this story indicates:

Recently I sent a letter to the editor to a major social science journal pointing out a problem in an article they'd published, they refused to publish my letter, not because of any argument that I was incorrect, but because they judged my letter to not be in the top 10% of submissions to the journal. I'm sure my letter was indeed not in the top 10% of submissions, but the journal's attitude presents a serious problem, if the bar to publication of a correction is so high. That's a disincentive for the journal to publish corrections, a disincentive for outsiders such as myself to write corrections, and a disincentive for researchers to be careful in the first place. Just to be clear: I'm not complaining how I was treated here; rather, I'm griping about the system in which a known error can stand uncorrected in a top journal, just because nobody managed to send in a correction that's in the top 10% of journal submissions.

Posted by Jim Zellmer at May 13, 2013 12:18 AM
Subscribe to this site via RSS/Atom: Newsletter signup | Send us your ideas