Why we should not fear treating preregistration as a gold standard

A commentary

I wrote the following commentary in response to the post “Why treating preregistration as a gold standard might incentivize poor behavior” by Richard Morey.

Dear Richard,

With all due respect, I don’t think we need to “have fears” about preregistrations. As you rightly pointed out, preregistration does not prevent checking for robustness as additional exploratory measures (or even as preregistered outcome neutral checks), and as long as preregistration is coupled with the sharing of raw data, we have the best of both worlds, as you point out in the conclusion of your post.

If there is a case for fearing preregistrations, perhaps the issue is one of disclosure and trust. Two decades ago, we were trained to conduct research in cosy ways, in the safety of our private offices and labs, refining our thoughts, research questions, hypotheses, and data files as we went along, deciding a posteriori when we were ready to make our work public (often the polished, cleaned, supposedly error-free version of our work). I am not talking about faking data here. I am talking about praxis, that is practice as distinct from theory, research as it happens in real life. There was no training in data management back then. How many of us can claim to have never created multiple versions of a data file under different names, or different folders? How many can say they have never consolidated their data analysis plan(s) while they were analysing their data? If you did, that’s OK! This is how we learnt to become a researcher, through trial and error.

Now, we are told, and are telling our students, we must plan everything ahead. The problem is, those among us who are more experienced with research know: plans rarely if ever survive contact with reality.

So perhaps the fear that preregistration instills is the fear of “being wrong” a priori or “being found out”. Open disclosures are frightening because they make us vulnerable. So the biggest challenge is for us, as a community, to change our outlook on other people’s work. To recognise that those who open up their work to scrutiny via preregistration are making themselves more vulnerable to criticism for their design and analysis choices. If we happen to know or believe there is a better way, we owe it to Science to share it, but we also owe it to our colleagues to do so with respect and constructive feedback. We need to recognise that, contrary to what the old ways of conducting and reporting research may have led us to believe, in fact, no-one is omniscient or should be expected to be. We should remind ourselves that preregistration is not about “getting everything right” a priori, that doing research and making hypotheses is not about “being right” or “being wrong” as a person, or conducting the “right kind” or the “wrong kind” of data analysis. It is about trying to find out, to the best of our abilities, both individually and collectively, what is true and what is not. That should include pre-registration but also registered replications, and positive feedback suggesting or implementing alternative analyses were needed, but always with the utmost consideration for others and their work.

Sincerely,

Gaëlle.

Avatar
Gaëlle Vallée-Tourangeau
Professor of Behavioural Science
comments powered by Disqus