The research replication crisis in psychology

Blog #69 – The research replication crisis in psychology

By Dr. Mark Barry

Research psychology in recent times has begun to grapple with a so-called ‘replication crisis’. When we talk about replication in this context we are referring to the process whereby a study is conducted again in a different setting – ideally multiple times by different researchers – with a view towards establishing whether or not similar findings are reported with different populations in different places and situations and across time.

Why is this important? Well, if findings hold up across time, in different places, and among different populations, then that suggests that those findings are reliable. However, if the initial findings cannot be replicated, then that indciates there may have been something unique about the circumstances surrounding the original study, or perhaps errors we made somewhere along the line. Either way, when results don’t replicate, then they are not as relevant to a wider population, and therefore, are of less value when it comes to our understanding of the human mind.

There are many facets to this problem and it will not be possible to go into the required depth here; however, we can touch upon some of the relevant issues, such as the make-up of research populations and the evolution of the academic publishing industry.

It’s not quite a ‘deep, dark secret’, but certainly one of the limitations often seen in social psychology research relates to participants. Basically, university undergraduates tend to be over-represented. This is because most academic researchers work in third-level settings and students represent a ready-made research population. Of course, there is nothing wrong with students being research participants, but when interesting findings are published based on such a population, it represents a fairly narrow snapshot of humanity, and without replication, we cannot know if such findings are generalisable to a wider population.

To make this point even more clearly, consider Sigmund Freud. Freud is arguably the most well known name ever to emerge in psychology and related fields, but within the discipline he is nowadays appreciated more for historical value than the validity or otherwise of his theories. One of the main reasons for this shift is the realisation that he worked with a very narrow sample of society. Freud based his theories on his encounters with his patients, but they tended to fit a distinct profile: upper-class, middle-aged Viennese housewives. That’s fine as far as it goes, but it doesn’t go very far. In other words, you cannot reasonably claim to have achieved insight into the broad human condition based on a narrow sample of one kind of person. In Freud’s case, at best, his theories and insights may have been applicable to the typical upper-class middle-aged Viennese housewife of his time, but of perhaps less relevance to the rest of us.

The same applies to findings based on studies conducted with university students. Without further work with more varied populations, it is questionable whether the original findings can be claimed as being relevant to other groups. And with replication studies being neglected, these gaps in our knowledge are being left unfilled in places.

But that merely points to one of the downsides of a research world in which replication is neglected; it doesn’t necessarily tell us why that work is not being done, at least not to the extent that it might or should be. This brings us to what is perhaps the heart of the matter: Replication isn’t sexy. There, I said it. Nowadays, not many academic careers are going to be advanced by replicating pre-existing research, so not as many researchers are taking the time to do that work as may have been the case upon a time.

Why this change? There is no single or simple answer to that, but it cannot be doubted that the evolution of the acaademic journal publishing industry is one of the puzzle pieces. It has long since been observed that many journals demonstrate a bias towards positive results, i.e., if researchers report that their hypothesis was supported by their findings then that paper is more likely to be accepted for publication. That is not to say that reporting negative results might be valuable in its own right, merely that such papers appear to be less attractive to journals. Why? Again, there is no single answer, but positive results tend to be regarded as more interesting and therefore more likely to get attention from the media, both for the journal and researchers in question.

In the same way, journals have also been accused of having a bias against publishing replication studies. Again, this is not about some grand conspiracy, but rather the more prosaic reality that the original findings are deemed more interesting than subsequent attempts to demonstrate that those findings transfer across time and context. And again, researchers see what types of study are favoured by journals and cannot be blamed for opting against research paths that appear less likely to yield publication credits.

This most certainly is not just a matter of concern for well-being researchers, but is instead an issue that social psychology as a whole needs to grapple with, but neither is positive psychology immune to these concerns.

Writing in Greater Good Magazine – published by UC Berkeley – in 2015, prominent positive psychology researcher Robert Biswas-Diener acknowledged the potential implications for those working in this area. “Heavy commercialization of this relatively new field — and the fact that it is an applied science — leads to a consumer economy of blogs, books, and seminars that favors single study results.”

Single study results may be interesting and may lend themselves to media interviews, book deals, and slots on the commerical lecturing circuit, but without replication those single studies are of questionable true value.

Biswas-Diener stresses that positive psychology needs replication, but he also points out that it is a relatively young field, and that replication – particularly among researchers conscious of its importance – can come naturally over time.

Of course, the neglect of replication is merely one facet of the problem. Without replication we cannot know if single study findings are as vaulable as they might seen; but what if well-known studies are found to not be replicable? That will be our focus next time.

Mark Barry was awarded a PhD by University College Cork in 2015 for his research into adolescent well-being. He has lectured psychology at UCC since 2013 and is also a freelance writer.