Monitor - June 2016

have critiqued the methods and findings of the replication proponents.

Psychologists are not the only ones grappling with this issue. Many fields of scientific research are facing similar concerns. Take biomedical science. A decade ago, Stanford University epidemiologist and biostatistician John Ioannidis, PhD, published a paper about biomedical research titled “Why most published research results are wrong” (PLOS Medicine, 2005) that has become the most cited in the journal’s history. Ioannidis is watching the discussion in psychology with great interest.

“Psychology is clearly going through a lot of soul searching and has some of the best ideas and most contested ideas right now,” he says. “There are many research practices that people are re-examining, trying to see what they can do better.”

From the lab to the front page

In psychology, the replication discussion moved from labs and academic conferences to newspaper pages last summer, when the Open Science Collaboration (OSC) — a group of more than 250 researchers led by University of Virginia psychology professor Brian Nosek, PhD — published the results of their attempt to reproduce 100 studies from three top journals in social and cognitive psychology. The researchers found that although 97 percent of the original studies produced significant results, only 36 percent of the replications did (Science, 2015).

In the paper, the OSC researchers did not use the word “crisis” and were careful to say that their findings were meant solely to provide an estimate of reproducibility in psychology, not to definitively prove any of the original research wrong.

“Original studies are not definitive,” Nosek says. “And
replications are not definitive — no study is definitive. In
science, we seek definitiveness and almost never get it. We have
to be self-critical and modest with our claims.”
Nosek does, however, believe that the OSC results are
meaningful, and reflect significant problems in the way that
psychological research is conducted and published. Nosek,
who spent much of his career studying unconscious biases, has
focused increasingly on these methodological issues instead.
Three years ago, he founded the Center for Open Science, a
nonprofit devoted to increasing the “openness, integrity and
He and his colleagues see several problems plaguing
psychological research. First, there is the “file-drawer problem.”
Journals in psychology (and other fields) want to publish
positive results. So when researchers find a positive result in the
lab, they write it up for publication. When they find a negative
result, they shove it in their file drawer — leaving the research
literature rife with unchallenged false positives.

And where do those false positives come from? Some will
occur at random, of course. But common research practices
make them more likely, especially what’s sometimes called
“p-hacking” — essentially, when researchers fish around in their
data and perform multiple analyses until they find effects that
reach the standard criterion of significance: p<.05. P-hacking
sounds nefarious — like cheating — but in practice it is more
complicated and thus more difficult to address than that.
Science involves many small decisions at all stages of research
— what data to collect, what variables to analyze, how and
when — and researchers’ small, unconscious biases can affect
those decisions at every stage.

So Nosek and his colleagues at the Center for Open Science are advocating for a series of reforms and creating infrastructure to bring negative results out of the file drawer, and to make inadvertent p-hacking less likely. One way to do so, they believe, is for researchers to make their data and research methods publicly available. That would make replication easier and more likely. Another is to encourage researchers to pre-register more studies, essentially “tying their hands” by having them describe their research methods and analysis plan in advance so that posthoc data analysis is clearly labeled as exploratory. They have written guidelines for journals interested in adopting their ideas, have developed cloud software to make data sharing easier, and are even offering $1,000 prizes to researchers who pre-register their studies and then publish the results.

Inzlicht, for one, finds the work of Nosek and the other replication efforts convincing. He was particularly taken aback by the results of the ego depletion study, which was part of a new series of registered replication reports supported by the Center for Open Science and published in the journal Perspectives on Psychological Science.

“You could call me a replication convert,” Inzlicht says. “I used to think that there were errors [in the way psychology research was conducted], but that the errors were minor and it was fine. But as I started surveying the field, I started thinking we’ve been making some major mistakes.” At this point, he says, “If your faith is not shaken, you’re not paying attention or you’re in a state of denial.”

A replication backlash

But not all researchers are convinced of the OSC paper’s validity, or that the state of the research literature is so dire. In March, Harvard University psychologist Daniel Gilbert, PhD, University of Virginia psychologist Timothy Wilson, PhD, and colleagues published a critique of the OSC study, also in Science. In it, they argue that the replication effort had several critical flaws. First, they say, the replicated studies were not chosen at random from the three journals. Instead, some studies were deemed too difficult to replicate and were excluded.