have critiqued the methods and findings of the replication
proponents.
Psychologists are not the only ones grappling with this
issue. Many fields of scientific research are facing similar
concerns. Take biomedical science. A decade ago, Stanford
University epidemiologist and biostatistician John Ioannidis,
PhD, published a paper about biomedical research titled “Why
most published research results are wrong” (PLOS Medicine,
2005) that has become the most cited in the journal’s history.
Ioannidis is watching the discussion in psychology with great
interest.
“Psychology is clearly going through a lot of soul searching
and has some of the best ideas and most contested ideas right
now,” he says. “There are many research practices that people
are re-examining, trying to see what they can do better.”
From the lab to the front page
In psychology, the replication discussion moved from labs and
academic conferences to newspaper pages last summer, when
the Open Science Collaboration (OSC) — a group of more
than 250 researchers led by University of Virginia psychology
professor Brian Nosek, PhD — published the results of their
attempt to reproduce 100 studies from three top journals in
social and cognitive psychology. The researchers found that
although 97 percent of the original studies produced significant
results, only 36 percent of the replications did (Science, 2015).
In the paper, the OSC researchers did not use the word
“crisis” and were careful to say that their findings were meant
solely to provide an estimate of reproducibility in psychology,
not to definitively prove any of the original research wrong.
“Original studies are not definitive,” Nosek says. “And
replications are not definitive — no study is definitive. In
science, we seek definitiveness and almost never get it. We have
to be self-critical and modest with our claims.”
Nosek does, however, believe that the OSC results are
meaningful, and reflect significant problems in the way that
psychological research is conducted and published. Nosek,
who spent much of his career studying unconscious biases, has
focused increasingly on these methodological issues instead.
Three years ago, he founded the Center for Open Science, a
nonprofit devoted to increasing the “openness, integrity and
He and his colleagues see several problems plaguing
psychological research. First, there is the “file-drawer problem.”
Journals in psychology (and other fields) want to publish
positive results. So when researchers find a positive result in the
lab, they write it up for publication. When they find a negative
result, they shove it in their file drawer — leaving the research
literature rife with unchallenged false positives.
And where do those false positives come from? Some will
occur at random, of course. But common research practices
make them more likely, especially what’s sometimes called
“p-hacking” — essentially, when researchers fish around in their
data and perform multiple analyses until they find effects that
reach the standard criterion of significance: p<.05. P-hacking
sounds nefarious — like cheating — but in practice it is more
complicated and thus more difficult to address than that.
Science involves many small decisions at all stages of research
— what data to collect, what variables to analyze, how and
when — and researchers’ small, unconscious biases can affect
those decisions at every stage.
So Nosek and his colleagues at the Center for Open Science
are advocating for a series of reforms and creating infrastructure
to bring negative results out of the file drawer, and to make
inadvertent p-hacking less likely. One way to do so, they believe,
is for researchers to make their data and research methods
publicly available. That would make replication easier and more
likely. Another is to encourage researchers to pre-register more
studies, essentially “tying their hands” by having them describe
their research methods and analysis plan in advance so that posthoc data analysis is clearly labeled as exploratory. They have
written guidelines for journals interested in adopting their ideas,
have developed cloud software to make data sharing easier, and
are even offering $1,000 prizes to researchers who pre-register
their studies and then publish the results.
Inzlicht, for one, finds the work of Nosek and the other
replication efforts convincing. He was particularly taken aback
by the results of the ego depletion study, which was part of
a new series of registered replication reports supported by
the Center for Open Science and published in the journal
Perspectives on Psychological Science.
“You could call me a replication convert,” Inzlicht says. “I
used to think that there were errors [in the way psychology
research was conducted], but that the errors were minor and it
was fine. But as I started surveying the field, I started thinking
we’ve been making some major mistakes.” At this point, he
says, “If your faith is not shaken, you’re not paying attention or
you’re in a state of denial.”
A replication backlash
But not all researchers are convinced of the OSC paper’s
validity, or that the state of the research literature is so dire. In
March, Harvard University psychologist Daniel Gilbert, PhD,
University of Virginia psychologist Timothy Wilson, PhD, and
colleagues published a critique of the OSC study, also in Science.
In it, they argue that the replication effort had several critical
flaws. First, they say, the replicated studies were not chosen at
random from the three journals. Instead, some studies were
deemed too difficult to replicate and were excluded.
“You have to find a representative sample to replicate if
m
stay
/iSt
oc
k