that vary according to time or place, such as race or political
attitudes. Context-insensitive studies examine things that, at
least in theory, should stay constant in different settings, such as
the Stroop effect.
The fact that the context-sensitive studies were less likely to
replicate suggests that there are “hidden moderators” at work,
says Van Bavel — important contextual differences between the
original studies and the replications that, although not obvious
to the replicators, contributed to the replication’s failure.
Van Bavel does believe that replication studies are valuable,
and he sees two takeaways from his study. The first is that
researchers and those who want to replicate their work should
work more closely together to clarify the theory behind the
research and decide exactly what stimuli and experimental
designs are likely to elicit the effect they’re looking for.
The other takeaway, he says, is that researchers may need to
become more clear in their original papers about the limits of
their research. “Most of the things we study aren’t universal,
they are going to have boundary conditions,” he says. “So part
of these is being modest, and being more explicit about the
A blow or a boon to psychology?
Gilbert’s concern about journalistic misinterpretation hints at
another worry for many psychologists:
that the replication research is harming
psychology’s public image.
Nosek believes that the concern is
not warranted. “By and large, the actual
science journalists have done a good job
unpacking this, though nuance gets lost
in the headlines,” he says.
Crucially, he also believes that
questioning some of the basic tenets
of psychology will not cause funding
agencies to abandon psychological
research. “My sense is that funding
agencies are actually looking to
psychology as a leader in addressing these
replication issues,” he says. “These are not
issues that are limited to psychology.”
In fact, among the next projects
that the Center for Open Science will
undertake are replication studies in
cancer biology and tropical ecology.
Within psychology, the replication
movement, which so far has mostly
focused on social and cognitive
psychology, is expanding to other
areas of the discipline. For example,
Michael Frank, PhD, a developmental
psychologist at Stanford University, is
organizing a multi-lab pre-registered
replication study of a classic finding in developmental
psychology: that infants prefer infant-directed speech, or
mother-ese, to adult speech.
“The study will serve many purposes,” Frank says. “Not just
to test the specific effect, but also to allow us to get a handle
on some of these methodological issues that make replication
So what about all of the psychologists who, like Inzlicht, are
worrying that the seemingly reliable findings on which they
have based their research and their careers might turn out to be
“puffs of smoke?”
Inzlicht himself sees the silver lining. “We’re better off now
than we were yesterday, or 10 or 20 years ago, even if it means
we overturn findings,” he says. “If something is not true, it’s
obviously better to know. I don’t want to waste my time and I
don’t want to waste my students’ time chasing smoke.”
From outside the field, John Ioannidis agrees. “I don’t expect
100 percent of psychology to be wrong by any means, but there
may be a pretty high rate of irreproducibility that will vary
across fields,” he says. “There will be many classics that hold
up, and some will be refuted. Some recent literature with small
effects may not hold up. I think this is all very good. It tells us
what we know and don’t know. Psychology will not disappear, it
will become more reliable and more exciting to work on.” n
www.aatbs.com • 800-472-1931 • 5126 Ralston St., Ventura, CA 93003
AATBS provides yearly membership to mental health
professionals with multiple tools and resources to grow
and advance your career. Welcome to The Association.
Learn more at www.aatbs.com/monitor
Therapy Notes, Client
Billing and much more.
Member Benefit #2:
Members receive a 30-day Free trial
plus 1 additional month FREE
of SimplePractice® software.