Another failure to replicate Daryl Bem's research

[sbu]

Correcting the Past: Failures to Replicate Psi by Jeff Galak, Robyn LeBoeuf, Leif Nelson, Joseph Simmons :: SSRN

Across seven experiments (N = 3,289) we replicate the procedure of Experiments 8 and 9 from Bem (2011), which had originally demonstrated retroactive facilitation of recall. We failed to replicate that finding. We further conduct a meta-analysis of all replication attempts of these experiments and find that the average effect size (d = .04) is no different from zero. We discuss some reasons for differences between the results in this paper and those presented in Bem (2011).

[eveshi]

Steven Novella commented on this on his blog:
NeuroLogica Blog The Power of Replication – Bems Psi Research

[AusMan]

Originally Posted by eveshi

Steven Novella commented on this on his blog:

I quote from the comments section:

"They latch onto a single study, an anecdote, a graph, or a supposed expert’s name and wave it around as if it was an everlasting totem of power to ward off skeptics."

I'm sorry but this is just hilarious in a genuinely amusing way! It raises all kinds of "Far Side" styled images as I think about it.

[Paul C. Anagnostopoulos]

Thanks for the link, sbu. I'll print it, read it, and add it to my "Feeling the Future" pile.

~~ Paul

[Johann]

This is not a new study. The page Sbu links to was posted in January, and only updated in June. I can't be sure what they updated, since Ritchie, Wiseman, and French mentioned this in response to Bem in March: PLoS ONE : accelerating the publication of peer-reviewed science

- Johann

[Johann]

I thought it might be instructive also to post the replies from Davidsmith and Michael Duggan, that were on Novella's blog, here:

# davidsmithon 29 Aug 2012 at 5:41 pm

Interesting paper. As a proponent who is keen on nullifying null results, it concerns me that many participants in Galak et al.’s experiments (particularly the online ones) may have known about the ‘surprise’ memory test (of course, it’s not a surprise if you know that it’s coming). I am aware of quite a number of people who did indeed know about the memory test and who completed an online session. I guess the experiment may have been circulated around many more friends of these people who were also made aware of the memory test or who just knew about this experiment from the wide media coverage. Why is this important? As Michael Franklin pointed out in response to Ritchie et al.’s failed replication, the original study was supposed to involve incidental encoding during the visualisation phase. However, if participants know that a memory test is coming up then the task is converted from one of incidental encoding to explicit encoding. Perhaps then, conscious/intentional encoding strategies serve to extinguish or reduce this particular effect? Franklin gives verbal overshadowing as an example of this happening in other psychological studies. I only skimmed the Galak paper, but I don’t think this point is mentioned. It is probably the most reasonable explanation for the failure to replicate other than the null is true.

# Michael Dugganon 29 Aug 2012 at 9:59 pm

I have to agree with David Smith. Additionally Bem warned about the likelihood of negative results emerging from online tests. How do you maintain motivation, interest, and belief in success with remote tests? When you remove the online data from the meta-analysis reported by Galak, et al, there are 6 positive significant studies from 15, or a 40% success rate. This rates comparably to other programs in Parapsychology. In fact I have never heard of a successful online study in psi research! Readers are encouraged to investigate the data from Psi research. Some of it is quite impressive IMHO. See here for example: Frontiers | Extraordinary Claims Require Extraordinary Evidence: The Case of Non-Local Perception, a Classical and Bayesian Review of Evidences | Frontiers in Quantitative Psychology and Measurement.

- Johann

[Paul C. Anagnostopoulos]

Originally Posted by mszlazak

It's just the same never-ending-story with [SNIP]"skeptics." Biased reporting, bad experimental designs that masquerade as replications, moving the goal posts and so on.

Geez, relax. Are Galak et al skeptics? Even if you remove the online replications, they still have failure to replicate. And they spend a lot of words in the paper discussing possible problems with online experiments.

The parapsychology community has to be careful not to ignore every study by researchers they think are "skeptics" or "nonproponents." This is especially true if they are also going to complain that no one seems to want to perform psi experiments. Wny would I bother to get involved if I thought the community had a prepared pile of excuses to reject negative findings?

As far as the objection that subjects might know a memory test is coming up after the visualization phase, who are we kidding? If I'm participating in a psychology experiment and I'm given a list of words, obviously there's a good chance I'm going to be tested on them. And the longer we go running Bem replications, the more likely subjects are to know this, and even to know that there will be a practice after the test. I suppose we can use this as the excuse for any decline effect.

And regarding motivation and interest for online tests, what does Duggin think researchers ought to do to maintain these things in lab tests? I hope he's not thinking of the researchers interacting with the subjects while they are taking the test. And let's be sure to distinguish motivation to participate from motivation to do well.

~~ Paul

[Maaneli]

Originally Posted by Johann

I thought it might be instructive also to post the replies from Davidsmith and Michael Duggan, that were on Novella's blog, here:



- Johann

I think their criticisms are dead-on. In particular, David is absolutely right that an unmonitored online experiment is nowhere comparable in terms of motivation and focus to a monitored lab experiment. And Duggan's finding that 6/15 (40%) studies are independently significant is very important because, in spite of the majority being failed replications, this proportion of successful replications is way more than can be accounted for by the null hypothesis (we would expect only 5% by chance, and the exact binomial p-value for this outcome is 5.28*10^-5 or odds against chance of 18,939 to 1).

[Paul C. Anagnostopoulos]

Originally Posted by Maaneli

n particular, David is absolutely right that an unmonitored online experiment is nowhere comparable in terms of motivation and focus to a monitored lab experiment.

But might be much better in terms of relaxation. Which is more important?

~~ Paul

[Johann]

Originally Posted by Paul C. Anagnostopoulos

But might be much better in terms of relaxation. Which is more important?

~~ Paul

Clearly focus and motivation.

- Johann