Replication and Meta-Analysis

[anonymous]

I'm re-reading "The Conscious Universe" by Dean Radin and the chapters on replication and meta-analysis struck me as relevant to the discussions in this forum. The key points are:

You can use an experiment with a small sample size to invalidate a phenomena. The book gives an hypothetical example. If you do an experiment where you get a result of 70 hits in 100 trials (70%) where 50 hits (50%) would be expected by chance, the odds against chance of that result would be 10,000 to one. If someone tried to replicate this experiment and did only ten trials and got seven hits (70%), would that confirm the original experiment? No. With only ten tirals, that would yield odds against chance of only five to one. Twenty to one odds against chance are usually required to claim a statistically significant result so this experiment could be called a failure and it could be said that when the original experiment was replicated, they failed to obtian a statistically significant result.

This is why meta-analysis is so valuable. The book illustrated this by describing the meta-analysis that was used to demonstrate that asprin reduced heart attacks. There were many studies that found that asprin reduced heart attacks and a few that didn't. However most of these experiments did not exclude chance with sufficient confidence. When all the trials in all studies were combined in a meta-analysis, the overall effect was positive and not explanable by chance with very high confidence.

The book also describes a fascinating process by which the latest values for the physical properties of fundamental particles are updated by the Particle Data Group (PDG) as new data are obtained. According to the PDG the process includes excluding data when "The results involve some assumptions we do not wish to incorporate." or "The measurement is clearly inconsistent with other results which appear to be highly reliable." Changes for values in the PDG reports can be due to "discarding of older data" when "it is felt that the newer data had fewer systematic errors."

I don't disagree with that process. I'll just let the reader form their own opinion on if and how this might be relevant to the controversy on parapsychology.

Interestingly, the book describes an analysis that compared replicability in hard science to replicability in social science and found that, "About 45% of the reviews in both domains exhibited statistically significant disagreements when no studies were omitted from the results." and that "Comparison [of research results in physics and the social sciences] suggests that the results of physical experiments may not be strikingly more consistent than those of social or behavior experiments." Hard science isn't hard.

[eyemsougly]

Hi, I'm new.

The problem with meta analysis of psi experiments is we don't really know the quality of those experiments.

The post Hyman Honorton communique ganzfeld seemed to have solved that (significant 31% hit rate from 86 to 99) but upon the introduction of stricter standards in 2000 the hit rates declined to a non significant 27%. Radin is a sharp guy but I don't think the evidence is as robust as he seems too.


what we need is a repeatable experiment, not excuses why we can't find one. I think the best bets right now are:
1. dogs that know
2. medium trials
3. staring experiments
4. telephone telepathy

[anonymous]

Originally Posted by eyemsougly

The problem with meta analysis of psi experiments is we don't really know the quality of those experiments.

The way it says to address quality of studies in a meta-analysis is to identify high quality studies and low quality studies and then see if effect is smaller in the high quality studies. This will work if some studies considered the particular issue you think is a flaw.

The post Hyman Honorton communique ganzfeld seemed to have solved that (significant 31% hit rate from 86 to 99) but upon the introduction of stricter standards in 2000 the hit rates declined to a non significant 27%.

How were the controls tightened?

What were the flaws in the previous studies?

Do you have a link or other reference with more infomation about this?

Thanks,

[eyemsougly]

it sounds like the big flaw was using VHS tapes, which was pretty much all that was available back then. As the tapes get worn down subjects might beinadvertantly cued that a certain segment was the target. Apparently there being more hits in later parts of trials gave weight to this or something.

Hopefully this link works: Ganzfeld Experiment


Anyway, since 2000 everything has been computerized. Unfortunately the article doesn't specify how true to the ganzfeld protocol the 20 post-2000 studies have been so the results might not be as bad as they seem on the surface.

[anonymous]

Originally Posted by eyemsougly

it sounds like the big flaw was using VHS tapes, which was pretty much all that was available back then. As the tapes get worn down subjects might beinadvertantly cued that a certain segment was the target. Apparently there being more hits in later parts of trials gave weight to this or something.

Hopefully this link works: Ganzfeld Experiment


Anyway, since 2000 everything has been computerized. Unfortunately the article doesn't specify how true to the ganzfeld protocol the 20 post-2000 studies have been so the results might not be as bad as they seem on the surface.

According to "The Conscious Universe" published 1997, in chapter 13, p220 says this:

In fact, as we saw in chapter 5, the ganzfeld system at the University of Edinburgh used two separate video players to address this criticism, and successful effects virtually identical to those Honorton had reported earlier were still observed. Again, the implication of the criticism is that the ganzfeld results are explainable by this potential flaw, and it is not true.

Dean Radin has responded to some of the more recent criticisms on his blog:

http://deanradin.blogspot.com/2006/0...44514367222267

The post on the ganzfeld experiments that you mention by Andrew Endersby is quite comprehensive. He concludes that “... anyone who asserts that the ganzfeld has been proven by scientific standards is wrong. The field's reliance on incomplete meta-analyses seems to be cause for concern.”

However, if we assume that the “true” hit rate is the average of the 28.6% and 28.9% figures he cites, or 28.75%, and that the total number of ganzfeld-like studies is 6,700 as he cites, then the associated z = 7.09, p = 6.8E-13. This is equivalent to odds againg chance above a trillion to 1. And the funnel plot he presents rules out any obvious filedrawer effect. There is insufficient information in the article to gauge whether variations in experimental quality might be a problem, but I suspect that it wouldn't eradicate the very significant overall effect.

http://deanradin.blogspot.com/2006/0...82249701338264

The ganzfeld MA in Entangled Minds follows up on a more uniform circumscription (i.e. definition) for what constitutes the ganzfeld method. Endersby included any experiment conceivably related to the ganzfeld, whereas the published ganzfeld MAs have considered only those with hit/miss scoring. The circumscription is important because you could create a MA for any telepathy study ever conducted, and that could include perhaps millions of trials, because there were many ESP card tests conducted as telepathy tests. So where do you draw the line on what you're going to include? That's the purpose of creating a circumscription, to limit the scope of what you're studying. In any case, as I noted above, even with Endersby's database (which hasn't been independently checked as far as I know) you end up with an unambiguously non-chance effect.

http://deanradin.blogspot.com/2007/0...66483003109667

After listening to the Skeptiko interview, it reconfirms how I responded to Alcock's assertions in Entangled Minds (because his arguments in the interview are basically the same as those I wrote about in the book).

Each critique can be answered with, I believe, a reasonable counterargument. I won't bother to repeat them here.

One thing I disagree with that Alcock claims to strongly believe is that if parapsychology could provide clear evidence of a psi effect, then academic psychologists would trample each other to try to replicate these effects. And yet after Bem and Honorton's 1994 Psychological Bulletin article provided such evidence to the psychological mainstream, the response by academic psychologists was deafeningly silent.

So Alcock's view of how academia would respond to intriguing data is almost dead wrong.

I say "almost" because a recent study published in The Humanistic Psychologist does report a series of ganzfeld telepathy experiments by psychologists at Notre Dame. The article was "Finding and Correcting Flawed Research Literatures” by Delgado-Romero and Howard (2005, 33 (4), 293–303)."

Briefly, they conducted a series of eight new ganzfeld experiments. Their experiments resulted in a significantly positive overall hit rate of 32%, which is exactly the same hit rate found in a meta-analysis of 88 experiments consisting of 3,145 ganzfeld trials conducted from 1974 to 2004, as I report in EM.

Then amazingly, they explained their own successful results away by conducting a single follow-up experiment based on an ad hoc "psychic theory" they came up with, which resulted in a significantly negative hit rate!

So while psychologists weren't trampling each other to try to repeat the ganzfeld studies, there was at least one admittedly skeptical group at Notre Dame who did publish a replication, and it was successful. Somehow no one (including Alcock) paid attention to this.

I know of at least one other mainstream academic group that also tried to replicate the ganzfeld experiment, and it too was successful. But they've not yet published their results.

[anonymous]

continued from previous post

http://deanradin.blogspot.com/2007/0...36713347458643

This is a popular skeptical mantra. The more frequently it's repeated the more it seems plausible. Unfortunately, it's not true. This is not to say that there aren't occasional flaws or loopholes found in experiments. But when this happens the next round of studies fixes the problems and the same or similar results are still obtained.

I am not aware of any class of psi experiments where earlier observed effects have disappeared. There are certainly fads in which some experiments become more or less popular, but this has more to do with (the handful of) researchers following current fashions rather than previous effects going away.

That said, researchers frequently report a decline effect in these studies. I've seen this in my own studies. I suspect that most of the declines (at least the declines that I've seen) are due to psychological factors, such as boredom and/or anxiety. But as I point out in EM, decline effects can be found in all sorts of experimental disciplines, not just psi.

http://deanradin.blogspot.com/2007/0...85253780265346

Alcock is questioning whether parapsychologists have forgotten science by making scientifically unwarranted claims about the existence of paranormal phenomena.

The gigantic difference between psi research and many other denizens of uncertain frontier realms is that psi effects can be inferred solely based on empirical data collected under conditions that would be considered rigorously acceptable in any other area of science (if the topics didn't involve psi).

That is, one doesn't need a theory to demonstrate that unexpected synchronous correlations can appear in two isolated brains when the owners of those brains are asked to keep each other in mind. The fact that the resulting correlation is suspiciously similar to quantum entanglement in photons, observed under conceptually similar circumstances, gives one at least a working metaphor to play with.

If we had to wait for theory to inform us what is acceptable to observe, we'd all still be living in dank caves and eating grubs for dinner. People who require a satisfactory explanation of empirical data before accepting those data as genuine are not practicing science as I understand it.

In my view, people who are heavily theory-driven tend to resemble those who rely on religious doctrine to determine their beliefs. I am skeptical of theories not only because historically they are often found to be wrong, but because pledging allegience to a theory blinds us to alternatives that are not accounted for by the theory.

http://deanradin.blogspot.com/2007/0...80271487966629

This line of argument overlooks why proof-oriented psi experiments are conducted in the first place. People report experiences suggesting the existence of interconnections through space and time. Experiments are devised to see if those experiences are delusions or genuine. The data from those experiments suggest that in some cases these interconnections are genuine.

How to explain these experiences presents a challenge for theorists. But the fact that they occur is theory-independent.

Imagine if someone had stumbled across photon entanglement in the optics lab before invention of quantum theory. How would those correlations at a distance be regarded by most scientists? Probably as Einstein did: spooky actions at a distance, which everyone knows (based on prior theory) cannot be. and yet they are.

http://deanradin.blogspot.com/2005/0...78896396196169

I discuss these topics in some detail in Entangled Minds.

Briefly, effect size declines occur in many research domains. In psi research the declines are likely due to shifts from proof-oriented to process-oriented research. The former studies are designed to optimize effect sizes, while the latter studies are designed to investigate parameters that modulate psi ability.

Experimenter expectancy effects are pervasive throughout science. They are especially noticable in the behavior, social and medical sciences, but they also occur in the so-called hard sciences, including physics. See, e.g. this article.

In general, armchair skeptics usually offer weak critiques that fail to pass the double-standard test. That is, if the criticism applies equally to conventional scientific domains, then the criticism aimed at psi research is invalid.

The principle valid skeptical argument is that psi effects observed under laboratory conditions cannot be demonstrated with high confidence on demand. Meta-analyses suggest that the phenomena are independently repeatable, but in any given study it is not yet possible to provide a proof-positive recipe for success.

Those who hold a strong view that current scientific knowledge is more or less complete take this as evidence that psi does not exist, and that any evidence presented for psi is either flawed or an illusion.

Those who hold a more moderate (and humble) view, that science is a very recent invention in historical terms, take it as evidence that our understanding of the fabric of reality is not sufficiently comprehensive to fully explain these (and many other) natural phenomena.

My opinion is that psi will one day be explanable in rational, scientific terms. It's difficult to guess when this might occur, but given current accelerating trends in knowledge, I'd estimate that by 2020 we will have reliable demonstrations of some psi effects. To achieve this will require, among other things, some changes in current scientific epistemology.

[David Bailey]

Originally Posted by eyemsougly

it sounds like the big flaw was using VHS tapes, which was pretty much all that was available back then. As the tapes get worn down subjects might beinadvertantly cued that a certain segment was the target. Apparently there being more hits in later parts of trials gave weight to this or something.

Hopefully this link works: Ganzfeld Experiment


Anyway, since 2000 everything has been computerized. Unfortunately the article doesn't specify how true to the ganzfeld protocol the 20 post-2000 studies have been so the results might not be as bad as they seem on the surface.

It is always important to realise that tightening conditions on these experiments may also change them in other ways - a poor result should not be automatically interpreted as an indication that (possibly unintentional) cheating was occurring.

It is worth reading Sheldrake's accounts of how some skeptics tried to repeat his results.

I agree with your list of experiments with the most promise. However, I can imagine a series of experiments designed to increase the rigour of the dogs experiment:

1) Take the dog away from home and place him in a carefully monitored dog kennel - this will make it easier to avoid a variety of possible problems with the original experiment.

2) Since the dogs may be using conventional senses, the rigour of the experiment could be improved further by surgically removing or anaesthetising the dog's noses and ears.

My guess would be that after those experimental improvements had been made, the original anomalous result would disappear - if not for other reasons, because the dog would feel utterly dejected and unloved. Since the procedures I have suggested would be considered perfectly acceptable in conventional scientific experiments, I can even imagine the above scenario being played out, and the results being published as a refutation of Sheldrake's experiments!

My point is that IMHO there is such intense antipathy to Ψ evidence, that experiments can be biassed away from such results, almost unconsciously.

It is considered valid to invent an objection to a Ψ experiment without giving any actual evidence that the supposed flaw could genuinely lead to an anomalous result, as your phrase "or something", tacitly admitted!

David

PS How do you pronounce your nickname?

[eyemsougly]

pronounced like, "I am so ugly"

to anon: first of all, thanks for posting Radin's responses. I gave Hyman's criticism weight because the article I linked said that all the above chance results came from the 2nd or subsequent appearances of the target, which seems to suggest an artifact. It's potential flaws like this that make replication so important. Other labs apparently have had a tough time matching Honorton's 34% hit rate.


David: I don't think anybody except maybe Randi seriously thinks there is any real cheating going on in the ganzfeld, but things could happen to subconciously cue a receiver as to what the right target is as you said. You say a poor result with stricter controls doesn't mean that the initial results were no good and I agree, but it does mean that we don't have a psi effect that can be repeated under satisfactory controls, which is what we need.

As for the Dog's that Know thing, whatever the reason for the current results I can say with 100 percent confidence that they have nothing whatsoever to do with hearing and smell and I can't really imagine anybody who understands the experiment arguing that they do with a straight face.

I agree that antipathy towards psi makes it impossible for hardline skeptics to get significant results by themselves but what is needed, and should be possible if psi is real, is to get positive results in joint experiments between skeptics and proponents, like Alex is trying to do. Hopefully in 15 to 20 years this debate will basically be settled.

[David Bailey]

Originally Posted by eyemsougly

I agree that antipathy towards psi makes it impossible for hardline skeptics to get significant results by themselves but what is needed, and should be possible if psi is real, is to get positive results in joint experiments between skeptics and proponents, like Alex is trying to do. Hopefully in 15 to 20 years this debate will basically be settled.

I hope that happens, but I suspect that any skeptic that oversees an experiment that produces the 'wrong' result will get pilloried by his peers if he endorses the result.

Sometimes I think the only reason people believe conventional science is because it produces various types of technology. Perhaps an army of dowsers looking for oil would change opinion more (assuming they got results)!

David

[eyemsougly]

it swings both ways, if proponents and skeptics work together and get a null result there is going to be a ton of excuses from the pro psi camp. The point of any joint experiment is to convince fair minded neutral people one way or the other.

I agree that I can't see any possible experiment or results being strong enough to convince the most committed skeptics, but on the other hand considering how far I am from convinced about psi myself I don't want to criticize the doubters too hard.