Review of: Observation of A Psychokinetic Effect Under Highly Controlled Conditions

[Im a Hedge]

OBSERVATION OF A PSYCHOKINETIC EFFECT UNDER HIGHLY CONTROLLED CONDITIONS
Author: Helmut Schmidt
Observation of a PK effect under highly controlled conditions

In another thread on this forum, I have been asking for commenters here to point me towards some of the good experimental studies in parapsychology. This study is one that was suggested, so I thought I would post my initial review of the study here.

Thanks to Larry Boy for pointing me to this paper.

General comments:
It looks like this experiment is modeled after the delayed-choice double-slit experiments in quantum physics. The basic idea of those being that the past appears to be effected by choices made in the present (or the present is effected by choices mad in the future).
I won't try to summarize the whole thing, as any interested parties can read the original at the link given above.


Problems with the experiment:
No negative controls. I will continue, and describe some other problems, but this single problems renders the experiment inconclusive. A negative control is critical in order to determine that a particular variable (in this case, the intention of the subjects) is responsible for the observed results, and not some other variable. This is central to experimental science. This is the biggest oversight I see in experimental designs (in all areas, not restricted to psi). Whenever you are examining an experiment, always ask yourself, "where is the negative control?". If you can't find it, you cannot trust the results.

On to other issues. It is not clear to me that the statistical methods being used are the best in this circumstance. Usually, when there is a binary outcome and you are looking for deviations from chance, the binomial distribution is the method of choice. This basically lets you calculate the chances of getting a particular number of heads when flipping a (fair) coin some number of times. This seems to be the natural way to analyze the data from this experiment, but this is not what Schmidt chose. (He may have good reasons, and maybe someone can suggest why his method makes more sense). Instead, he uses a z-score. It is not clear how he calculates the standard deviation, and it is not clear to me that this method is statistically valid in this case.

Five experiments are performed, and the results are not significant, by the authors criteria. Schmidt states,
"All experiments gave score deviations from chance in the desired direction, but the z values for most experiments were too low to provide independent statistical significance. "
That is, every experiment failed to show an effect distinguishable from chance.
Then the z scores from the five experiments are combined. I hope someone familiar with this can help me out here, because I don not understand the justification for the formula used to combine the scores. I could see using a mean of the z scores as possibly making sense, but that is not what he does. He uses the formula:
z = (z1 + z2 + ... + zn) / sqrt(n)

The sum of the z-scores is divided by the square root of the n. To get the mean, of course, you would divide by n. This seems to have the effect of inflating the resulting 'combined z score'. The final z score of 3.67 is presented. The mean of the five z scores is 1.64 (not significant). I'm not sure that it is legitimate to manipulate z-scores like this, and I certainly don't understand why the square root of n is used instead of n. I am not a statistician, so perhaps someone can set me straight on this.

Summary data are presented. It would be good to see the data that underly Table 2. Table 2 presents the calculated z scores for each experiments. I would like to see that 'raw' data from each run. That is, how many 1s and how many 0s for the run, plus the assignment of the independent observer for the run. With these data, anyone (e.g. myself) could re-analyse using other statistical methods.

In conclusion, what I am seeing from this study is an experiment without any negative controls that failed to demonstrate a significant deviation from chance results. I readily acknowledge that I read the paper quite quickly and have not studied it in depth, so I may be overlooking key elements. If so, I hope someone here can point this out to me so that I will better understand the study.

I am a Hedge

[Larry Boy]

Very good initiative, Im a Hedge.

What do you think about the following?

"If the subject's effort was successful, the independent observer can confirm this first-hand by a tendency of the scores to point in the directions (positive or negative) that he had randomly specified. Because the independent observer had randomly assigned the directions, he or she can be certain that no such systematic tendency should occur (in the absence of the claimed anomaly)."

In theory, a random sequence should produce an approximately equal amount of "green" and "red" hits, creating a base line that fluctuates around chance:

a)
--------------------------------------

The results, however, seemed to follow the intended direction (positive or negative) decided at random by the experimenters, thus giving us something like this:

Positive

b)
+____________________________________--------------------
0-------------------------------------------------------------
-

Negative

c)
+
0_________________________________________________________
-------------------------------------------------_______________


Ideally, a control situation where no one is trying to interfere psychically with the random number generator should have been included. However, such experiments have been conducted quite a number of times before by parapsychologists, and the results in the actual test sessions have proven better than in the control sessions, which as far as I'm aware have only produced random results. One could thus assume that Schmidt trusted his random number generators to produce random results this time also. Which, by the way, is what we should expect by such a machine if it isn't malfunctioning. And even if it is malfunctioning, it shouldn't be able to produce results consistent with the randomly decided intended directions of the individual trials.
In other words, I'm not convinced the control issue really is an issue in this experiment. However, I would be glad to hear your reasons for being concerned about this if you disagree.

I can't comment on the statistical issues.

[Im a Hedge]

Originally Posted by Larry Boy

In other words, I'm not convinced the control issue really is an issue in this experiment.

Let's see if I can explain why it is an issue. The experiment is looking at a single variable: the ability of human intention to alter the output of a random number generator. The purpose of the negative control is to isolate that single variable from as many other variables as possible. The great thing is that we don't have to know what these other variables are in order to remove them, as long as we have a good negative control. The fact that other researchers have established the validity of the random number generator only tells us that it can serve as a good control. That alone does not remove the need to include the control in the experiment. Think of all of the variables that are involved in any experiment. Time, temperature, humidity, solar flares, wind speed, bird migrations, volcano eruptions, and so on... If the control performed at a different time, and a different place by different people, then none of these things are actually controlled for. However, if the negative control is performed alongside the other tests, then all of these things are now in common. So now, we can generally rule out extraneous variables.

In this experiment, the independent observers could be given additional sealed printouts that they would treat exactly the same as the others. In fact, they would not know which ones are the controls. The only difference between these printouts and the experimental printouts would be that no subject would attempt to alter the results of the control printouts. The controls would be scored alongside the others, and again the person doing the scoring would not know which are the controls.

Without including a control along these lines it is not possible to know if there was any effect due to the variable of interest.

I hope that makes some sense, and helps explain why negative controls must always be included for experimental results to be meaningful.

I am a Hedge

[Larry Boy]

Originally Posted by Im a Hedge

I hope that makes some sense, and helps explain why negative controls must always be included for experimental results to be meaningful.

I see your point, and I agree that such a control should have been included. However, I'm not sure I would go as far as to say that the results aren't meaningful. I simply don't see how any unknown factor of the kind you mentioned ("Time, temperature, humidity, solar flares, wind speed, bird migrations, volcano eruptions, and so on...") could explain statistically significant results in the intended directions randomly decided upon in advance. Such factors may perhaps be able to bias the outcome in one way or another in an individual trial, but for these factors to correspond over and over again - that is, to a statistically significant degree - to the intended directions, you would have to invoke some kind of consciousness to these natural forces! (Or attribute the results to chance, which of course you can do anyway, whatever controls you put in place.)

[Im a Hedge]

I intentionally used a list of things that I can't imagine impacting this particular experiment. My point is that confounding variables often turn out to be things nobody would have imagined. That's the beauty of a good negative control. You don't have to know what your controlling for. You effectively control for almost everything, no matter what it is.

Also, keep in mind that this particular study doesn't seem to have shown an actual effect. A negative control helps you narrow down the cause of an observed effect. In analyzing these results, the lack of a negative control is not very important, because there is no effect that needs to be explained. That is, unless I have really misunderstood the analysis (which is always a good possibility). The flaw with the experimental design is that the lack of a negative control would prevent you from drawing any meaningful conclusions even if there had been a strong observed effect.

I am a Hedge

[BannedBySkeptiko]

Originally Posted by Im a Hedge

In another thread on this forum, I have been asking for commenters here to point me towards some of the good experimental studies in parapsychology. This study is one that was suggested [...]

Wow -- this paper was supposed to be a *good* experimental study? It's atrocious. In this post I'll answer the statistical questions, which is not really where it goes wrong. I'll come back and rag on the paper later, unless the cowards in charge here ban me from posting again.

Originally Posted by Im a Hedge

It is not clear to me that the statistical methods being used are the best in this circumstance. Usually, when there is a binary outcome and you are looking for deviations from chance, the binomial distribution is the method of choice. This basically lets you calculate the chances of getting a particular number of heads when flipping a (fair) coin some number of times. This seems to be the natural way to analyze the data from this experiment, but this is not what Schmidt chose. (He may have good reasons, and maybe someone can suggest why his method makes more sense). Instead, he uses a z-score. It is not clear how he calculates the standard deviation, and it is not clear to me that this method is statistically valid in this case.

Schmidt is using "the normal approximation to the binomial" (Google the terms for more). He has enough data that he doesn't want to deal with the large numbers that calculating the exact binomial would require. The standard deviation is just that of the binomial; that much is easy to calculate: square root of (number of trials X probability of success X probability of failure).

He's making a mistake in that each of his "units" is just one coin toss as far as the observers can verify, so computing a Z for each and combining them is invalid. His argument that this shouldn't matter much is, like many hand-waves, wrong in principle though almost right in how the experiments turned out.

Originally Posted by Im a Hedge

Then the z scores from the five experiments are combined. I hope someone familiar with this can help me out here, because I don not understand the justification for the formula used to combine the scores. I could see using a mean of the z scores as possibly making sense, but that is not what he does. He uses the formula:
z = (z1 + z2 + ... + zn) / sqrt(n)

The sum of the z-scores is divided by the square root of the n. To get the mean, of course, you would divide by n. This seems to have the effect of inflating the resulting 'combined z score'. The final z score of 3.67 is presented. The mean of the five z scores is 1.64 (not significant). I'm not sure that it is legitimate to manipulate z-scores like this, and I certainly don't understand why the square root of n is used instead of n. I am not a statistician, so perhaps someone can set me straight on this.

Schmidt is using Stouffer's z statistic, which is one of several technically defensible ways to combine scores. Combining imprecise values tends to amplify the imprecision, and in this paper he does it at two levels: Within each experiment he combines the (unverifiable) z-scores for the units into a single z-score for the experiment; then he combines each experiment's z-score to claim the one-in-8000 result of the paper. Imprecise as it is, it is not that big a deal compared to the major defects.

Why is the Stouffer z valid, and one should not just average the z-scores? Again GIYF, but I'll try to motivate the issues. First, when using Stouffer's z statistic, we represent mostly-miss outcomes with negative z-scores. No matter how many values we combine, the expected sum under the null hypothesis is zero.

Why not just average the z-scores? Let's look at an example: Suppose our experiment is to flip a fair coin 3 times while trying to psychically cause 'heads'. We repeat our three-flip experiment 1000 times, and each time we get 2 'heads' and one 'tails'. Each individual experiment is insignificant, and if we averaged all those identical scores we'd get that same insignificant number. Hitting an even-odds shot twice in 3 trials is insignificant, but the chance of hitting it 666 times in a thousand independent trials is less than one in 1000000000000000000000000.

Stouffer's Z is mathematically defensible in meta-analysis. The real problem with it is that parapsychologist pretend that they are just following normal scientific methods, when in fact real scientific disciplines know such meta-analysis to be notoriously unreliable. Other sciences offer repeatable demonstrations; parapsychology offers excuses and special pleading.


-Bryan

[Larry Boy]

Originally Posted by Im a Hedge

I intentionally used a list of things that I can't imagine impacting this particular experiment. My point is that confounding variables often turn out to be things nobody would have imagined. That's the beauty of a good negative control. You don't have to know what your controlling for. You effectively control for almost everything, no matter what it is.

That's why I agree a negative control should have been used, because then we could have been even more certain about the results. As I explained in my earlier post, though, I think the directions randomly decided for each trial rules out the possibility of systematic bias. I cannot be certain, of course, but until somebody can come up with an alternative explanation I think it's reasonable to assume that this is the case. I mean, how really would natural forces of one kind or another somehow "know" how to correspond to a randomly determined set of directions?

Originally Posted by Im a Hedge

Also, keep in mind that this particular study doesn't seem to have shown an actual effect.

Can't comment on the statistics, but it seems bold to make a claim like this at the same time as you admit that you do not fully understand the methods being used. Maybe you're right, though. I don't know.

[Larry Boy]

Originally Posted by BannedBySkeptiko

I'll come back and rag on the paper later, unless the cowards in charge here ban me from posting again.

I hope you will be able to continue to contribute to this discussion, your comments are much appreciated. I must admit though that I have no idea what you're talking about, as my knowledge about statistics is close to zero. That only goes to show, however, that without a firm grasp of scientific methods it's difficult to evaluate the status of psi research.

[Im a Hedge]

Originally Posted by Larry Boy

As I explained in my earlier post, though, I think the directions randomly decided for each trial rules out the possibility of systematic bias. I cannot be certain, of course, but until somebody can come up with an alternative explanation I think it's reasonable to assume that this is the case.

I disagree that this would be reasonable. The fact that we can't imagine what the confounding variable might be does not mean there could not be one. The history of scientific discovery demonstrates that human imagination is not sufficient to the task; the universe will still surprise and amaze us. The methods of science have been honed to deal with this. The great thing about a good negative control is that it is not limited by our personal lack of knowledge or lack of imagination. It is not a good idea to assume that a control is not needed because you can't imagine an unknown factor that would interfere with the results.

Originally Posted by Larry Boy

Can't comment on the statistics, but it seems bold to make a claim like this at the same time as you admit that you do not fully understand the methods being used. Maybe you're right, though. I don't know.

I don't think it was more bold than warranted. You'll notice I said the study doesn't seem to have shown a actual effect. I think that is a true, and justifiable, statement. One thing that adds to this is the fact that I can envision a much simpler way of presenting the results (provided I have understood the methods). The fact that the results are presented in a manner which I have some difficulty following is troubling. It sounds like there were a lot of 'coin tosses', and we want to know if the number of 'heads' is significantly different from what we would expect by chance. I would think a simple binomial test would tell that. The fact that this single result is not presented raises some questions. It doesn't mean that the results are not valid, it just makes one wonder why this simple analysis wasn't performed. As reported, I can't figure out how many 'tosses' there where, or how many were 'heads'. If these two values were reported, we could do the statistical test ourselves. (Although the lack of controls means we won't be able to interpret the results.)

Someone with a better understanding of statistics may be able to explain why my expectation is flawed.

I am a Hedge

[BannedBySkeptiko]

Originally Posted by Larry Boy

I hope you will be able to continue to contribute to this discussion, your comments are much appreciated. I must admit though that I have no idea what you're talking about, as my knowledge about statistics is close to zero. That only goes to show, however, that without a firm grasp of scientific methods it's difficult to evaluate the status of psi research.

Thanks, and I hope you'll regard your lack of knowledge about statistics as a temporary and curable condition.

-Bryan