Review of: Observation of A Psychokinetic Effect Under Highly Controlled Conditions

[BannedBySkeptiko]

What I find really wrong with the paper is that it dismisses the actual result of the "highly controlled conditions" in about half a sentence: "but the z values for most experiments were too low to provide independent statistical significance." The first experiment got the right direction in 9 of its 10 units, which is significant at p=.011. The rest show failure to replicate.

All the good practice of pre-specifying the rules and procedure applies to each individual experiment. Now what about this meta-analysis? What observer witnessed it described before the fact?

Schmidt got the 8000 to 1 result of this paper not by following pre-stated rules, but by choosing what data to analyze and what analysis to do after he saw the data. He's certainly not alone in that, but it takes some nerve to do in a paper that emphasizes "tight supervision by independent observers".

The real results here are that under good controls, Schmidt got repeated failure to replicate; then when he abandoned proper practice in favor of post-hoc analysis, he got 8000 to 1. The major phenomenon demonstrated is self-deception.

[davidsmith73]

Originally Posted by BannedBySkeptiko

Schmidt is using "the normal approximation to the binomial"

Where does it say this?

For experiments 1-4 I thought he was using a non-parametric rank order test to calculate his z scores?

[BannedBySkeptiko]

Originally Posted by davidsmith73

Where does it say this?

For experiments 1-4 I thought he was using a non-parametric rank order test to calculate his z scores?

Looks like I jumped to a conclusion there. I had misunderstood how each experiment is structured, until I followed the ref to the 1986 paper.

Do you know how to justify the mean and standard deviation for his "W" statistic?

-Bryan

[davidsmith73]

Originally Posted by BannedBySkeptiko

Looks like I jumped to a conclusion there. I had misunderstood how each experiment is structured, until I followed the ref to the 1986 paper.

Do you know how to justify the mean and standard deviation for his "W" statistic?

-Bryan

Unfortunately not. I was hoping you might be able to shed some light on this!

[Im a Hedge]

I want to see if we can reach a consensus on this particular study. I'll propose a consensus statement, and folks can comment on whether they agree or disagree. Feel free to propose changes to the statement. Having re-read the posts in this thread, I believe the following statement can be accepted by everyone who has commented so far.

Proposed consensus statement:

Without access to the data, allowing independent statistical analysis, it is not possible to conclude the existence of a psychokinetic effect on the basis of this report.


I am a Hedge

[BannedBySkeptiko]

I've had a bit more time to look into the statistics in Schmidt's 1986 paper,

Observation of a PK effect under highly controlled conditions

and found Schmidt is using a "Wilcoxon signed-rank test" on the individual "sections". Once one understands how the rankings are computed, the mean (under the null hypothesis) is obvious. The standard deviation is, well, easy enough to look up.

I find some sources describe the test so that the mean is zero and the standard deviation is twice what Schmidt's formula states. Both procedures are, in the end, equivalent.

In a previous wrote in this thread, I wrote:

Originally Posted by BannedBySkeptiko

The first experiment got the right direction in 9 of its 10 units, which is significant at p=.011.

That's a valid way to look at it, except for being post-hoc. Schmidt's 1986 paper (what the 1993 paper calls experiment 1) claims the witnesses agreed on his statistical analysis methods in advance (though the programs they were given do not include computing it), so he's on reasonable ground claiming the stronger figure for experiment one of p = .0032.

I stand by my analysis that the result of the 1993 paper,

Observation of a PK effect under highly controlled conditions

is garbage. The method of the 1993 meta-analysis was *not* chosen in advance. Schmidt had to abandon the principles of his "HIGHLY CONTROLLED CONDITIONS" to manufacture his "8,000 to 1" result.


Now that I understand these experiments, they seem easy to implement -- in fact with modern Internet/Web tools and cryptographic schemes, the experiments could be much more efficient and better verified than when Schmidt did them.

If Jacob is still interested in doing experiments, this looks like a good candidate, and my own $5000 invitational paranormal challenge stands.

-Bryan Olson

[jacob]

Originally Posted by BannedBySkeptiko

If Jacob is still interested in doing experiments, this looks like a good candidate, and my own $5000 invitational paranormal challenge stands.
-Bryan Olson

Hi, Brian.

Could you come up with a good experiment design?
If it's simple enough, I could implement it. All the infrastructure is in place.

Jacob.

[BannedBySkeptiko]

Originally Posted by jacob

Could you come up with a good experiment design?
If it's simple enough, I could implement it. All the infrastructure is in place.
Jacob.

I'm up for it... with reasonable qualifications...

I'm good for controls and O.K. for analysis, but there's one key area where I'm useless: eliciting psi.

Schmidt's papers leave unclear the importance of "selected" subjects. Finding subjects with this ability will be entirely up to the pro-psi side. I'm a skeptic; I don't believe such an ability exists.

The structure of Schmidt's work here is reasonably straightforward, just hard to follow for being spread over the papers and bloated with irrelevant ramblings. I've only read the two that are on-line, and the on-line versions whacked a few details. The 1993 paper,

Observation of a PK effect under highly controlled conditions

that claims the 8,000 to 1 result (that I say is bogus) considers five experiments. Experiments are comprised of "sections"; sections are comprised of "trials"; trials are comprised of "runs". The runs correspond 1-to-1 to the random bit-vectors that subjects try to psychically bias. In Schmidt's example, the vector is 100 bits.

Assuming Jacob takes me up on my invitational $5000 challenge -- which could be a lot of fun and maybe get significant attention -- I'd be the "observer", as Schmidt described in the paper linked above, and specifically:

"Leaving the printout sealed, the independent observer randomly specifies for each test run whether the subject's PK effort should be aimed at a high score (excess of red signals in the example) or a low score (excess of green signals)."

The one major flaw in using Schmidt's procedure is that I could cheat, by unsealing the printout. We know how to fix that: from the start of your experiments here, Jacob, you posted cryptographic hashes of the targets. That works. I don't need to see, and must not be allowed to see, the target bit-vectors before I choose high-or-low; I just need assurance that they cannot be changed based on whether I choose high or low.

This post falls short of an experimental design. I need to do some more thinking, and the longer a post the fewer people willing to read it. I'll follow up, and Jacob, if you have not done so already, please carefully read the two papers at issue that are on-line.

-Bryan Olson

[BannedBySkeptiko]

Originally Posted by BannedBySkeptiko

Experiments are comprised of "sections"; sections are comprised of "trials"; trials are comprised of "runs". The runs correspond 1-to-1 to the random bit-vectors that subjects try to psychically bias.

Schmidt uses somewhat different terms in the different papers. In Observation of a PK effect under highly controlled conditions, experiments are comprised of "units", and units of "runs". I got "sections" and "trials" from the older Channeling Evidence for a PK Effect, and I have an extraneous level.

Hence forth, I'll use Schmidt latter terms, to follow table 3 in Observation of a PK effect under highly controlled conditions.

[BannedBySkeptiko]

The key idea is, as in Schmidt's title, "HIGHLY CONTROLLED CONDITIONS". Observation of a PK effect under highly controlled conditions. We negotiate all the details in advance, which is what we're on now. We fix the formats of the files and messages, and publish a program for the final scoring and stats. We should do at least one dry run; that is, test our system without the subjects.

Here's the basic outline I see:

* The parapsychologist randomly generates 7000 target bit vectors, writes them all to one file, cryptographically hashes the file, and publishes the hash digest.

* The observer cryptographically signs a messages acknowledging the hash digest and stating 7000 high-or-low calls, in on-to-one correspondence with the bit vectors.

* The subjects try to bias the bit-vectors in the direction the observer assigned.

*After 7000 trials, the observer publishes the bit-vector file.

* With the bit-vector file and the observer's high/low calls, we score the runs and compute a final p-value.


Why 7000 runs? To replicate Schmidt's meta-analysis, we need at least the statistical power he had. I counted 6836 runs over his five experiments; see table 3.

Runs are grouped into units, and Schmidt used varying numbers of runs. The variation seems arbitrary, and Schmidt doesn't suggest it has any importance to the phenomenon. I suggest 100 runs in each unit.

For those that have not carefully read Channeling Evidence for a PK Effect, the bit vectors (which correspond to "trials" in that paper and to "runs" in the meta-analysis), are grouped into pairs, in which one will be assigned a high-bias goal, and the other a low-bias goal. Thus the skeptic generates 3500 random bits to make the 7000 high/low assignments.


-Bryan Olson