Review of: "Electrodermal Presentiments of Future Emotions"

[BannedBySkeptiko]

Originally Posted by Larry Boy

I appreciate your comments and I think they're useful, but I think we could have a lot more productive discussion if you didn't refer to people as idiots.

Oops, yes, sometimes my true feelings accidentally slip out.

Have you seen Penn and Teller's program on Showtime, or heard Christopher Hitchers argue against religion? Those guys are my heroes.


-Bryan Olson

[Larry Boy]

Originally Posted by BannedBySkeptiko

Oops, yes, sometimes my true feelings accidentally slip out.

Have you seen Penn and Teller's program on Showtime, or heard Christopher Hitchers argue against religion? Those guys are my heroes.

Well, I respect people like those because they stand up for the weak in society and oppose injustice and fraud. I don't like the way they do it, though.

[davidsmith73]

Originally Posted by BannedBySkeptiko

The paper reports three attempts to replicate a previous result, of which two were failures. The way Radin chooses to spin things, a careless reader might mistake the results as a a case *for* the effect being repeatable.

So we have two experiments that were individually significant and two that were not (in this paper anyway; there are other presentiment experiments reported by other researchers). It's an unsatisfactory situation indeed but in the absence of an experimental procedure that is guaranteed to produce a significant effect, the best way to proceed is meta-analysis. Otherwise, we don't know whether the number of experiments that produced positive results are the number expected by chance alone. Wouldn't you agree that doing a meta-analysis on all the data is more satisfactory than doing no meta-analysis at all?

[BannedBySkeptiko]

Originally Posted by davidsmith73

Wouldn't you agree that doing a meta-analysis on all the data is more satisfactory than doing no meta-analysis at all?

I would not agree that Radin's meta-analysis here is better than useless, beyond that it motivated him to report the rest of his data. Here's why:

(quoted out of order):

Originally Posted by davidsmith73

the best way to proceed is meta-analysis. Otherwise, we don't know whether the number of experiments that produced positive results are the number expected by chance alone.

Not really. The individual results of the four experiments answer the was-it-all-just-chance question better than does the meta-analysis, and that's the only question that the meta-analysis attempts to answer.

Radin claims that out of four experiments, one hit p=0.0004, and another p=0.002. Does that not resolve the chance issue? If not, how do the weaker p=0.001 and p=0.008 (not independent) of the meta-analysis help? Simple logic: if any of the experiments' results are not due to chance alone, then it is not the case that they are all due to chance alone.

Don't choose an over-all meta-statistic after the fact. With four experiments, just state the four outcomes. The conclusions are clear: Are Radin's results due to pure chance? Obviously not. Has Radin found a consistent demonstration of psi? Obviously not.


Elsewhere, I recall one author stating that this kind of meta-analysis convinced him that the effect at issue replicated reasonably well. That is so totally, utterly, completely wrong; I rolled my eyes. It would be funny were it not so sad. The combined p-value here, like the Stouffer z for for the Ganzfeld database, says nothing about replication, consistency, repeatability, etc.

The p-value of an outcome is the probability of getting a result at least as extreme entirely by chance*. Meta-analysis combines the outcomes of several experiments, and if any experimenter made any significant mistake in any of the trials of any of the experiments, then the combined result set not entirely due to chance, and the statistic is correct to scream that news at us.

Today, parapsychology is largely, perhaps primarily, justified by abuse of meta-analysis. Whether or not psi exists, the more experiments we add to a data set, the less likely that 100% of the results are due to chance. The less likely that 100% of the results are due to chance, the farther out the p-value is likely to go, because the p-value states the probability of a result this far out under the assumption that 100% of the results are due to chance.

The best arguments on the side of parapsychology are entirely consistent with the non-existence of psi.


-Bryan
*For more, search up "p-value", "null hypotheses", and "research hypothesis".

[davidsmith73]

Originally Posted by BannedBySkeptiko

Radin claims that out of four experiments, one hit p=0.0004, and another p=0.002. Does that not resolve the chance issue? If not, how do the weaker p=0.001 and p=0.008 (not independent) of the meta-analysis help? Simple logic: if any of the experiments' results are not due to chance alone, then it is not the case that they are all due to chance alone.

That's not my understanding of statistically based science. Saying that the results of an experiment are not due to chance is ultimately based on an agreed cut-off point, i.e., p-value of 0.05 or 0.01 etc.

If the null hypothesis is true then we would expect 1 in 20 (or 1 in 100 depending on your choice) experiments to obtain a significant result. So let's say we perform five experiments. The first four get a p-value of 0.5. The fifth gets a p-value of 0.05 and we have to decide whether this experiment represents a real effect of just chance. Would you be confident that this experiment was due to a real effect? I wouldn't. I would take note that all the other experiments failed to reach significance and it is likely the case that the fifth experiment was just a fluke since we should expect 1 in 20 experiments to obtain significant results at a 0.05 probability. For that reason, I would combine the data from all the experiments to see if the results of the entire data set are in line with chance expectation.

The p-value of an outcome is the probability of getting a result at least as extreme entirely by chance*. Meta-analysis combines the outcomes of several experiments, and if any experimenter made any significant mistake in any of the trials of any of the experiments, then the combined result set not entirely due to chance, and the statistic is correct to scream that news at us.

It is quite possible that inadvertant methodological error could have produced the results of any of these experiments. Is there reason to think that is the case?

It's also possible that the results of the experiments were due to a real effect, not error.

Whether or not psi exists, the more experiments we add to a data set, the less likely that 100% of the results are due to chance.

From my understanding this would only be true if there were a systematic error occurring in most experiments that were added to the database. If there is no systematic error in the experiments then adding more to the database will make it less likely that a significant result will be obtained.

[BannedBySkeptiko]

Originally Posted by davidsmith73

That's not my understanding of statistically based science. Saying that the results of an experiment are not due to chance is ultimately based on an agreed cut-off point, i.e., p-value of 0.05 or 0.01 etc.

David, I am not responsible for your level of "understanding of statistically based science". If I'm wrong, please prove it. I try to present my case in definite, specific statements, and to use standard terminology. When I get something factually wrong -- as I did in another thread when I said Schmidt was using the normal approximation to the binomial -- there is no doubt about it.

Sure, the "cut-off point"s you cite are the values anyone would find as 'significant' and 'highly significant' if they Googled the terms with no real understanding of the issues. What do they have to do with the questions still open here?

I noted two of four experiments reaching p=0.0004 and p=0.002. David, you now respond: "Saying that the results of an experiment are not due to chance is ultimately based on an agreed cut-off point, i.e., p-value of 0.05 or 0.01 etc". Did you not even realize what I conceded? How many orders of magnitude do I have to give you?

Agree, disagree, or don't respond -- all of those are fine. But this non-sequitur is just a waste of time. I tried to state it clearly: "Are Radin's results due to pure chance? Obviously not. Has Radin found a consistent demonstration of psi? Obviously not."

Originally Posted by davidsmith73

If the null hypothesis is true then we would expect 1 in 20 (or 1 in 100 depending on your choice) experiments to obtain a significant result. So let's say we perform five experiments. The first four get a p-value of 0.5. The fifth gets a p-value of 0.05 and we have to decide whether this experiment represents a real effect of just chance.

Would you be confident that this experiment was due to a real effect? I wouldn't. I would take note that all the other experiments failed to reach significance and it is likely the case that the fifth experiment was just a fluke since we should expect 1 in 20 experiments to obtain significant results at a 0.05 probability. For that reason, I would combine the data from all the experiments to see if the results of the entire data set are in line with chance expectation.

Blah blah blah... In my previous post, David, I already argued your side better than you do above. The results are clearly not due entirely to chance.

On the other hand:

Originally Posted by davidsmith73

It is quite possible that inadvertant methodological error could have produced the results of any of these experiments. Is there reason to think that is the case?

Yes, there is definitely reason to think this is a case of error. Error is a common and well-documented phenomenon; there is no serious question of its existence; in fact it happens quite frequently. Psi, on the other hand, appears to be myth, as well as science can tell. In seventy years or so of serious effort, parapsychology has failed to find even one single consistent demonstration that the subject it is studying even exists.

In the case of Radin, I can definitely show uncorrected errors in other work, and personally I find them hard to excuse as "inadvertent methodological".

Originally Posted by davidsmith73

It's also possible that the results of the experiments were due to a real effect, not error.

"It's also possible that" monkeys will fly out of the Queen of England's butt.

Originally Posted by davidsmith73

From my understanding this would only be true if there were a systematic error occurring in most experiments that were added to the database. If there is no systematic error in the experiments then adding more to the database will make it less likely that a significant result will be obtained.

I already explained this, and if you think my explanation to be wrong, please state where it fails. The meta-analysis p-value is the probability of results as extreme as what was actually observed happening entirely by chance. Even in the absence of psi, the more experiments in a data set, the greater the chance that the results are not entirely due to chance. Where do you think is my mistake?

Your understanding is different? Well either tell me specifically where mine is wrong, or fix yours. Or shut up. Or post more garbage for me to make fun of. Your options; I stated them in my own personal order of preference.

-Bryan

[davidsmith73]

Wow, this topic really touches a nerve with you Bryan!

I'm not here just to 'explain' things to the 'other side'. I'm here to discuss issues and learn about other people's ideas and thoughts. If I start a post with "it's my understanding...", take that as a sign of uncertainty on my part and an invitation to further explain your point if you so wish.

Originally Posted by BannedBySkeptiko

"Are Radin's results due to pure chance? Obviously not. Has Radin found a consistent demonstration of psi? Obviously not."

Your argument differs from mine. You are only considering the results of the individual experiments in isolation. I am considering the result of the entire data set combined. My previous post was intended to make the argument that Radin was right to analyse the entire data set as a whole because this strengthens the conclusion that the positive results of the individual experiments were not due to chance.

I think that the main reason why parapsychologists combine results of experiments in this way is because of the critics. Not all experiments get positive results so the critics say that the positive results could have been a fluke. Hence the motivation to combine the data into one big experiment.

Yes, there is definitely reason to think this is a case of error. Error is a common and well-documented phenomenon; there is no serious question of its existence; in fact it happens quite frequently.

The fact that errors happen is not really a good reason to think it happened here. Why do you think error was responsible for the results of these particular experiments?

The meta-analysis p-value is the probability of results as extreme as what was actually observed happening entirely by chance. Even in the absence of psi, the more experiments in a data set, the greater the chance that the results are not entirely due to chance. Where do you think is my mistake?

No mistake. Like I said, that is true only if there is a systematic error occuring in the experiments that are added to the database. Is that what you think is happening in meta-analyses of psi experiments? That the results are not due to ESP but Error Some Place?

[BannedBySkeptiko]

Originally Posted by davidsmith73

You are only considering the results of the individual experiments in isolation.

Were the experiments done in isolation, or did Radin state before the fact that these experiments were meant to be analyzed together?

Originally Posted by davidsmith73

My previous post was intended to make the argument that Radin was right to analyse the entire data set as a whole because this strengthens the conclusion that the positive results of the individual experiments were not due to chance.

I see nothing in your previous post that resembles an argument for that. Can you reasonably argue that p=0.008 in post-hock analysis is stronger evidence against chance than experiment one reaching p=0.002 and experiment three reaching p=0.0004?

Originally Posted by davidsmith73

I think that the main reason why parapsychologists combine results of experiments in this way is because of the critics. Not all experiments get positive results so the critics say that the positive results could have been a fluke. Hence the motivation to combine the data into one big experiment.

By "fluke", do you mean essentially perfect experiments that got significant results by pure chance, or does an experimenter mistake, or even a lie, count as a fluke?

Originally Posted by davidsmith73

The fact that errors happen is not really a good reason to think it happened here. Why do you think error was responsible for the results of these particular experiments?

Because it is so much more likely than the alternative. When a bank teller's cash drawer comes up short, perhaps it was dishonesty or error, but it could be a paranormal phenomenon. If you think it was either theft or error, what evidence do you have, beyond your closed-mindedness to the idea of psi dematerializing currency?

Originally Posted by davidsmith73

No mistake. Like I said, that is true only if there is a systematic error occuring in the experiments that are added to the database.

No mistake, yet you think it is wrong. Great.

Originally Posted by davidsmith73

Is that what you think is happening in meta-analyses of psi experiments? That the results are not due to ESP but Error Some Place?

I think there's a whole bunch happening. Incompetence, compromise, fraud, misunderstanding, selective reporting, spin, etc. Just not psi.

Other fields use this kind of meta-analysis when an effect size is small and they need greater statistical power. That's not at issue here. Radin's results are simply inconsistent.


-Bryan

[davidsmith73]

Perhaps the discussion would be better served by revisiting the following point. You said:

Originally Posted by BannedBySkeptiko

Simple logic: if any of the experiments' results are not due to chance alone, then it is not the case that they are all due to chance alone.

That is incorrect. By chance alone, we would expect 1 experiment in 20 to reach a significance level of 0.05 and thus be judged as 'not due to chance alone' if considered in isolation. That's just a principle of probability and makes your 'simple logic' invalid.

Until we rectify this disagreement I see little point in continuing the discussion.

[Im a Hedge]

Originally Posted by davidsmith73

By chance alone, we would expect 1 experiment in 20 to reach a significance level of 0.05 and thus be judged as 'not due to chance alone' if considered in isolation. That's just a principle of probability and makes your 'simple logic' invalid.

Pardon me for interrupting. I think if it is stipulated that one of the results is in fact not due to chance, rather than being judged to be not due to chance, then Bryan's reasoning would hold. It's possible that one result would reach a level of statistical significance, but the set of results would fail to reach significance when combined. This can happen if there is an actual effect and it can happen if there is not an actual effect. If I understand Radin's approach, he is combining all the results in a meta-analysis to show that the positive results are not a fluke. I don't think this is valid, because it's possible that the few positive results are swamping out the negative results, making the entire set look positive.

What if he combined all of the experiments that are not significant individually, but excluded those that are significant individually? Would this allow greater statistical power to identify an actual effect, without the danger of biasing the outcome by a couple significant individual results? This would seem to avoid the "Bill Gates walks into a bar" fallacy. The negative results could be failing to reach statistical significance simply due to sample size, which is a valid reason for pooling results.

I am a Hedge