The Wiseman/Watt registry

[fls]

Originally Posted by Steve Volk

Wiseman's incredibly short time limit on Ben replications is a very telling case in point. He positions himself as the authority, the guy who will collect data and studies from other people to determine if Bem's findings on precognition were accurate. Surely, Wiseman's education does qualify him for such an undertaking. But is he any more qualified than Bem? No, but he secures the mantle of leadership for himself. Then he announces the tiny window in which he will accept replication attempts, never mind that it might take twice that long even to secure funding for such an experiment.

To my own mind, there is absolutely no possible justification for limiting his consideration only to studies done over the following six months or even a year. What? He'll be too busy in a year, or two? If that's the case he shouldn't undertake the project at all.

Putting aside the baggage of the proponent/skeptic fight, if an experiment is to be replicated in order to test the idea that a 'positive' result isn't spurious and that a consistent effect is being measured, then it does become important to limit the extent to which biases give the appearance of an effect. A bias is "the combination of various design, data, analysis, and presentation factors that tend to produce research findings when they should not be produced" (PLoS Medicine: Why Most Published Research Findings Are False). Please note that this is a scientific use of the term, rather than the common use where "bias" means "prejudice".

One of the biases which leads to a false effect is testing by multiple independent teams. This is especially the case when the studies are small (less than 100 subjects) and there is flexibility within the criteria. (Note for Pollux - just because I mentioned a specific example does not mean that this represents an inclusive list of all potential biases within research.). This makes it important to specify beforehand what will be regarded as a replication, which teams are participating, and the set time period in which this will take place. Once you introduce flexibility (for example, including a study published after the cut-off date), selection and other biases can introduce the appearance of an effect where none is present.

Since the heavy lifting has been done with respect to setting up this study (Bem made the software available to run the trials), it should only take a few months for a replication. Giving people longer than necessary only increases the opportunity for multiple attempts within the same research group (which we are trying to avoid since that will increase bias). A year seems like plenty of time to complete a study in psychology. And since this study was so heavily publicized, researchers can establish whether they are interested in a short period of time (there isn't a lag for this study to be disseminated).

Realistically, if you are looking at this solely from the perspective of trying to interest scientists outside of parapsychology by following good research practices, extending the window beyond a year does little more than increase the risk of generating false results. None of this prevents other research based on extending the experiment to test other hypotheses or to establish whether 'positive' results represent something we might consider "psi". As to who should establish the registry (and the point is moot if Wiseman was the only person to step up and propose it, which seems to be the case?), neither Bem nor Wiseman seem to be regarded as trustworthy by their opponents. But since we are trying to interest non-proponent scientists, it would make sense to err on the side of someone who isn't regarded as untrustworthy by non-proponents, instead of someone who is.

On the other hand, none of this prevents Bem or another parapsychologist from setting up her/his own registry or setting criteria for replication, if this is regarded as important by the proponent community.

Linda

[David Bailey]

Originally Posted by fls

This makes it important to specify beforehand what will be regarded as a replication, which teams are participating, and the set time period in which this will take place. Once you introduce flexibility (for example, including a study published after the cut-off date), selection and other biases can introduce the appearance of an effect where none is present.

Linda

Would you apply this to the rest of science? Should there be a date, after which any higgs announcement would be invalid? I mean normally in science replications arrive as and when they arrive!

Experiments can be delayed for any number of reasons - such as potential student subjects gone on holiday, to software glitches. Placing a time limit of a replication is utterly artificial.

Furthermore, one might ask, why the urgency? If we were talking about a potential flaw in an important vaccine, it might make more sense (though I suspect a rigid cut off would be unacceptable even there).

The idiocy of this position is that the roof wouldn't fall in if there was a tentative acceptance of some evidence for precognitive learning. It would simply stimulate more work, and I suspect that is what Wiseman desperately wants to avoid - psi research going on in mainstream laboratories.

David

[fls]

Originally Posted by David Bailey

Would you apply this to the rest of science?

Of course. It is because it applies to the rest of science that there's no reason for it not to apply here.

Should there be a date, after which any higgs announcement would be invalid? I mean normally in science replications arrive as and when they arrive!

Experiments can be delayed for any number of reasons - such as potential student subjects gone on holiday, to software glitches. Placing a time limit of a replication is utterly artificial.

It depends upon the extent to which positive results are simply a measure of the prevailing bias. If you read the paper I linked to in my post, the author lists a set of conditions under which 'effect' measures are more likely to merely represent bias, and unfortunately, parapsychology operates under many of those conditions. Not all of the conditions are remedial, but some of them are. So taking more care when it comes to issues of replicability is a step towards remedying the situation. This practice has been useful in medicine (if only to show us that its absence led to false conclusions in the past), for example.

Furthermore, one might ask, why the urgency? If we were talking about a potential flaw in an important vaccine, it might make more sense (though I suspect a rigid cut off would be unacceptable even there).

The idiocy of this position is that the roof wouldn't fall in if there was a tentative acceptance of some evidence for precognitive learning. It would simply stimulate more work, and I suspect that is what Wiseman desperately wants to avoid - psi research going on in mainstream laboratories.

David

Like I said, this doesn't apply to research which extends the idea, nor does it prevent others from continuing to perform replications. It only represents one researcher's plan to do so in a way which will improve reliability and validity. Since improving reliability and validity is what gets the attention of mainstream science, it shouldn't be a surprise that more attention may be made to Wiseman's registry. However, there isn't anything preventing proponents from doing the same, especially if it can be argued that changing the criteria, like a longer replication period, reduces the number of false results. If many more people are keen to perform replications than those who registered with Wiseman (and your criticism is moot if there aren't), then they should have at it.

Linda

[Johann]

From the outset, we should make it clear that Ioannidis´paper was not meant to apply to parapsychology, but almost entirely to studies in clinical medicine - particlarly those testing correlations between genes and biological factors, where Ioannidis calculated the greatest risk of false positives.

I also feel that the unregulated overuse of the word "bias" on this board should be checked by specializing what these biases are or could be. Vague generalizations serve only to muddy the waters and create confusion, for "bias" can be used to support any position, as there is almost no limit on defintion.

If multiple teams create bias, then how? Are you referring to the file drawer? Or are you referring to a low sample size - which can be corrected by meta-analysis?

- Johann

[Sandy B]

Originally Posted by fls

One of the biases which leads to a false effect is testing by multiple independent teams.

So why bother with replications at all? If one was to follow your premise to the logical conclusion, then only the original study should be considered at all. We both know how silly that idea is. You sure jump through hoops to justify some really bad behavior on the part of people like Wiseman.

[fls]

Originally Posted by Johann

From the outset, we should make it clear that Ioannidis´paper was not meant to apply to parapsychology, but almost entirely to studies in clinical medicine - particlarly those testing correlations between genes and biological factors, where Ioannidis calculated the greatest risk of false positives.

The issues which Ioannidis brings up are not confined to any field. Multiple testing generates false-positives in the same way whether it is done in an interventional study or looking for associations with personality traits. Slapping the label "parapsychology" onto a study does not suddenly make it immune to the effects of insufficient blinding or publication biases.

I also feel that the unregulated overuse of the word "bias" on this board should be checked by specializing what these biases are or could be. Vague generalizations serve only to muddy the waters and create confusion, for "bias" can be used to support any position, as there is no limit on its accursed definition!

- Johann

I agree. That is why I tend to give specific examples of biases and how they produce false results or (like I did in my post above) refer to the list Ioannidis provided in his paper.

Linda

[Sandy B]

Originally Posted by fls

Multiple testing generates false-positives in the same way whether it is done in an interventional study or looking for associations with personality traits.

Again, this suggests that replications are wasted efforts. We both know that this isn't true.

[fls]

Originally Posted by Johann

If multiple teams create bias, then how?

It makes it easier to focus only on sets of trials which returned a 'positive' result.

Statistical tests are applied under the assumption that the results were obtained in isolation - an assumption which is violated in this case.

Multiple teams easily allow for flexibility in terms of outcomes, analysis and tested relationships.

Multiple teams tends to mean smaller studies which increases the false-positive rate (decreased power means that the number of true-positives decreases relative to an unchanging number of false-positives).

Are you referring to the file drawer?

That too. It is easier to abandon less successful studies when the individual cost is low because the total effort has been divided up into little pieces.

Or are you referring to a low sample size - which can be corrected by meta-analysis?

- Johann

Meta-analysis can increase the power, but it does not correct bias. And multiple teams are introducing bias (specific examples given above and in Ioannidis' paper).

Linda

[David Bailey]

Linda,

I think what you, and Ioannidis in his paper are saying, is that great swathes of conventional science are in deep trouble. In that I would agree - that such a percentage of published medical research results are irreproducible, is nothing short of a scandal.

If there was a level playing field to judge all research things (psi or conventional) would look very different.

I would also remind you that Bem's result does not come from nowhere. There is the replicated presentiment results which also show a precognition effect.

Maybe the truth is that conventional science is often trying to answer impossible questions. Say for example, you have an anti-inflammatory drug used for arthritis, and you want to know if it causes a small excess of death or serious complications. The problem, I suspect, is that it is simply impossible to get a suitable control group. If you compare with a group that don't take any drug, they may end up taking less exercise, if you compare with a group taking another drug, you may still not have a valid comparison for one reason or another. Despite this, endless epidemiological work of this type is published.

In the case of climate science, if you measure a small increase (of about 1 deg C) in global average temperatures, and you want to attribute it to CO2, you have to assume the climate would otherwise be constant. Thus a huge amount of work went into trying to show that the climate hadn't changed for 1000 years! BTW, if you are interested in the misuse of statistics, you might enjoy reading about climate science!

David

[fls]

Originally Posted by David Bailey

Linda,

I think what you, and Ioannidis in his paper are saying, is that great swathes of conventional science are in deep trouble.

I don't think either of us are saying that. You have to remember that much of science operates under conditions which obviates the problems mentioned (like higher pre-test probabilities). Where you see problems is where research is exploratory (gene assays) or people try to act on research which have a high rate of false results (alternative medicine). The extent to which false results may be generated tends to correspond to those areas regarded as less well established. In that way, a general conservatism helps to keep conventional science out of trouble.

Linda