[anonymous]

I'm re-reading "The Conscious Universe" by Dean Radin and the chapters on replication and meta-analysis struck me as relevant to the discussions in this forum. The key points are:

You can use an experiment with a small sample size to invalidate a phenomena. The book gives an hypothetical example. If you do an experiment where you get a result of 70 hits in 100 trials (70%) where 50 hits (50%) would be expected by chance, the odds against chance of that result would be 10,000 to one. If someone tried to replicate this experiment and did only ten trials and got seven hits (70%), would that confirm the original experiment? No. With only ten tirals, that would yield odds against chance of only five to one. Twenty to one odds against chance are usually required to claim a statistically significant result so this experiment could be called a failure and it could be said that when the original experiment was replicated, they failed to obtian a statistically significant result.

This is why meta-analysis is so valuable. The book illustrated this by describing the meta-analysis that was used to demonstrate that asprin reduced heart attacks. There were many studies that found that asprin reduced heart attacks and a few that didn't. However most of these experiments did not exclude chance with sufficient confidence. When all the trials in all studies were combined in a meta-analysis, the overall effect was positive and not explanable by chance with very high confidence.

The book also describes a fascinating process by which the latest values for the physical properties of fundamental particles are updated by the Particle Data Group (PDG) as new data are obtained. According to the PDG the process includes excluding data when "The results involve some assumptions we do not wish to incorporate." or "The measurement is clearly inconsistent with other results which appear to be highly reliable." Changes for values in the PDG reports can be due to "discarding of older data" when "it is felt that the newer data had fewer systematic errors."

I don't disagree with that process. I'll just let the reader form their own opinion on if and how this might be relevant to the controversy on parapsychology.

Interestingly, the book describes an analysis that compared replicability in hard science to replicability in social science and found that, "About 45% of the reviews in both domains exhibited statistically significant disagreements when no studies were omitted from the results." and that "Comparison [of research results in physics and the social sciences] suggests that the results of physical experiments may not be strikingly more consistent than those of social or behavior experiments." Hard science isn't hard.