Quote:
Originally Posted by davidsmith73  I have sent Steve Novella an email about the aspirin study effect size and what he thinks the study shows and whether the effect size is comparable to Radin's meta-analysis. |
I've received a reply. Basically, aspirin reduced the number of people having heart attacks from 189 in 10,000 to 105 in 10,000 people. This was a 44% reduction which just considers the proportional reduction in the number of deaths. Bem, Honorton and also Utts coverted these numbers into Cohen's h which is the difference between the arcsine transformed
absolute proportions of heart attacks within the population, ie, 2(arcsine √189/10000 - acrsine √105/10000). This results in a very small absolute effect size of 0.068 for the aspirin study.
Steve's reply suggested that the relative effect size is a more meaningful measure for the aspirin study, but he said that such a small absolute measure was reason to be sceptical of the findings and look for methodological flaws etc. He then went on to describe how the findings were later replicated in other studies etc, but that is not really the issue here.
As I'm still unsure of how the Cohen's h estimate reported by Utts compares to effects sizes in psi, I'm going to email her with some more questions.
For example, is the effect size reported in Radin's meta-analysis an absolute or relative effect size? It looks like a relative one. But if so, is it possible to calculate a corresponding absloute effects size from this?
I think this issue is quite important, because Steve Novella's argument seems to rest upon an objection to putting confidence in the detection of a signal from small effect sizes. As someone mentioned in another thread, all the other criteria he mentioned for acceptable evidence seem to have been met in psi research. It's just this small effect size that is being interpreted as "noise". However, we know that
something other than chance is responsible for the results in Radin's staring meta-analysis and I presume that Steve also also assumes it's not chance. From listening to the SGU podcast, it seems that the "noise" that Steve is using to explain the results must be methodological error. Indeed, he hints at a possible mechanism of experimenter bias through stopping trials because of a suspicion that something isn't calibrated correctly. So, I think that Steve's position ultimately comes down to - a small effect size is most likely bias in the experimental procedure. Is this fair? Is this how science should operate?