Quote:
Originally Posted by Topher Cooper 
What is not a valid criticism is to say that you are unable to find any problem with the experiment's design or conduct but the effect size is small and there must be some error in it. That is not criticism -- that is just stating ones prior belief without any sign of supporting evidence.
|
I agree. This is why when Steve Novella said the result of Dean Radin's meta-analysis was "just noise", he actually meant the results were down to methodological error. He can't seriously be suggesting that the results were due to chance at such a high significance level.
So, like you say, this is really a belief that methodological error got in there
somehow even though it can't be proven. Fortunately for Steve Novella, he suggests a source of bias that would not necessarily be reported in the methods section of a paper, such as subconsciously stopping bad performance trials to check for "callibration". In the staring experiment, I'm not sure what the experimenter would need to "calibrate" however!
If large effect sizes were reported, I wonder how many sceptics would still believe that some unidentified methodological error was to blame?