Dr Steve Novella and The (Pseudo?)Skeptics' Guide to the Universe

[Open Mind]

The Skeptics Guide - Skepticast #123: 11/28/2007

starting at 31.10

Novella very embarrassingly defends his 'research' going to a 'psychic fair' having doctored photographs, approaching 3 fortune tellers and proclaiming his feeling on the outcome of the matter was 'zero' or 'zippo' .

I'm sure if someone faked a medical condition (e.g. produced artificial damage to skin surface) , gave false information and went along to tape record Dr Steven Novella, he would grossly misdiagnose or get 'zippo' right. Following his own logic, is he implying such experiments would make him unqualified to be in medicine or a fraudulent practitioner?

During the skepiko interview Novella claims 'The goal is to design an experiment in such a manner it doesn't matter what you believe Nobel sounding words indeed, yet in his own trial, we see the very opposite action at work, he chose money seeking fortune tellers, unaware they were being tested or deceived, set the bar high (assuming any scientific measurement was employed at all) so that what is under test is extremely unlikely to contradict ones own cherished viewpoint.

One thing become increasingly clear from listening to his broadcast, Novella feels somewhat hostile towards paranormal claims and dismisses 80 years of controlled lab research ( i.e. not a trip to a psychic fair for an easy debunk) as 'noise' . Para psychological effects have been shown to be above 'noise', would he also dismiss the existence of neutrinos from the sun because the measurement of effects are also weak and erratic?

Novella statements about what percentages indicate a real effect are extraordinary arbitrary, one could dismiss much of his own medical field's statistical outcomes merely because these are lower than other areas of science such as physics. If Novella is unwilling to trust meta-analysis in that case particle physics, clinical medicine, psychology are all in serious trouble in his 'guide to the universe'.

Novella's viewpoints merely add 'noise' and confusion upon the work of those parapsychologists doing serious minded research into the subject of psi.

[Open Mind]

Perhaps the most erroneous statement in this programme was the badly informed opinion that most or the better research is done by the skeptics. The example of James Randi is given as 'plentiful and bountiful' Let us look closer ....

James Randi who used to advertise himself as the 'world's most tireless paranormal investigator' has tested less than 2 people per year for the 1 million dollar challenge , according to his own former Challenge Manager.

Buried in amongst thousands of JREF forum topics and post ... is this post ...

VERY few tests take place, as very few applicants ever agree to an acceptable test protocol. There have been two tests since my arrival here one year ago. My understanding is that there has never been more than a couple of tests per year. - Kramer, JREF Prize Challenge Manager in 2005

Can we get a Challenge historical wrapup/tally? [Archive] - JREF Forum

In 2007, JREF decided to reduce the number of actual tests further by adding 2 further stipulations (1) The entrant must have a media presence (2) Support from a university academic (which they seem to have broke shortly afterwards to test a woman who claimed she could make people urinate)

Now lets look at what is easily found on JREF website main pages (not hidden amongst countless JREF forum posts)

James Randi Educational Foundation - Fudging Information EXAMPLE 1

JREF are trying to sound sooooo busy ....

1.3 How many people have applied for the Challenge?

Between 1964 and 1982, Randi declared that over 650 people had applied. Between 1997 and February 15, 2005, there had been a total of 360 official, notarized applications. Applications continue to pour in!

JREF Challenge FAQ

They could simply have given the important information less than 20 people were tested in 9 years??? but it doesn't sound very impressive, does it? The general public reading that sounds like it could be over a thousand who have been tested.

Who cares about the number of applications, who is interested in failed applications? What does that prove? That JREF really don't want to test anyone?

James Randi Educational Foundation - Fudging Infomation EXAMPLE 2

(3) To date, how many persons have been tested for the million-dollar prize offered by JREF?

That's not a simple question to answer. Many hundreds have applied, and most have had to be instructed to reapply — sometimes several times — because they did it incorrectly or incompletely. There are, at any given time, about 40 to 60 applicants being considered, but from experience we know that the vast majority will drop out even before any proper preliminary test can be designed. Of those who get to the preliminary stage, perhaps a third will actually be tested, and some of those will quit before completion. To date, no one has actually passed the simple preliminaries and arrived at the formal test stage, though a couple hundred have completed and failed the preliminaries. So, no one has been formally tested for the big prize, though we're ready and willing. [James Randi reply]

JREF Challenge FAQ

Let me explain how to work that out .... take 40 to 60 applicants at any given time, deduct the majority who drop out = ??? Divide this mysterious number by 3 = ??? Then you have the ??? number of applicants who were tested per year? Or per century?

Transparency? No. Magical fudging? Yes.

Has anyone ever passed a preliminary test, it seems JREF can't make their mind up

In a 1981 interview with Randi, the reporter reported ...

'.... Randi's $10,000 lure as been out there for 15 years and some 600 have tried for it. Only 60 passed the preliminary test ...


Newspaper interview in 1981

Tri City Herald - Google News Archive Search

Yet today ... the James Randi Education Foundation 'frequently asked questions' states no one has ever passed a preliminary test ... ..

1.3 How many people have applied for the Challenge?

Between 1964 and 1982, Randi declared that over 650 people had applied. Between 1997 and February 15, 2005, there had been a total of 360 official, notarized applications. Applications continue to pour in!

1.4 Has anyone ever gotten past the preliminary test?

No. Some people use this fact as a reason not to apply – and yet the protocol is never altered once the applicant agrees to it. In fact, we ask the applicant to design the test.

JREF Challenge FAQ

Which version is correct? Or is it all just magician's methodology because in 2005, Randi's challenge manager wrote ...

'.....based upon my preliminary investigation, is that Challenge records were VERY incomplete prior to my arrival here. There wasn't even a real application file before I got here. We have a few VHS tapes of some tests, but it's unclear how many of those were official JREF Challenge tests, as many of these tapes are VERY old and without any paperwork or accompanying data.....' - Kramer 2005

I very much doubt James Randi tested over 600 people (as claimed in the 1981 newspaper) or 650 people between 1964 and 1982 either ... if one searches old newpapers there are very little reports of tests ... did showman James Randi test 600 people in private without the usual media show? Unlikely.

Now incomplete contrast to what Kramer said in that forum topic ABOVE ... compare this to Randi's presentation of it around the 40:30 minute mark in video

'....'We have filing cabinets just full .....[reference to picture of Kramer with cabinets] .... all with little red stickers on them. That means they have been tested - no good.....'
YouTube - Authors@Google: James Randi

Dr Steve Novella also claims to have conducted preliminary trials for the JREF challenge. Where are these details (person under test, protocol, researchers involved) published for inspection? The public presentation seems to be Randi passes out these preliminary trials to open minded researchers. After listening to Novella's internet radio program, one could perhaps doubt that too.

[davidsmith73]

The entire commentary seems to rest on the idea that the effect size for Dean Radin's meta-analysis on staring experiments is too low. The staring weighted mean effect size is about 0.06, which is quite small. However, interpreting this as noise seems to go against how small effect sizes are interpreted in other areas of science. For example, Daryl Bem says this when comparing the effect size of ganzfeld experiments with a meta-analysis on the effects of aspirin on heart attacks:

"Taking aspirin reduces the probability of suffering a heart attack by only 0.008. The corresponding effect size (h) is 0.068..."

http://dbem.ws/Does%20Psi%20Exist%3F.pdf


Now, I don't know how Cohen's h compares to Radin's measure but I would expect them to be pretty similar. In other words, they are both "just noise" according to Novella but one of them resulted in the termination of a large scale randomised controlled trial because the effect was so reliable.

Although I'm not a statistician. Maybe the aspirin and staring effect sizes are not comparable after all.

[davidsmith73]

I have sent Steve Novella an email about the aspirin study effect size and what he thinks the study shows and whether the effect size is comparable to Radin's meta-analysis.

[davidsmith73]

Originally Posted by davidsmith73

I have sent Steve Novella an email about the aspirin study effect size and what he thinks the study shows and whether the effect size is comparable to Radin's meta-analysis.

I've received a reply. Basically, aspirin reduced the number of people having heart attacks from 189 in 10,000 to 105 in 10,000 people. This was a 44% reduction which just considers the proportional reduction in the number of deaths. Bem, Honorton and also Utts coverted these numbers into Cohen's h which is the difference between the arcsine transformed absolute proportions of heart attacks within the population, ie, 2(arcsine √189/10000 - acrsine √105/10000). This results in a very small absolute effect size of 0.068 for the aspirin study.

Steve's reply suggested that the relative effect size is a more meaningful measure for the aspirin study, but he said that such a small absolute measure was reason to be sceptical of the findings and look for methodological flaws etc. He then went on to describe how the findings were later replicated in other studies etc, but that is not really the issue here.

As I'm still unsure of how the Cohen's h estimate reported by Utts compares to effects sizes in psi, I'm going to email her with some more questions.

For example, is the effect size reported in Radin's meta-analysis an absolute or relative effect size? It looks like a relative one. But if so, is it possible to calculate a corresponding absloute effects size from this?

I think this issue is quite important, because Steve Novella's argument seems to rest upon an objection to putting confidence in the detection of a signal from small effect sizes. As someone mentioned in another thread, all the other criteria he mentioned for acceptable evidence seem to have been met in psi research. It's just this small effect size that is being interpreted as "noise". However, we know that something other than chance is responsible for the results in Radin's staring meta-analysis and I presume that Steve also also assumes it's not chance. From listening to the SGU podcast, it seems that the "noise" that Steve is using to explain the results must be methodological error. Indeed, he hints at a possible mechanism of experimenter bias through stopping trials because of a suspicion that something isn't calibrated correctly. So, I think that Steve's position ultimately comes down to - a small effect size is most likely bias in the experimental procedure. Is this fair? Is this how science should operate?

[skidoo]

Originally Posted by Open Mind

Novella very embarrassingly defends his 'research' going to a 'psychic fair' having doctored photographs, approaching 3 fortune tellers and proclaiming his feeling on the outcome of the matter was 'zero' or 'zippo' .

Just to be clear, he never claimed it was any sort of formal research. He says specifically at about 36:35, "Not lab research...we're not lab researchers in parapsychology or psychology." He was obviously making the point that, contrary to Tsakiris' insinuation, many (most?) scientific skeptics are not simply armchair deniers. Most of us are more than willing (eager even) to review any paranormal claims.

But when this willingness is extrapolated to the level of controlled double-blind experiments and whatnot, then you're talking about resources. Of all sorts. And that's really what this comes down to, isn't it? Sheldrake et al and their fanclubs: They're trying to convince the world that their research is still worth it. Dr. Novella explains in this segment that there are specific legitimate reasons mainstream science disregards the current state of paranormal research. He explains that rather than some grand conspiracy of religionists or a monumental example of groupthink, it really just comes down to the question, Is paranormal research worth it?

Credibility problems, serious experiment design problems, small effect sizes.... Lots of factors play into the considered disbursement of resources. Because of course science doesn't care what you choose to research. You can seek to prove that there is sharp cheddar cheese on the moon in sufficient quantity to lend truth to the ancient myth that the moon is, in fact, made of cheese. Knock yourself out. A clever moon cheese researcher could come up with lots of clever tests and look in lots of clever places and occupy lots and lots of time. And other resources.

I'm sure if someone faked a medical condition (e.g. produced artificial damage to skin surface) , gave false information and went along to tape record Dr Steven Novella, he would grossly misdiagnose or get 'zippo' right.

Maybe. But what does that matter? A medical diagnosis is supported by and then affirmed or discounted by scientific procedures. At the very least, a faked sitting demonstrates in no uncertain terms that these vaunted otherworldly powers aren't really so powerful, for whatever that's worth.

Following his own logic, is he implying such experiments would make him unqualified to be in medicine or a fraudulent practitioner?

You're not interpreting the experiment correctly. The whole point is that the psychic claims to have some independent source of information. This sort of thing is just another rail of many in the fence of evidence that keeps so-called "psychics" out there on the criminal fringe.

During the skepiko interview Novella claims 'The goal is to design an experiment in such a manner it doesn't matter what you believe Nobel sounding words indeed, yet in his own trial, we see the very opposite action at work,

His own "trial"? It was a trip to the psychic fair.

he chose money seeking fortune tellers, unaware they were being tested or deceived,

In the very first example, Jay apparently simply presented a picture of him and his wife and asked for a reading on his wife, without identifying her. And the psychic totally blew it. But whatever. It wasn't a scientific test, and it wasn't presented as such. It is what it is. As for being "tested" without their knowledge, how is that relevant?

set the bar high (assuming any scientific measurement was employed at all)

Set the bar high how? What do you mean?

so that what is under test is extremely unlikely to contradict ones own cherished viewpoint.

I think it's disingenuos to present the SGU's visit to the psychic fair as representative of formal research by scientific skeptics. At about 26 minutes into that episode, Jay talks about how the whole point of the skull photo was to do something humorous, not something scientific.

One thing become increasingly clear from listening to his broadcast, Novella feels somewhat hostile towards paranormal claims and dismisses 80 years of controlled lab research ( i.e. not a trip to a psychic fair for an easy debunk) as 'noise' .

He doesn't simply dismiss it. He carefully explains why he equates it with noise.

Para psychological effects have been shown to be above 'noise', would he also dismiss the existence of neutrinos from the sun because the measurement of effects are also weak and erratic?

This is a false analogy. Neutrino detection has been reproduced many many times. Its statistical signal is anything but weak.

Novella statements about what percentages indicate a real effect are extraordinary arbitrary, one could dismiss much of his own medical field's statistical outcomes merely because these are lower than other areas of science such as physics.

This is a tu quoque fallacy. But it's meaningless anyway, because you don't cite specific examples. Are we talking risk assessment, treatment effectiveness, epidemiology, ...? I'm sure improper medical decisions have been made based on insufficient statistical evidence.

If Novella is unwilling to trust meta-analysis in that case particle physics, clinical medicine, psychology are all in serious trouble in his 'guide to the universe'.

This is a slippery slope fallacy. There are many reasons for evaluating meta analyses many different ways, e.g. the importance of absolute versus relative effectiveness. Your assertion here also relies on the unstated major premise that Dr. Novella's interpretation of this information is incorrect. I don't think you've demonstrated that.

Novella's viewpoints merely add 'noise' and confusion upon the work of those parapsychologists doing serious minded research into the subject of psi.

I submit that the psi field is lucky anyone with credibility is paying them any attention at all.

[Ersby]

Cohen's h concerns itself with describing ratios, as far as I can tell. In terms of a ganzfeld experiment, that would mean an experiment with 10 trials and 3 hits would have the same effect size h as an experiment with 100 trials and 30 hits. That's my understanding, at least.

The abstract of the aspirin paper itself uses statistical terms that are more familiar:

Final report on the aspirin component of the ongoing Physicians' Health Study. Steering Committee of the Physicians' Health Study Research Group

Abstract

The Physicians' Health Study is a randomized, double-blind, placebo-controlled trial designed to determine whether low-dose aspirin (325 mg every other day) decreases cardiovascular mortality and whether beta carotene reduces the incidence of cancer. The aspirin component was terminated earlier than scheduled, and the preliminary findings were published. We now present detailed analyses of the cardiovascular component for 22,071 participants, at an average follow-up time of 60.2 months. There was a 44 percent reduction in the risk of myocardial infarction (relative risk, 0.56; 95 percent confidence interval, 0.45 to 0.70; P less than 0.00001) in the aspirin group (254.8 per 100,000 per year as compared with 439.7 in the placebo group). A slightly increased risk of stroke among those taking aspirin was not statistically significant; this trend was observed primarily in the subgroup with hemorrhagic stroke (relative risk, 2.14; 95 percent confidence interval, 0.96 to 4.77; P = 0.06). No reduction in mortality from all cardiovascular causes was associated with aspirin (relative risk, 0.96; 95 percent confidence interval, 0.60 to 1.54). Further analyses showed that the reduction in the risk of myocardial infarction was apparent only among those who were 50 years of age and older. The benefit was present at all levels of cholesterol, but appeared greatest at low levels. The relative risk of ulcer in the aspirin group was 1.22 (169 in the aspirin group as compared with 138 in the placebo group; 95 percent confidence interval, 0.98 to 1.53; P = 0.08), and the relative risk of requiring a blood transfusion was 1.71. This trial of aspirin for the primary prevention of cardiovascular disease demonstrates a conclusive reduction in the risk of myocardial infarction, but the evidence concerning stroke and total cardiovascular deaths remains inconclusive because of the inadequate numbers of physicians with these end points.

[Open Mind]

Originally Posted by skidoo

. I think it's disingenuos to present the SGU's visit to the psychic fair as representative of formal research by scientific skeptics.

Which 'scientific skeptics' would these be? Who are they, what research have they done and where is it published?

This is a false analogy. Neutrino detection has been reproduced many many times. Its statistical signal is anything but weak.

I think the analogy is fine. The 'solar neutrino problem' was a major discrepancy between measurements for 4 decades. Most solar neutrinos have energies that are below the detection thresholds for the heavy water detectors, therefore one could have a contrived a 'noise' argument until more recent years. Better detectors of the last few years have allowed 0.01% of the neutrinos the Sun emits to be measured and these still miss 99.9% of the neutrinos emitted (apparently electron neutrino change into less detectable muon and tau neutrinos)

[davidsmith73]

Originally Posted by Ersby

Cohen's h concerns itself with describing ratios, as far as I can tell. In terms of a ganzfeld experiment, that would mean an experiment with 10 trials and 3 hits would have the same effect size h as an experiment with 100 trials and 30 hits. That's my understanding, at least.

Yes, that's my understanding too. A Cohen's h score in this case would take the proportion of hits relative to chance for observed and expected so the value would be the same in these two situations. I think it would actually be about 0.43 if my calculations are right.

What I can't figure out is what it means to transform the aspirin data into Cohen's h, with regards to making an effect size comparison between psi and the aspirin study.

[skidoo]

Originally Posted by Open Mind

Which 'scientific skeptics' would these be? Who are they, what research have they done and where is it published?

Er, scientific skepticism is a term I would expect anyone serious about these debates to be familiar with. The Wikipedia entry explains it as well as anyone:

Scientific skepticism...is a scientific or practical epistemological position in which one questions the veracity of claims lacking empirical evidence.... Scientific skepticism is different from philosophical skepticism, which questions our right to claim knowledge about the nature of the world and how we perceive it. Scientific skepticism utilizes critical thinking and attempts to oppose claims which lack suitable evidential basis.

I think the analogy is fine. The 'solar neutrino problem' was a major discrepancy between measurements for 4 decades. Most solar neutrinos have energies that are below the detection thresholds for the heavy water detectors, therefore one could have a contrived a 'noise' argument until more recent years.

Huh? How do you "contrive" a noise argument in this context? Scientists had a theoretical model of the sun that predicted more neutrinos impacting the earth than what they were actually seeing. And in fact, after evaluating all of the evidence over time, the scientists actually self-adjusted one of their most fundamental models to account for apparent neutrino mass.

So the subject of your analogy is actually a great example of scientists at their non-dogmatic, evidence-based, peer reviewed best. When there's a compelling argument for pursuing a particular line of reasoning, you can bet some enterprising Nobel-hopeful somewhere will go after it. In the case of the neutrinos, a tiny effect size wasn't the issue. They had plenty of other evidence supporting their model of the sun, but they began to suspect that their fundamental understanding of the neutrino was incorrect after they were just unable to get the sun model's predictions to jive with detected neutrinos (even after totally re-doing the sun model a few times).

Better detectors of the last few years have allowed 0.01% of the neutrinos the Sun emits to be measured and these still miss 99.9% of the neutrinos emitted (apparently electron neutrino change into less detectable muon and tau neutrinos)

This is irrelevant. The physical model -- all of its parts intricately interrelated -- makes predictions. About neutrinos, among other things. Those predictions come true. Repeatedly. And when there is a discrepancy, the model is updated if necessary.

Look, you're ultimately trying to argue that even if a line of inquiry presents only the tiniest sliver of a shred of hope for an explanation of this nonsense that is life, it's worth pursuing. I can dig that motivation. But I think that that hope might be blinding you to the reality of the situation.

There really is nothing there. No one is conspiring to squelch a bunch of positive psi experiment results. The fruits of psi research through history simply are what they are: Non-existent.