A Study Showed....

[really]

Something to consider.


A Study Showed....
by Dr. Steven Novella

Scientific skepticism is, perhaps above all else, a process for looking at claims and evidence. Skeptics value that process above any particular conclusion that it may come to. Conclusions are tentative and need to be updated and almost perpetually refined. It is therefore folly to invest one's ego in any particular static conclusion, although that is the common "default mode" of human behavior. By investing in the process, however, we are free to alter our tentative conclusions as new evidence or arguments come in.

Saying that we value the process, however, is only the beginning. Understanding how to evaluate complex bodies of evidence is a lifelong endeavor in and of itself. That is why I don't understand those who criticize skeptical writing and lectures as "preaching to the choir," as if once someone self-identifies as a skeptic the battle is over. Rather, we need to continuously educate ourselves and each other about the findings of science as well as the many complex ways in which to think about claims and evidence.

With that in mind, I would like to address a habit of argument that is unfortunately common, even among skeptics - referencing a single study as support for a position, or as if the conclusion of the study can then be taken as an established premise.

Single studies are, of course, important to the process of science. They are the units of which the scientific literature - our body of scientific knowledge - is comprised. It is therefore very important to understand how to dissect a scientific study, or at least to recognize the major potential weaknesses of studies and have some idea if a study is reliable or nonsense. We can also rely partly on experts in each specialized field to examine and critique studies for us, realizing that a definitive analysis may depend upon rarefied technical knowledge.

Understanding that individual studies fall on a spectrum from rigorous to utter crap is the first step. Many people appear to not understand this, and act as if "a study shows" X then we can conclude that X is true. The media generally acts this way when reporting science news, greatly reinforcing this fallacy.

Just as important, however, is the recognition that no single study can reliably confirm any phenomenon. The more complex the phenomenon, then the more this is true. There are many reasons for this.

The outcome of single studies may be the result of experimenter bias, which can vary from subtle to borderline fraud. Or the results may be the result of a systematic and unrecognized error in the observations or experimental methods. The results may also be quirky - just due to chance.

But even when the results of a study are accurate, the interpretation of the study may be complex. The study, in essence, represents a single line of evidence. In order to reach a reliable conclusion about how the world operates it is better to have multiple independent lines of evidence converging on one conclusion. That, of course, requires multiple studies.Studies gain power when they are independently replicated. This helps average out or eliminate the effects of random chance or individual bias. Exact replications endeavor to copy the original study exactly. This is important because then it helps to eliminate certain effects of experimenter bias. Researchers may subconsciously mine data or cherry pick in order to come up with a series of data that appears positive (exploiting so-called researcher degrees of freedom) An exact replication with a fresh data set will eliminate the effects of data mining or cherry picking.

There are also replications that are not exact but look at the same question with some differences in the methods. These types of replications are important also, because then we can see which variables have an effect on the outcome.

Finally, there are studies which are not replications but which address the same or a closely related phenomenon. There are often multiple different ways to ask the same question or to look at data, each with it's own strengths and weaknesses. If every different kind of study all appear to reflect the same phenomenon, then we begin grow confident that the phenomenon is real and works the way we think it does.

Let's take, for example, the question of whether or not cell phones increase the risk for certain types of brain cancer. We can address this question in many ways. We can take groups of people who use and do not use cell phones and then follow them over time to see how many of each group develop brain cancer. We can look at people who have and do not have brain cancer and then ask them about their past cell phone use. We can see if the typical side of use (left or right) correlates
Continued: A Study Showed....

[Mystical Atheist Sniffy]

Very interesting, thank you for sharing really