Why Small, Self-Selected Studies Are A Bad Way To Make Health Care Recommendations

[HuckleB]

Are Most Medical Studies Wrong?
http://theness.com/neurologicablog/index.php/are-most-medical-studies-wrong/

"...

The implications of Ioannidis’ research, therefore, is not to undermine or abandon scientific medicine, but rather to demonstrate the importance of re-introducing prior probability in our evaluation of the medical literature and in deciding what to research. As much as I am in favor of the Evidence-Based Medicine (EBM) movement, it does not consider prior probability. I have said before that this is a grave mistake, and the work of Ioannidis provides statistical support for this. One of the best ways to minimize false positives is to carefully consider the plausibility of the intervention being studied. CAM proponents are deathly afraid of such consideration for they live in the world of infinitesimal probability.

...

With regard to scientific medicine, I would emphasize what Tabbarok said about evaluating the literature, not a single study. Ioannidis compared single studies to the later literature – using the literature as its own gold standard. This does not mean that medical research is wrong. It just means that a research question has to mature, that multiple studies by independent researchers are required before we arrive at a reliable conclusion.

The implications for the practicing physician are clear – don’t overreact to every study, do not practice “knee-jerk” medicine. Take a cool and skeptical eye at published research, and follow the rules of thumb above. When this is done it is possible to practice science-based medicine that is very reliable.

The implications for society are also clear – a rational health care system must be based upon sound scientific reasoning as well as the best evidence available (what I call science-based medicine, to distinguish it from the laudable but inadequate evidence-based medicine). Also – and this applies to individuals as well as the science media – any single study must be put into the context of the broader literature. Great mischief has been inflicted upon the public by the media touting the resuls of a single study (almost always described as a “breakthrough”) without providing any scientific context."

----------------------------------------------

A bit more on the topic:

We're so good at medical studies that most of them are wrong
http://arstechnica.com/science/news/2010/03/were-so-good-at-medical-studies-that-most-of-them-are-wrong.ars

Why Most Published Research Findings are False
http://marginalrevolution.com/marginalrevolution/2005/09/why_most_publis.html


----------------------------------------------


It's odd to note that this has been used to attack science and medicine, in general. At the same time, it has been ignored by many who use it to attack science, while they find small, select studies to justify the quack treatments they are selling. The repeated use of double-standards, along with ridiculous hyperbole ("health freedom!" = "regulate big pharma but not big supplement!") ought to cause any skeptical consumer to question these pushers every step of the way. (It's not freedom they're selling, btw.)