Reply #40: Oh come on, we've done better than that

[Euromutt]

As you note, we've taken Kellermann's work apart repeatedly, so that's a little more than saying "they twist the numbers" without providing supporting evidence. And admittedly, Kellermann's studies form only a handful of the studies out there (though it should be noted that a relatively small number of researchers crank out these studies), but his work is emblematic of the whole field, in that it contains all the flaws, at least some of which are found in every study I've ever seen in the public health literature on this topic.

The big one, the one that every public health study on this topic suffers from, is failure to establish a causal relationship and the nature of that relationship. It's not an unreasonable assumption that there's a casual link between a gun being kept in the household and a household member getting shot, but it's bad science to treat that assumption as fact, given that a fundamental rule of good science is that "correlation does not imply causation." But even leaving that aside, you simply can't draw any useful conclusions without researching how the causal relationship operates. Is there a higher likelihood of a household member being shot because of the presence of a firearm in the household? Or is the presence of the firearm due to the fact that one or more household members consider themselves to be at elevated risk of becoming a victim of criminal violence? Or is the correlation created by a third, as yet unidentified factor? These are possibilities that have to be investigated to draw any useful conclusions.

This, the public health literature almost invariably fails to do. Study after study establishes no more than correlation, and then states--or at least strongly implies--in its conclusion that increased likelihood of a family member being shot is caused by the presence of the firearm, seemingly oblivious to the fact this is a blatant post hoc ergo propter hoc fallacy. In the study "Investigating the Link Between Gun Possession and Gun Assault" (Branas et al., AJPH http://ajph.aphapublications.org/cgi/content/abstract/AJPH.2008.143099v1) that came out two months ago, the authors acknowledged that they "did not account for the potential of reverse causation between gun possession and gun assault," and yet the conclusion, abstract and press release were written as if they had, which is just sheer intellectual dishonesty.

The Dahlberg study from the AJE that you cite falls into the same pattern. The abstract starts:

Data from a US mortality follow-back survey were analyzed to determine whether having a firearm in the home increases the risk of a violent death in the home <...>

So the authors are seeking to establish a causal relationship: is the risk of violent death increased by having a firearm in the household? However, what they conclude with is:

Results show that <...> having a gun in the home was associated with an increased risk of firearm homicide and firearm suicide in the home.

Note the phrase "associated with"; i.e. the authors established a correlation. That's all "associated with" means (having insulin in the house is "associated with" having diabetes). But correlation, I reiterate, does not imply causation. Because the authors fail to acknowledge this difference between their stated objectives and their actual findings, they are in effect pulling a bait-and-switch. It also reduces the findings to Tooth Fairy Science.

You could measure how much money the Tooth Fairy leaves under the pillow, whether she leaves more cash for the first or last tooth, whether the payoff is greater if you leave the tooth in a plastic baggie versus wrapped in Kleenex. You can get all kinds of good data that is reproducible and statistically significant. Yes, you have learned something. But you haven’t learned what you think you’ve learned, because you haven’t bothered to establish whether the Tooth Fairy really exists.

http://www.sciencebasedmedicine.org/?p=44

This figures, because the fundamentally flawed premise that the epidemiological approach to firearms is based on is that firearms are a pathogen, that they affect everyone in more or less the same way. For example, you run the same health risks from smoking if you're an office worker as you do if you're a member of a drug gang. But what you're going to do with a firearm, or what someone else is likely to (try to) do to you with a firearm, is very different. The epidemiological approach to firearms consistently fails to take this into account.

This is reflected in the methodology that every one of these studies employs, which is to take as a study group a bunch of people who've been shot, and then compare them to a control group of people who weren't. Right there, they make the selection based on a variable that is not the hypothetical causal variable (namely the presence of a firearm in the household), but instead, the hypothetical outcome variable (someone in the household getting shot). This is exactly the wrong way to approach the question, because instead of isolating the variable the causal effect of which you're (supposedly) trying to study, you're comparing a study group which, by definition, has a higher chance of having been shot (100%, because that's what you selected them for) than your control group (0%, because that's what you selected them for), allowing any number of possible confounding factors to creep in, and then trying to identify and control for them after you've allowed them to creep in. But unless you've thought of every possible variable, there are going to be errors.

The findings of each one of these studies can be at least as plausibly be argued to indicate that engaging in high-risk and/or criminal behaviors increases the risk of a household member dying a violent death, and also increases the likelihood that the subject will keep a firearm in the home or on his person; leaving open the possibility that a person who keeps a firearm, but does not engage in high risk and/or criminal behavior, does not suffer a similar risk.