My Critique of Nate Silver’s Critique of Exit Polls
Last edited Wed May 18, 2016, 12:31 PM - Edit history (1)
I very well recall the stolen Presidential election of 2004 because that is the event that incited me to join DU, just as the stolen Presidential election of 2000 was, I believe, the event that incited the birth of DU (which I believe was born on inauguration day of January, 2001, but I might be off by a little bit on that). I joined DU shortly after the stolen 2004 election, primarily to participate with DUers in the investigation of the election. Some of my DU posts caught the attention of a small group that was organized to lobby the U.S. Senate to officially object to the election results, and I was invited to join the group, which I did, my role being to present my statistical findings, primarily those regarding the exit polls.
It should be noted that discussion of the 2004 stolen election was rampant on DU after the election and for many months and even years afterwards. Concurrently, election integrity organizations sprung up all over the country. The large discrepancy between the exit polls and the official vote counts, nationally and in most states, was a central part of the discussion on DU and focus of the election integrity organizations. According to the exit polls, John Kerry won both the national popular vote and the Electoral College (the deciding state being Ohio), though George W. Bush won the official vote counts of both. The discrepancies, taken as a whole, were way outside the margin of error. The vast majority of DUers who participated in the discussions about this (as well as many outside of DU) agreed that the exit poll discrepancies were a very strong indication of election fraud. There were some dissenting voices on DU, which many or most DUers involved in the discussions considered to be trolls. I did not and do not agree with that characterization. I’m pretty sure that at least a handful of the dissenters were knowledgeable and honest in their dismissal of the exit polls, though wrong. I find it very interesting that today, where we see even greater exit poll discrepancies than we saw in 2004, opinion of the value of exit polls on DU is much more evenly divided. Clearly the fact that a Democratic candidate, rather than George W. Bush, is the one whose official vote count consistently outperforms the exit poll predictions is the reason for the difference on DU regarding the value of exit polls, from what it was following the 2004 Presidential election.
Since the Clinton supporters on DU, who universally (as far as I can tell) dismiss the value of exit polls, point to various statements on exit polls made by our corporate news media, and especially those of Nate Silver, as proof that they are useless, I think that it is important to look in detail at Silver’s critique of exit polls and see how much sense they make.
I am a recently retired public health epidemiologist who worked for more than 40 years as such. My work primarily involved scientific studies of public health issues, all which involved detailed statistical analysis. 39 of those studies were published in peer reviewed medical or public health journals, most for which I was the first author. I also worked for one of the newly formed election integrity organizations, the Election Defense Alliance (EDA), as their data coordinator. The EDA conducts its own exit polls and has published several articles on the subject, a couple which I co-authored. I mention all this to make the point that I am qualified to critique Silver’s statements on exit polls. I certainly am not saying that I have as much knowledge or experience in polling as he does. But statistics is statistics, no matter what field it is applied to.
And I do have one very important qualification that Silver doesn’t have: the willingness to consider the possibility that exit poll discrepancies from official vote counts might indicate something different and much more serious than a problem with the exit polls, especially when they consistently favor one candidate. Scientific studies in public health all have various problems with them. But we don’t just say that because the study is imperfect in various ways that its findings are useless. Instead, we consider all the various alternative explanations for our findings, analyze them, decide what the most likely explanations are and why, discuss them in our manuscript, and submit it for publication.
So let’s take a close look at Silver’s "Ten Reasons Why you Should Ignore Exit Polls”.
Nate Silver’s “Ten Reasons Why you Should Ignore Exit Polls”
Silver ends his discussion on exit polls by saying that an independent panel created by CNN following the disastrous Florida election of 2000 recommended that they ignore exit polls for assisting them in calling elections, and he recommends that we do the same. What he is referring to when he calls the Florida election a “disaster” is not the fact that it was stolen but the fact that the TV networks called Florida first for Gore, then for Bush, then too close to call, quite an embarrassment for the networks. What he doesn’t tell us is that the first wrong call for Gore was the result of exit polls that were off because of the “butterfly ballot” in heavily Democratic Palm Beach County, which was very confusing and caused many thousands of voters to think that they were voting for Al Gore when they actually didn’t (many of those even wrote Gore’s name in, but their vote still was never counted). He also doesn’t tell us that the second bad call, for Bush, was the result of an electronic machine “malfunction” in Volusia County that suddenly subtracted about 15,000 votes from Gore’s total in a single precinct with less than a thousand voters. And he also doesn’t tell us that exit polls are still used to call elections, despite what one panel recommended 15 years ago. You may recall that Maryland was called for Hillary Clinton in this year’s primary with 0.0% of the vote in. What do you think they used to call that election?
So let’s take a look at Silver’s reasons why we should ignore exit polls:
1. Exit polls have a much larger intrinsic margin of error than pre-election polls
Silver correctly notes that the margin of error in exit polls is generally larger than in pre-election polls because of the cluster sampling that is used related to the precincts that are sampled in exit polls.
So what’s the big deal? Richard Charnin’s publication of exit poll discrepancies made appropriate adjustment of the margins of error based on cluster sampling, and still there have been 10 states (I don’t know about Oregon or Kentucky yet) with exit poll discrepancies from the official vote count outside the margin of error, and they all favor Hillary Clinton in the official vote count compared to the exit polls. Adjustment made, problem solved. Anyhow, margin of error is a statistical concept that depends totally on random error. Clinton outperformed Sanders relative to exit poll predictions in the good majority of states that weren’t quite outside of the margin of error too. A large margin of error does not cause findings to consistently point in the same direction. That’s why it’s called random error.
2. Exit polls have consistently favored the Democratic share of the vote
Yeah, since the stolen election of 2004 and the widespread use of easily manipulated electronic voting machines, indeed they have. It’s interesting that Silver uses the 2004 Presidential election as his one and only example. He doesn’t even consider the possibility that electronic vote manipulation was the cause of the exit poll discrepancies in that election – he just merely assumes that it was due to exit poll problems rather than electronic voting problems. That is circular reasoning.
3. Exit poll discrepancies were particularly bad in this year’s primaries
He was referring there to the primaries of 2008 and the fact that Barack Obama consistently under-performed related to the exit poll predictions. Again, he doesn’t even consider the possibility that the exit poll discrepancies may have been related to electronic machine manipulation rather than exit poll problems. And guess who Obama was running against in those primaries?
4. Exit polls challenge the definition of random sample
He says that it is hard to get an accurate random sample because some polling places are very busy and the pollster may be standing “yards away” from the voters. Yeah, ok, fine. The sample may not be perfectly random. Compare that to pre-exit polls, where some demographics of voters can’t even be reached by telephone or in any manner. How many “yards away” are these potential voters (not actual voters, as in exit polls) from the pollster?
5. Democrats may be more likely to participate in exit polls
This is almost a total reiteration of number 2. But I guess he needed it to reach 10. He also provides a study to support the point. And guess who conducted that study that Silver referred to. Scott Rasmussen, the right wing pollster who consistently over-estimates the Republican share of the vote even after whatever election fraud there is helps them out.
6. Exit polls may have trouble calibrating results from early voting
Silver acknowledges that most, but not all exit polls attempt to include early voting in there analysis. And he points out that doing so can be problematic because it requires an estimation of the number of early voters relative to Election Day voters, which may be wrong. But there are two things be leaves out of this explanation: One, that the states that don’t include early voting are the ones where the amount of early voting is considered too small to make a difference, and; two, that any trouble with exit polls that apply to difficulties in estimating the relative number of early voters applies in exactly the same manner to pre-election polls.
7. Exit polls may also miss late voters
That’s kind of a weak statement. He qualifies his statement with “may”, meaning if the exit pollsters go home early, before the polls close. To the extent that that happens it may or may not bias the sample, depending upon the demographics or candidate preference makeup of the late voters. But Silver says nothing about how common it would be for the pollsters to leave significantly prior to poll closing. Exit polling is a science, based on the necessity of polling a random sample. Anyhow, I have not seen radical swings in exit polls towards the end of the day, and if there were any, they should reflect similar radical swings in the official vote count, which could be assessed.
8. Leaked exit polls may not be the genuine article
What Silver means by this is that sometimes some people will leak interim results of exit polls, rather than the final exit polls (just prior to “adjustment” to fit the official vote count). But the exit polls that Richard Charnin and others have published this year, which I and others have cited frequently, are final exit polls. They are the “genuine article”, as Silver puts it. So for the purposes of our discussions of the huge exit poll discrepancies that favor Clinton in the official vote count, this is a non-problem.
9. A high turnout election may make demographic waiting difficult
In his discussion of this problem Silver for the first time in his article acknowledges that pre-election polls may have the same problem. But what he doesn’t say is that the problem of demographic weighting should be much greater in pre-election polls compared to exit polls. The reason for that should be obvious. In pre-election polls, models are needed to predict which demographic groups are more likely to vote. In exit polls such models are hardly needed at all to predict the final vote count because the polling is done on voters who actually showed up to vote.
Perhaps he brings this issue up because (though he doesn’t say so) some say that younger voters, for example, may be more likely to respond to an exit poll, and if so, that would skew (bias) the exit poll (in this case towards Bernie Sanders). But if that is the case, then why did Alabama, Georgia, and South Carolina, all with a relatively very old population of voters, exhibit some of the greatest exit poll discrepancies in the Democratic primaries this year? The theory just doesn’t hold up to the data, at least not in this year’s Democratic primaries.
10. You’ll know the actual results soon enough anyhow
What kind of reason is that for ignoring exit polls? By saying this, Silver willfully ignores the main reason why almost all, if not all election integrity activists, and many other Americans believe that exit polls need to be done: to monitor our elections for signs of fraud, as is done in many other countries; to look for warning flags and follow them up with investigations, primarily with extensive hand counted audits to see whether the vote counts from our woefully insecure and easily manipulated electronic machines match the hand counts, and; where they don’t, to use hand counts of the whole state (publicly monitored) to identify the actual results of the election. Does Silver really believe that people interested in improving our democracy see the main value of exit polls as learning the “actual” results of the election a little bit earlier?
What Silver doesn’t tell us in his criticism of exit polls
There are four very important issues that Silver omits from his discussion of exit polls, some which I’ve already alluded to:
One is the circular nature of his reasoning, which shows itself in his 2nd, 3rd, 5th, and 10th reasons for ignoring exit polls. Essentially, what he is saying is that we should ignore exit polls because they have been “wrong” before. What he means by “wrong” is that they differ from the official vote count. But for the purpose of a discussion which considers whether exit polls are of value in monitoring the accuracy of the official vote count, differing from the official vote count in no way proves that the exit polls are wrong. They only prove that either the exit polls are wrong or the official vote count is wrong. Only extensive audits by hand counting paper ballots will tell you which is the case, and that is never done in this country, except when the official count is extremely close.
Two, in recent history, at least since 2004, whenever there are very large exit poll discrepancies, it is always the more conservative candidate who comes out ahead in the official count compared to the exit polls. That happened in the 2004 Presidential election (as well as 2008 and 2012, but in those years not enough to make a difference in the final count), in numerous Congressional elections between 2004 and 2014, and it happening again now, in the Democratic primaries. Isn’t there a pattern here that is worth commenting on? Related to that is the fact that the election machines used to count our votes are owned, programmed and run by right wing corporations.
Three, he doesn’t acknowledge that some of his criticisms of exit polls apply equally to pre-election polls (items 4 and 6) or worse (item 9)
Fourth, he doesn’t balance his discussion by noting some very substantial advantages that exit polls have over pre-election polls – far more substantial than the reasons he gives for ignoring them, which I hope I’ve made clear. I find that very ironic because he makes his living (I assume) from doing and analyzing pre-election polls, which he must believe have some value, otherwise why would he spend so much time doing them. The very substantial advantages that exit polls have over pre-election polls are: 1) They assess how the voter actually voted rather than how s/he intends to vote at a later time; 2) Pre-election polls rely on models to estimate who likely voters will be. Different pollsters use different models to estimate that, and obviously some of them are wrong. Sometimes most of them are wrong. Exit polls have no need for such models. Each respondent to an exit poll has approximately a 100% chance of voting because s/he has already voted, and; 3) despite the problems that exit polls have in obtaining a true random sample of voters, the problems with obtaining a true random sample with pre-election polls are far worse for the very simple reason that many voters cannot be sampled at all because they are unobtainable by phone.
Why doesn’t Silver even mention these substantial advantages of exit polls over pre-election polls? Could it be that it’s because he makes his living by doing pre-election polls?
= new reply since forum marked as read
