HomeLatest ThreadsGreatest ThreadsForums & GroupsMy SubscriptionsMy Posts
DU Home » Latest Threads » Forums & Groups » Retired » Retired Forums » 2016 Postmortem (Forum) » My Critique of Nate Silve...

Wed May 18, 2016, 11:18 AM

My Critique of Nate Silver’s Critique of Exit Polls

Last edited Wed May 18, 2016, 12:31 PM - Edit history (1)

I very well recall the stolen Presidential election of 2004 because that is the event that incited me to join DU, just as the stolen Presidential election of 2000 was, I believe, the event that incited the birth of DU (which I believe was born on inauguration day of January, 2001, but I might be off by a little bit on that). I joined DU shortly after the stolen 2004 election, primarily to participate with DUers in the investigation of the election. Some of my DU posts caught the attention of a small group that was organized to lobby the U.S. Senate to officially object to the election results, and I was invited to join the group, which I did, my role being to present my statistical findings, primarily those regarding the exit polls.

It should be noted that discussion of the 2004 stolen election was rampant on DU after the election and for many months and even years afterwards. Concurrently, election integrity organizations sprung up all over the country. The large discrepancy between the exit polls and the official vote counts, nationally and in most states, was a central part of the discussion on DU and focus of the election integrity organizations. According to the exit polls, John Kerry won both the national popular vote and the Electoral College (the deciding state being Ohio), though George W. Bush won the official vote counts of both. The discrepancies, taken as a whole, were way outside the margin of error. The vast majority of DUers who participated in the discussions about this (as well as many outside of DU) agreed that the exit poll discrepancies were a very strong indication of election fraud. There were some dissenting voices on DU, which many or most DUers involved in the discussions considered to be trolls. I did not and do not agree with that characterization. I’m pretty sure that at least a handful of the dissenters were knowledgeable and honest in their dismissal of the exit polls, though wrong. I find it very interesting that today, where we see even greater exit poll discrepancies than we saw in 2004, opinion of the value of exit polls on DU is much more evenly divided. Clearly the fact that a Democratic candidate, rather than George W. Bush, is the one whose official vote count consistently outperforms the exit poll predictions is the reason for the difference on DU regarding the value of exit polls, from what it was following the 2004 Presidential election.

Since the Clinton supporters on DU, who universally (as far as I can tell) dismiss the value of exit polls, point to various statements on exit polls made by our corporate news media, and especially those of Nate Silver, as proof that they are useless, I think that it is important to look in detail at Silver’s critique of exit polls and see how much sense they make.

I am a recently retired public health epidemiologist who worked for more than 40 years as such. My work primarily involved scientific studies of public health issues, all which involved detailed statistical analysis. 39 of those studies were published in peer reviewed medical or public health journals, most for which I was the first author. I also worked for one of the newly formed election integrity organizations, the Election Defense Alliance (EDA), as their data coordinator. The EDA conducts its own exit polls and has published several articles on the subject, a couple which I co-authored. I mention all this to make the point that I am qualified to critique Silver’s statements on exit polls. I certainly am not saying that I have as much knowledge or experience in polling as he does. But statistics is statistics, no matter what field it is applied to.

And I do have one very important qualification that Silver doesn’t have: the willingness to consider the possibility that exit poll discrepancies from official vote counts might indicate something different and much more serious than a problem with the exit polls, especially when they consistently favor one candidate. Scientific studies in public health all have various problems with them. But we don’t just say that because the study is imperfect in various ways that its findings are useless. Instead, we consider all the various alternative explanations for our findings, analyze them, decide what the most likely explanations are and why, discuss them in our manuscript, and submit it for publication.

So let’s take a close look at Silver’s "Ten Reasons Why you Should Ignore Exit Polls”.

Nate Silver’s “Ten Reasons Why you Should Ignore Exit Polls”

Silver ends his discussion on exit polls by saying that an independent panel created by CNN following the disastrous Florida election of 2000 recommended that they ignore exit polls for assisting them in calling elections, and he recommends that we do the same. What he is referring to when he calls the Florida election a “disaster” is not the fact that it was stolen but the fact that the TV networks called Florida first for Gore, then for Bush, then too close to call, quite an embarrassment for the networks. What he doesn’t tell us is that the first wrong call for Gore was the result of exit polls that were off because of the “butterfly ballot” in heavily Democratic Palm Beach County, which was very confusing and caused many thousands of voters to think that they were voting for Al Gore when they actually didn’t (many of those even wrote Gore’s name in, but their vote still was never counted). He also doesn’t tell us that the second bad call, for Bush, was the result of an electronic machine “malfunction” in Volusia County that suddenly subtracted about 15,000 votes from Gore’s total in a single precinct with less than a thousand voters. And he also doesn’t tell us that exit polls are still used to call elections, despite what one panel recommended 15 years ago. You may recall that Maryland was called for Hillary Clinton in this year’s primary with 0.0% of the vote in. What do you think they used to call that election?

So let’s take a look at Silver’s reasons why we should ignore exit polls:

1. Exit polls have a much larger intrinsic margin of error than pre-election polls
Silver correctly notes that the margin of error in exit polls is generally larger than in pre-election polls because of the cluster sampling that is used related to the precincts that are sampled in exit polls.

So what’s the big deal? Richard Charnin’s publication of exit poll discrepancies made appropriate adjustment of the margins of error based on cluster sampling, and still there have been 10 states (I don’t know about Oregon or Kentucky yet) with exit poll discrepancies from the official vote count outside the margin of error, and they all favor Hillary Clinton in the official vote count compared to the exit polls. Adjustment made, problem solved. Anyhow, margin of error is a statistical concept that depends totally on random error. Clinton outperformed Sanders relative to exit poll predictions in the good majority of states that weren’t quite outside of the margin of error too. A large margin of error does not cause findings to consistently point in the same direction. That’s why it’s called random error.

2. Exit polls have consistently favored the Democratic share of the vote
Yeah, since the stolen election of 2004 and the widespread use of easily manipulated electronic voting machines, indeed they have. It’s interesting that Silver uses the 2004 Presidential election as his one and only example. He doesn’t even consider the possibility that electronic vote manipulation was the cause of the exit poll discrepancies in that election – he just merely assumes that it was due to exit poll problems rather than electronic voting problems. That is circular reasoning.

3. Exit poll discrepancies were particularly bad in this year’s primaries
He was referring there to the primaries of 2008 and the fact that Barack Obama consistently under-performed related to the exit poll predictions. Again, he doesn’t even consider the possibility that the exit poll discrepancies may have been related to electronic machine manipulation rather than exit poll problems. And guess who Obama was running against in those primaries?

4. Exit polls challenge the definition of random sample
He says that it is hard to get an accurate random sample because some polling places are very busy and the pollster may be standing “yards away” from the voters. Yeah, ok, fine. The sample may not be perfectly random. Compare that to pre-exit polls, where some demographics of voters can’t even be reached by telephone or in any manner. How many “yards away” are these potential voters (not actual voters, as in exit polls) from the pollster?

5. Democrats may be more likely to participate in exit polls
This is almost a total reiteration of number 2. But I guess he needed it to reach 10. He also provides a study to support the point. And guess who conducted that study that Silver referred to. Scott Rasmussen, the right wing pollster who consistently over-estimates the Republican share of the vote even after whatever election fraud there is helps them out.

6. Exit polls may have trouble calibrating results from early voting
Silver acknowledges that most, but not all exit polls attempt to include early voting in there analysis. And he points out that doing so can be problematic because it requires an estimation of the number of early voters relative to Election Day voters, which may be wrong. But there are two things be leaves out of this explanation: One, that the states that don’t include early voting are the ones where the amount of early voting is considered too small to make a difference, and; two, that any trouble with exit polls that apply to difficulties in estimating the relative number of early voters applies in exactly the same manner to pre-election polls.

7. Exit polls may also miss late voters
That’s kind of a weak statement. He qualifies his statement with “may”, meaning if the exit pollsters go home early, before the polls close. To the extent that that happens it may or may not bias the sample, depending upon the demographics or candidate preference makeup of the late voters. But Silver says nothing about how common it would be for the pollsters to leave significantly prior to poll closing. Exit polling is a science, based on the necessity of polling a random sample. Anyhow, I have not seen radical swings in exit polls towards the end of the day, and if there were any, they should reflect similar radical swings in the official vote count, which could be assessed.

8. Leaked exit polls may not be the genuine article
What Silver means by this is that sometimes some people will leak interim results of exit polls, rather than the final exit polls (just prior to “adjustment” to fit the official vote count). But the exit polls that Richard Charnin and others have published this year, which I and others have cited frequently, are final exit polls. They are the “genuine article”, as Silver puts it. So for the purposes of our discussions of the huge exit poll discrepancies that favor Clinton in the official vote count, this is a non-problem.

9. A high turnout election may make demographic waiting difficult
In his discussion of this problem Silver for the first time in his article acknowledges that pre-election polls may have the same problem. But what he doesn’t say is that the problem of demographic weighting should be much greater in pre-election polls compared to exit polls. The reason for that should be obvious. In pre-election polls, models are needed to predict which demographic groups are more likely to vote. In exit polls such models are hardly needed at all to predict the final vote count because the polling is done on voters who actually showed up to vote.

Perhaps he brings this issue up because (though he doesn’t say so) some say that younger voters, for example, may be more likely to respond to an exit poll, and if so, that would skew (bias) the exit poll (in this case towards Bernie Sanders). But if that is the case, then why did Alabama, Georgia, and South Carolina, all with a relatively very old population of voters, exhibit some of the greatest exit poll discrepancies in the Democratic primaries this year? The theory just doesn’t hold up to the data, at least not in this year’s Democratic primaries.

10. You’ll know the actual results soon enough anyhow
What kind of reason is that for ignoring exit polls? By saying this, Silver willfully ignores the main reason why almost all, if not all election integrity activists, and many other Americans believe that exit polls need to be done: to monitor our elections for signs of fraud, as is done in many other countries; to look for warning flags and follow them up with investigations, primarily with extensive hand counted audits to see whether the vote counts from our woefully insecure and easily manipulated electronic machines match the hand counts, and; where they don’t, to use hand counts of the whole state (publicly monitored) to identify the actual results of the election. Does Silver really believe that people interested in improving our democracy see the main value of exit polls as learning the “actual” results of the election a little bit earlier?

What Silver doesn’t tell us in his criticism of exit polls

There are four very important issues that Silver omits from his discussion of exit polls, some which I’ve already alluded to:

One is the circular nature of his reasoning, which shows itself in his 2nd, 3rd, 5th, and 10th reasons for ignoring exit polls. Essentially, what he is saying is that we should ignore exit polls because they have been “wrong” before. What he means by “wrong” is that they differ from the official vote count. But for the purpose of a discussion which considers whether exit polls are of value in monitoring the accuracy of the official vote count, differing from the official vote count in no way proves that the exit polls are wrong. They only prove that either the exit polls are wrong or the official vote count is wrong. Only extensive audits by hand counting paper ballots will tell you which is the case, and that is never done in this country, except when the official count is extremely close.

Two, in recent history, at least since 2004, whenever there are very large exit poll discrepancies, it is always the more conservative candidate who comes out ahead in the official count compared to the exit polls. That happened in the 2004 Presidential election (as well as 2008 and 2012, but in those years not enough to make a difference in the final count), in numerous Congressional elections between 2004 and 2014, and it happening again now, in the Democratic primaries. Isn’t there a pattern here that is worth commenting on? Related to that is the fact that the election machines used to count our votes are owned, programmed and run by right wing corporations.

Three, he doesn’t acknowledge that some of his criticisms of exit polls apply equally to pre-election polls (items 4 and 6) or worse (item 9)

Fourth, he doesn’t balance his discussion by noting some very substantial advantages that exit polls have over pre-election polls – far more substantial than the reasons he gives for ignoring them, which I hope I’ve made clear. I find that very ironic because he makes his living (I assume) from doing and analyzing pre-election polls, which he must believe have some value, otherwise why would he spend so much time doing them. The very substantial advantages that exit polls have over pre-election polls are: 1) They assess how the voter actually voted rather than how s/he intends to vote at a later time; 2) Pre-election polls rely on models to estimate who likely voters will be. Different pollsters use different models to estimate that, and obviously some of them are wrong. Sometimes most of them are wrong. Exit polls have no need for such models. Each respondent to an exit poll has approximately a 100% chance of voting because s/he has already voted, and; 3) despite the problems that exit polls have in obtaining a true random sample of voters, the problems with obtaining a true random sample with pre-election polls are far worse for the very simple reason that many voters cannot be sampled at all because they are unobtainable by phone.

Why doesn’t Silver even mention these substantial advantages of exit polls over pre-election polls? Could it be that it’s because he makes his living by doing pre-election polls?

19 replies, 3941 views

Reply to this thread

Back to top Alert abuse

Always highlight: 10 newest replies | Replies posted after I mark a forum
Replies to this discussion thread
Arrow 19 replies Author Time Post
Reply My Critique of Nate Silver’s Critique of Exit Polls (Original post)
Time for change May 2016 OP
JackRiddler May 2016 #1
Fearful May 2016 #2
reformist2 May 2016 #14
Time for change May 2016 #15
YouDig May 2016 #3
Time for change May 2016 #4
YouDig May 2016 #5
Time for change May 2016 #7
reddread May 2016 #13
2cannan May 2016 #6
Time for change May 2016 #8
zipplewrath May 2016 #17
Time for change May 2016 #19
Godhumor May 2016 #9
Time for change May 2016 #10
Godhumor May 2016 #11
Time for change May 2016 #12
Godhumor May 2016 #16
Time for change May 2016 #18

Response to Time for change (Original post)

Wed May 18, 2016, 12:13 PM

1. bump


Reply to this post

Back to top Alert abuse Link here Permalink

Response to Time for change (Original post)

Wed May 18, 2016, 12:50 PM

2. This is a very important post

Exit polls consistently show Clinton outperforming her exit poll predictions, usually by large amounts.

This should be seen as a strong indication of election fraud and begs for discussion and investigation, including hand counted audits. Yet we let our national news media define exit poll abnormalities as unreliable. Nate Silver is the most frequently quoted person in this regard.

This OP shows that Nate Silver's critique of exit polls, telling us to ignore them, is bullshit. Exit poll discrepancies need to taken very seriously, especially when almost all of them point in the same direction. Don't let them get away with this.

Reply to this post

Back to top Alert abuse Link here Permalink

Response to Fearful (Reply #2)

Wed May 18, 2016, 03:12 PM

14. As the "official" results grow more suspect, exit polls need to be discredited.

I can remember, not too many years ago, when exit polls were considered so accurate that it was considered unprofessional to even talk about them before the polls closed! Because the margin of error on them is like +/-0.2%. That's right - way more accurate than the pile-of-crap, paid-for-by-SuperPAC polls that Nate Silver yammers on about obsessively for weeks on end.

Reply to this post

Back to top Alert abuse Link here Permalink

Response to reformist2 (Reply #14)

Wed May 18, 2016, 04:06 PM

15. That's precisely why we need them as a monitor on the accuracy of our elections

Large discrepancies should be taken as strong indication of fraud and the need for intensive investigation

Reply to this post

Back to top Alert abuse Link here Permalink

Response to Time for change (Original post)

Wed May 18, 2016, 12:51 PM

3. Looks like an entertaining conspiracy theory, but a little long of a read.

Reply to this post

Back to top Alert abuse Link here Permalink

Response to YouDig (Reply #3)

Wed May 18, 2016, 12:59 PM

4. It is not a conspiracy theory

It is a critique of a bullshit trashing of exit polls, which are used in other countries to monitor elections, but not in this country.

It is not a conspiracy theory that our votes are counted by electronic machines that are easily manipulated for election fraud. All or almost all election experts agree on that. That is a fact, not a theory.

The only question is how often that actually happens. Exit polls can be used to monitor elections and give us an idea of how often that happens.

I responded to Nate Silver's 10 bullshit reasons why we should ignore exit polls. If you have any thoughts on why my responses are wrong, other than to call them "conspiracy theories" with no other discussion, then I'd love to hear them.

Reply to this post

Back to top Alert abuse Link here Permalink

Response to Time for change (Reply #4)

Wed May 18, 2016, 01:01 PM

5. It is indeed that. I've come to be quite an expert by observing the good people of Bernopia.

Reply to this post

Back to top Alert abuse Link here Permalink

Response to YouDig (Reply #5)

Wed May 18, 2016, 01:44 PM

7. Yet do you have enough intelligence to respond to any one of my criticisms of

Nate's trashing of exit polls, other than to parrot the word "conspiracy theory"?

Reply to this post

Back to top Alert abuse Link here Permalink

Response to Time for change (Reply #7)

Wed May 18, 2016, 03:07 PM

13. they have a gameplan and theyre sticking to it.


in hindsight we will all smh that we wasted two posts on something that will not go offtask.
I think, given the modern warfare of this primary, we can observe some practices that might be dubbed
the Sidmodel.
at long last, the strategy of PISS THEM OFF becomes clear, and now it just becomes a challenge to integrate that into
a productive on topic discussion.
is disruption the last American freedom?

Reply to this post

Back to top Alert abuse Link here Permalink

Response to Time for change (Original post)

Wed May 18, 2016, 01:35 PM

6. Thank you. Excellent analysis and very easy to read/understand for us non-statisticians.

In answer to your last question: "Why doesn’t Silver even mention these substantial advantages of exit polls over pre-election polls? Could it be that it’s because he makes his living by doing pre-election polls?" Do you think there is more to it than this, i.e. that he is providing "cover" for his preferred candidate? It seems he has made some mistakes this year in other areas.

PS I worked in public health in NC for 16 years. Nice to see you are continuing to use your skills for the public good in retirement!

Reply to this post

Back to top Alert abuse Link here Permalink

Response to 2cannan (Reply #6)

Wed May 18, 2016, 01:51 PM

8. Thank you very much

I do not think that his article is cover for his preferred candidate, as it was written a long time ago (2008 I believe).

What I think is this. We all know that our corporate news media is adamantly against Bernie or any other candidate with a similar agenda. We also know that they are adamantly against any talk of exit polls to monitor elections in our country, probably because, as I say in the OP, election fraud almost always favors the more conservative candidate. They want to keep it that way.

So what I believe is that Nate couldn't possible really believe the stupid things that he says in his article. He is obviously way too intelligent for that. I believe that he was pressured to write the article by powerful people who I can only guess at. I believe that he believed that writing it would favorably affect his career or that not writing it would hurt his career, possibly badly. That is the only explanation that makes sense to me.

Reply to this post

Back to top Alert abuse Link here Permalink

Response to 2cannan (Reply #6)

Wed May 18, 2016, 04:55 PM

17. Nate doesn't do polling

"Could it be that it’s because he makes his living by doing pre-election polls?"

No, because he doesn't DO polling, he USES polling.

Reply to this post

Back to top Alert abuse Link here Permalink

Response to zipplewrath (Reply #17)

Wed May 18, 2016, 05:02 PM

19. Ok, he makes his living by analyzing pre-election polls

Reply to this post

Back to top Alert abuse Link here Permalink

Response to Time for change (Original post)

Wed May 18, 2016, 01:55 PM

9. I appreciate you taking the time to write it out, but you still misunderstand the purpose of exits

They are not and have not ever been intended to be a harbinger of who will win an election, though they have been used as such in media and by pollsters who have padded the bottom line by touting predictive questions. They are a tool for post-election demographic analysis, which is why they are performed in waves and true-upped to official results after results are realized.

NY is a good example of how this works--the first wave of exits, the ones that purported to show a close race, were overrepresented of Buffalo, NY--Bernie's best metro area in the state (and where I live). The reason for that is practical, Erie Country was one of the very few counties outside of NYC that opened polls in the morning instead of at noon. So it got hit with a lot of exit poll takers, again, for the first, early wave.

There is no exit poll conspiracy, none.

Reply to this post

Back to top Alert abuse Link here Permalink

Response to Godhumor (Reply #9)

Wed May 18, 2016, 02:21 PM

10. They are not used in this country for that purpose

Our country's election system has been rated 47th out of the 47 long established democracies in the world -- that means that it is very susceptible to election fraud. So the fact that they are not used in this country to monitor elections does not mean that they SHOULD NOT be.

And yes, we are frequently told by our corporate news media that exit polls are useless. That is because it is almost always the more conservative candidate who benefits from election fraud, and exit polls have the potential to serve as warning flags for election fraud. They don't want that. You must know that our national news media will do whatever it can to prevent Bernie or anyone else with a similar agenda from getting into a high position of power.

Reply to this post

Back to top Alert abuse Link here Permalink

Response to Time for change (Reply #10)

Wed May 18, 2016, 02:36 PM

11. OK, sure? Fine, exits shoud be used to predict winners

Last edited Wed May 18, 2016, 04:05 PM - Edit history (1)

But they don't, and they're not set up to do so, either. My point isn't what we wish exit polls should do, but what they actually are used for here in the US. It is just not possible to point to fraud because first wave exit polls are different than final results. The two don't go together.

Reply to this post

Back to top Alert abuse Link here Permalink

Response to Godhumor (Reply #11)

Wed May 18, 2016, 02:59 PM

12. I am not pointing to first wave exit polls

All of my posts that point to exit polls are the FINAL exit polls, the ones posted very shortly before poll closing, prior to the "adjustment" to fit the official count done by our news media.

It is not that they are not "set up" to predict the official vote count -- in the absence of fraud. They are perfectly capable of doing just that as they are, despite what we've been told by our national news media, including Nate Silver. Nate Silver's reasons for ignoring them as a predictive measure are nonsense, as I try to make clear in my OP.

Reply to this post

Back to top Alert abuse Link here Permalink

Response to Time for change (Reply #12)

Wed May 18, 2016, 04:52 PM

16. Those are still preliminary

They don't just poll and push the numbers out. They have to be weighted, modeled, assessed in the sample, etc. As a result, the demographic release a couple hours before polls closed and the match-up release when polls close are consist largely of first wave results plus a smattering of 2nd and 3rd depending on how fast results are processed.

Those will then change, even if not reported, as more exit data is processed and weighed against actual results.

The demographic exit, by the way, is the main purpose of this exercise. The one at poll close is primarily a bone for the pundits to have something to talk about.

Reply to this post

Back to top Alert abuse Link here Permalink

Response to Godhumor (Reply #16)

Wed May 18, 2016, 05:00 PM

18. They are not prelimary

I am sure of that.

We'll just have to disagree on that point.

They are "adjusted" to automatically fit the vote count. That hides any discrepancy that may have been present before the "adjustment". Rigging of the vote could steal 20% of the vote, but the "adjusted" exit polls done by the networks shortly after the final vote is tallied will NEVER show us that because they will ALWAYS exactly fit the official count. That is what they do.

Exit polls do not change by 10 or 20% in a few seconds. It is the artificial "adjustment" by the networks that make it look like that.

Reply to this post

Back to top Alert abuse Link here Permalink

Reply to this thread