Reply #88: Polls versus elections

[ConsAreLiars]

Pobeka, you ask the right question:
Can we really point to this election and say "wow, this one really did fly in the face of the odds, given the election history that preceded it?".

The first problem is that the margin of error ("results are accurate within + or - x%") means only that if you go through the same polling routine 20 times you can expect the numbers yielded by one sample to fall outside that range and 19 to fall within (usually). It says nothing at all about the utility of the poll to predict any real world action. Poll results only give estimates of the results one would get if everyone in a certain category were asked the same questions, but for anything else, like predicting actual decisions, a lot of assumptions and guesses are needed. Obviously there are numbers of steps that can be taken to improve the value of the poll as a predictor, and the better operations do a lot to try to devise methods for getting answers that are useful for predicting elections, but the margin of error numbers say nothing at all about how close the poll numbers might be to election numbers. The "margin of error" numbers simply tell you how many people were interviewed.

In fact, one source indicated that election results which actually fall within the margin of error for polling samples seem to be rather rare:
http://commonsense2000.tripod.com/poll.htm

The polling industry interest group NCPP, using a number called "candidate error," reports that of 159 polls on senate and governor races in 2002 16% fell outside that range, but the public document does not give actual distributions. Interestingly, "candidate error" numbers are actually one-half the real error. It seems that the justification is that that deviations from polling sample numbers occur when individuals switch sides at the last minute, rather than assuming the actual numbers reflect simply how many partisans from each side showed up and had their votes counted. Even so, using their industry-friendly analysis, the odds that election results will fall outside the expected margins is one in six rather than the assumed one in twenty. Using the less industry-friendly assumption, results outside the margin of error might be just as likely as results within.
http://www.ncpp.org/2002SenGovPoll/2002ElectionPolls.html

So the first question is "how closely correlated are poll-based estimates and election outcomes?" This is an empirical question. To answer it means devising some method for gathering together in an unbiased manner a large enough amount of data from polls and elections. The next question is whether some factors can be identified which have a significant impact on the correlation between polls and vote tallies.

One hypothesis might be that the more Diebold machines in use in an election the lower the correlation between polls and results. Another level of analysis might be to separate out "hotly contested" versus other elections, using some pre-election criteria to define the two groups. Etc.

The difficulties are several - 159 cases is not a lot to work with if this one cycle of statewide races was the sole focus. Getting more would be a lot of work even if the data behind the NCPP report could be obtained. Also there is no reason to suspect that ES&S systems are any less vulnerable that the Diebold machines, given their common parentage. Further, if there was some sort of tampering going on, and if a detailed comparison of polls and votes were done, the leap from "anomolous" to "fraudulent" is still a matter of inference rather than proof.

On the positive side, what could be done, if those 159 bits of data could be obtained, would be simply to graph them out and color-code the D and R sides and/or something about the voting tech used. Trying to put odds on a specific group of results is impossible with the info now available, and collating all the relevant data would be something that could only be done with a large research grant, unless this data has already been compiled at some location I haven't yet found. Such a graph might show a few races standing off in their own space. apart from the bulk of the cases, or the races mentioned here might just blend in with the rest due to all the noise in that chart.