EXIT POLLS: THE LATEST MYSTERY POLLSTER BLOG - AND MY COMMENTS.

[TruthIsAll]

My comments to his points are in bold. The Exit Poll Numbers are what they are. Very compelling circumstantial evidence of fraud. No amount of spinning can make them go away.

The crux of the spin is that we should use a 99.5% condidence interval, and not the standard 95% level. Of course, using 99.5% GUARANTEES THAT EVERY POLL RESULT WILL FALL WITHIN THE MARGIN OF ERROR.

THIS IS TOO, TOO OBVIOUS A CLASSIC DIVERSION TO OBSCURE THESE FACTS: THAT THE BUSH VOTE TALLIES EXCEEDED THE MOE IN AT LEAST SIXTEEN STATES, IF ONE USES THE CALCULATED EXIT POLL MOE IN EACH STATE; OR THAT THEY EXCEEDED THE MOE IN TWENTY-THREE STATES, IF ONE USES THE HISTORICALLY PROVEN 2.0% MOE FOR EXIT POLLS.

http://mysterypollster.typepad.com/

EXITS: WERE THEY REALLY "WRONG?"

Last week's posting of more detailed information on the sampling error of exit polls by the National Election Pool (NEP) allows for a quick review of the now well established conventional wisdom that the "exit polls were wrong."
Conventional wisdom. Of whom?

Let's first set aside the mid-day numbers that were widely leaked on the Internet but never intended as the basis for projections (numbers that even the exit pollsters recognized as flawed in some states -- see this earlier post for explanations on the differences among the many estimates provided by the exit pollsters on Election Day). Let us also stipulate that at least one state estimate of Hispanic voters (Texas) was obviously wrong, given the correction issued by NEP.

The conclusion that the exit polls were wrong is essentially about two possible failings:
• That the end-of-day numbers seemed to predict a Kerry victory.
• That the end-of-day numbers showed Kerry doing consistently better than he actually did.

What is the reality given what we now know about the exit poll's sampling error?

1) Did the just-before-poll-closing exit polls show a consistent and statistically significant "error" in Kerry's favor?

Yes, but that error has been exaggerated. Here is what we know:


• An internal NEP review of 1,400 sample precincts showed Kerry's share of the vote overstated by an average of 1.9 percentage points. As far as I can tell, no one from NEP questions the statistical significance of that overstatement.

TIA:
MY QUESTION TO THE MYSTERY POLLSTER: IS 1.9% OF 1400 SAMPLE PRECINCTS AN INSIGNIFICANT MARGIN?

• The before-poll-closing exit poll results posted by Steven Freeman (and included in the updated report by the Cal Tech / MIT Vote Project) show errors in Kerry's favor in 43 of the 51 surveys (the states plus DC). These overstate Kerry's vote by an average of 1.6 percentage points. If the surveys had been perfectly executed with perfectly random samples (an all but impossible assumption under real world conditions), the pattern of errors should have been the same as if we had flipped a coin 51 times: about half favoring Kerry, about half favoring Bush. The probability of 43 of 51 perfect surveys favoring Kerry by chance alone is less than 0.001%. Of course, this statistic only tells us that the surveys were imperfect. It says nothing about the cause or magnitude of the error.


TIA:
I HAVE CONSIDERED THE MAGNITUDE OF THE ERROR. IT IS BASED ON THE POLLING MARGIN OF ERROR, WHICH THE BUSH VOTE TALLIES EXCEEDED IN 16 STATES AND WHICH THE KERRY TALLIES EXCEEDED IN NONE.


To be clear: Everyone -- including the exit pollsters -- agrees they "overstated" Kerry's vote. There is some argument about the precise degree of certainty of that overstatement, but if all agree that the difference is statistically significant the degree of certainty has little consequence. The size of the error matters, and the reasons for it matter, but whether our level of confidence about the error's existence is 99.9% or something greater does not.

TIA:
THE EXIT POLLS OVERSTATED KERRY’S VOTE?
OR DID THE VOTES UNDERSTATE KERRY’S EXIT POLL NUMBERS?

Having said that, the second draft of the paper, "The Unexplained Exit Poll Discrepancy," continues to needlessly exaggerate the significance of the error, especially within individual states. For example, Freeman claims that there were significant "discrepancies" in Pennsylvania, Ohio and Florida when each state is considered separately. He has a graphic (Figure 1.2) showing a significant error in Florida, assuming a 95% confidence level. However, these assertions are not supported by the margins of error reported by NEP.
• I applied the appropriate "confidence intervals" reported by NEP (as distributed to its partner networks on or before Election Day) to each state. Contrary to Freeman's assertions, the separate "discrepancies" in Ohio, Florida and Pennsylvania fail to attain statistical significance even at a 95% confidence level. In fact, I see only four states (New Hampshire, New York, South Carolina and Vermont) with statistically significant errors favoring Kerry at a 95% confidence level. Of course, when using a 95% confidence level, 2-3 states should be out of range by chance alone.

TIA:
THAT IS FLAT OUT WRONG. ASSUMING THE EXIT POLL MARGIN OF ERROR FOR EACH STATE, KERRY’S VOTE LOSS WAS SIGNIFICANT WHEREVER HIS VOTE PERCENTAGE DEVIATED TO BUSH BEYOND THE MOE – AND THE MOE IS ALWAYS CALCULATED BASED ON THE 95% CONFIDENCE LEVEL, OR +/- 1.96 STANDARD DEVIATIONS FROM THE MEAN.

• I also followed the advice of Nick Panagakis and estimated confidence intervals at a 99.5% level of confidence (the standard used by NEP to make projections) using the actual estimates of the design effect obtained from Warren Mitofsky by blogger Rick Brady. By my calculations, this more demanding test renders the apparent errors in NH, NY, SC and VT non-significant.

TIA:
USE OF THE 99.5% CONFIDENCE LEVEL IS A STRAWMAN ARGUMENT. WHY NOT USE A 99.99% CONFIDENCE LEVEL? THEN YOU CAN BE ASSURED THAT IT WILL NEVER BE EXCEEDED.

Then there is the statistic heard round the world - Freeman's original "250 million to one" estimate of the odds against the discrepancy occurring simultaneously in Pennsylvania, Ohio and Florida. In his second draft, Freeman revised his estimate down to a mere 662,000 to one. Given that Freeman continues to understate the "design effect" used to calculate the sampling error in this year's exit polls, his revised estimate also remains too high.
Some have asked that I calculate my own estimate of the joint probability of an error in Ohio, Florida and Pennsylvania. I am reluctant to do so for two reasons: First, the rounding error in Freeman's data alone renders this sort of hairsplitting moot. Second, and more important, it really doesn't matter. Everyone concedes there was a small (2%) but significant average error in Kerry's direction. For those concerned about problems with the count, what matters most is why that error occurred.

TIA:
DON’T JUST LOOK AT FL AND OH AND PA. LOOK AT ALL 51 STATES. AND CALCULATE THE PROBABILITY THAT 16 OUT OF 51 STATES WOULD MOVE BEYOND THE MOE IN FAVOR OF BUSH.

I AND OTHERS AT DU HAVE CALCULATED THE ODDS THAT THIS COULD OCCUR DUE TO CHANCE:

ONE OUT OF 13.5 TRILLION.

IF YOU CONSIDER THAT FOR EXIT POLLS, THE HISTORICAL NORM MOE = +/- 2%, THEN 23 STATES DEVIATED BEYOND THE 2% MOE IN FAVOR OF BUSH.

THE PROBABILITY OF THAT IS ZERO.


2) Considering sampling error, did the end-of-day numbers predict a Kerry victory?
This one is easy: No. Not even close.

TIA:
OH YES THEY DID.

It is true that the early numbers fooled a lot of people, including the talking heads on the cable networks, pollsters and advisors of both campaigns and possibly -- depending on which accounts you believe -- even John Kerry and George Bush. The reason, as the Post's Richard Morin put it, is that the 1.9% average error in Kerry's favor "was just enough to create an entirely wrong impression about the direction of the race in a number of key states and nationally."

TIA:
ABSOLUTELY UNTRUE. ALL KERRY NEEDED TO WIN WAS EITHER FLORIDA OR OHIO. HE WAS AHEAD IN BOTH. BUSH NEEDED TO WIN BOTH.


True. But that error was not big enough to give Kerry statistically significant leads in enough states to indicate a Kerry victory. If we totally ignore sampling error, the exit polls showed Kerry ahead in only four states that he ultimately lost: Ohio, Iowa, Nevada and New Mexico. Obviously, Kerry would have won the election had he prevailed in all four.

TIA:
AGAIN, NOT TOTALLY ACCURATE. HE JUST NEEDED OHIO.

Before considering whether any of those leads were statistically significant, two quick reminders: First, as Nick Panagakis points out, NEP required at least a 99% confidence level for projections on Election Night. Second, the margin between two candidates on a survey involves two estimates; and the margin of error applies separately to each candidate. Statisticians debate which rules of thumb to apply when determining the significance of the margin between two candidates, but the consensus in the case falls somewhere between the 1.7, as recommended by the American Statistical Association and 2.0, as recommended by those who consider it more appropriate in a race where the vote for 3rd party candidates is negligible (this is a great topic for another day's post - thanks to alert reader Bill Kaminsky for guiding MP through the competing arguments).

TIA:
YOU CONVENIENTLY IGNORE THE DEVIATIONS:
OH 3.0% SIGNIFICANTLY BEYOND THE 2.21% MOE (1963 SAMPLE SIZE)
IA 1.5% CLOSE TO THE 1.97% MOE (2502 SAMPLE SIZE)
NV 2.0% CLOSE TO THE 2.13% MOE (2116 SAMPLE SIZE)
NM 2.0% CLOSE TO THE 2.22% MOE (1951 SAMPLE SIZE)


Fortunately, in this case, the statistical debate is irrelevant. None of Kerry's apparent exit poll leads in the four states were large enough to attain statistical significance, even if we assume a 95% confidence level and use a cautious multiplier (1.7) on the margin of error. As the preceding table shows, the exit polls had Kerry ahead by 4 percentage points in Ohio, by 3 in New Mexico, by 2 in Iowa and by one point in Nevada. The NEP 95% margin of error for these states multiplied by 1.7 is between 5 and 7 percentage points. At the more appropriate confidence level of 99.5% -- the one NEP actually uses to recommend projections - these relatively small margins would fall far short of that needed to call a winner.

TIA:
AGAIN, YOUR USE OF THE 99.5% CONFIDENCE INTERVAL BASED ON THE NEP CRITERIA TO CALL ELECTIONS IS NOT THE ISSUE.

THE ISSUE IS THE CONSISTENT RED SHIFT OF EXIT POLLING RESULTS IN FAVOR OF BUSH – 41 OUT OF 51 STATES, AND 16 ABOVE THE MOE.

******* ONE OUT OF 13.5 TRILLION*******

IS THAT A SIGNIFICANT ENOUGH NUMBER?


In a year-end review in this week's Roll Call (subscription required), Stuart Rothenberg concluded the following:
The problem wasn't the exits - it was the folks who treated the early-afternoon numbers as if they were a predictor of what would happen after everyone had voted. The exit poll was off by a couple of points, but that's well within the margin of error.
If only someone had warned us about those early afternoon numbers on Election Day. Oh, wait...

TIA:
THOSE "EARLY AFTERNOON" NUMBERS (ACTUALLY 4PM) WERE GOOD ENOUGH TO STAY OUT THERE UNTIL MIDNIGHT - AND THEY COMPRISED THE BULK OF THE EXIT POLL SAMPLES.

BUT MITOFSKY WON'T RELEASE THEM. WHY NOT?

UPDATE: I mentioned William Kaminski's blog, but not his commentary on this issue nor his elegant graphic -- it's worth the click just for the chart. My only remaining quibble with his graphic is that it displays a "safe estimate" of a 95% confidence of +/-4%; leaving the impression that the numbers for DE, AL, AK and NE fall outside that interval. However, the appropriate 95% confidence interval provided by NEP for these states is +/-5%.

TIA:
FIRST OF ALL THE CONFIDENCE INTERVAL IS 2.50%, NOT 5.0%.
WE ARE ONLY CONCERNED WITH THE BUSH HALF OF THE NORMAL DISTRIBUTION TAIL. AND IT'S 2.50%, JUST AS IT IS FOR KERRY. THIS IS VERY SOUND, ESTABLISHED AND FUNDAMENTAL STATISTICS. IT'S BASED ON THE EXIT POLL SAMPLE SIZE AND THE RESULTING MOE. THE EXIT POLLS HAD ALREADY ACCOUNTED FOR DEMOGRAPHICS IN THEIR DESIGN AND PREPARATION. AFTER ALL, THAT'S WHAT POLLSTERS ARE PAID FOR, ISN'T IT?

SO, LET ME SAY IT ONE MORE TIME: IN SIXTEEN STATES, THE BUSH VOTING TALLY EXCEEDED THE CALCULATED MOE BASED ON SAMPLE SIZE.
NONE DID FOR KERRY.

THE ODDS: ONE IN 13.5 TRILLION.

AND IN TWENTY-THREE STATES, THE VOTING TALLY EXCEEDED THE HISTORIC, WORLDWIDE, ESTABLISHED 2.0% EXIT POLL MOE FOR BUSH.
JUST TWO STATES FOR KERRY.

THE ODDS: IMPOSSIBLE.