Does Mitofsky’s presentation at AAPOR support his conclusions?

[Time for change]

The short answer is no, I don’t think so.

In a recent DU thread, I argued that the recent argument between Mitofsky and US Count Votes (USCV) over the meaning of the data Mitofsky presented at the recent AAPOR conference was peripheral to the main debate as to whether or not Mitofsky’s exit poll data is more indicative of exit poll bias or election fraud. Nevertheless, there seems to be a popular conception that Mitsofsky’s presentation supported his reluctant Bush responder (rBr) hypothesis, and therefore supports what he’s been saying all along: that the reason for the large discrepancy between his exit polls and the official vote count in the 2004 Presidential election was due to bias in the exit polls rather than corruption of the vote count.

The primary finding on which that conclusion is based appears to be the scatterplot that Mitofsky presented which shows a non-significant correlation between the “bias index” that Febble developed, ln (alpha), and precinct partisanship. I put bias in quotes because what it really refers to is not bias per se, but a discrepancy between the exit polls and the official vote count (with a positive “bias” indicating that Kerry does better in the exit polls than the official count) – which could theoretically be due to either exit poll bias OR corruption of the official vote count. Febble readily acknowledges that. But since nobody that I know of has yet come up with a good term for this, I will continue in this post to refer to it as “bias”. Because of the widespread belief that this finding supports or proves Mitofsky’s contention (and because I don’t agree with that contention) I believe that it is important to present arguments to the contrary. This post is my attempt to do that.

In the thread that I refer to above, “A non-statistician’s view of the E-M exit poll controversy” http://www.democraticunderground.com/discuss/duboard.php?az=show_mesg&forum=203&topic_id=371726&mesg_id=371726, I describe in general terms (item #s 3, 4, and 5 of my post) why Mitofsky’s rBr hypothesis seems implausible. In this post I will argue that his presentation at the AAPOR conference does little or nothing to change that.

At the same time I recognize that this is a very complex and controversial issue, that these issues have been widely discussed on the DU recently, and that my understanding of these matters is far from complete. So I welcome criticisms of these opinions from all interested parties, in the hope of further developing ideas which can help us understand what happened in the 2004 election, and perhaps argue for fuller investigation of our election system.


Why is Mitofski’s AAPOR presentation considered so important by some?
Again to refer to item #s 3, 4, and 5 of my previous post, Mitofsky’s modified rBr hypothesis would predict that the greatest discrepancies (between his exit polls and the official vote count), or “bias”, would appear in precincts other than Bush strongholds – and yet the data presented in his original report http://exit-poll.net/election-night/EvaluationJan192005.pdf show just the opposite of that. This fact would suggest a positive correlation between “bias” and Republican partisanship. Yet, the scatterplot referred to above indicates that there is no such statistically significant correlation. Therefore, presumably, his rBr hypothesis is saved. Here is why I don’t agree with that conclusion:


1. First, a very brief review of the rBr hypothesis is in order: If Bush voters are relatively reluctant to respond to exit polls, that would ordinarily suggest that the lowest response rates would be found in Bush strongholds. Since that is not the case, the modified rBr hypothesis was developed, which states that only Bush voters who vote in bi-partisan or Kerry precincts are relatively reluctant to respond to exit polls. This would suggest greater “bias” in bi-partisan or Kerry precincts than in Bush strongholds. This in turn would predict a negative relationship between “bias” and Bush partisanship. So, it seems to me that it would make more sense to require that a negative correlation between bias and Republican vote count margin be shown in order to support the rBr hypothesis than to require that a positive correlation be shown to refute it.

2. Even if one disagrees with the above, there should be no controversy over the fact that refutation of the rBr hypothesis should require showing a significant difference between what rBr would predict (a negative “bias” index) and what the data actually show, RATHER THAN showing a significant difference between no correlation and what the data actually show. What the data actually show are a positive correlation between “bias index” and Bush vote count margin of 0.034, with a p value of 0.23 when compared with no correlation. But when compared with the negative correlation predicted by rBr (How negative? I don’t think anyone has worked that out), the difference will obviously be greater and therefore more likely to be statistically significant.

3. USCV recently put out a paper (http://uscountvotes.org/ucvAnalysis/US/exit-polls/USCV_... ) in which they presented the results of simulated response rates for Kerry and Bush voters by precinct partisanship category, assuming the response rates proposed by Mitofsky (56% for Kerry voters, 50% for Bush voters) to explain the discrepancy between his exit polls and the official vote count (i.e., the rBr hypothesis). Because, as discussed above, the data presented in Mitofsky’s original report does not appear to be consistent with Mitofsky’s rBr hypothesis, USCV suspected that running these simulations would show implausible response rates. In fact, that’s what the simulations showed, with Kerry voters demonstrating implausibly high response rates in the Bush strongholds (average of 70%, about 17% higher than Bush voters in those precincts), with response rate differentials as high s 40% or more in individual precincts.

Ron Baiman discusses this in a recent thread http://www.democraticunderground.com/discuss/duboard.php?az=show_mesg&forum=203&topic_id=372464&mesg_id=372464, although I warn non- statisticians that they will probably have a hard time following this. Febble also gave a lengthy response to Ron’s discussion in the same thread http://www.democraticunderground.com/discuss/duboard.php?az=show_mesg&forum=203&topic_id=372464&mesg_id=372689&page= She disagrees with USCV’s use of the simulations, although it is not clear to me why she disagrees with them.

4. There was not full disclosure of the data on which Mitofsky based his presentation. Therefore, independent verification of his results cannot be made at this time.

5. Other exit poll related arguments are made in my above noted post, specifically item #s 6, 7, and 8.

6. There are also a myriad of non-exit poll related arguments that have been and are still being made, which I will not discuss in this report.

[pauldp]

1. I'm not a statistician, but THIS is the bottom line for me...

4. There was not full disclosure of the data on which Mitofsky based his presentation. Therefore, independent verification of his results cannot be made at this time.

...So, Mitofsky's rBr hypothesis continues to be under attack by respected statisticians from reputable institutions and yet he still refuses to release his raw data. Makes me sick.

[Time for change]

4. Here's what USCV says about this on the 2nd to last page of their report

"Edison-Mitofsky can materially imporove the collective understanding of the exit polls -- and whether they are evidence of vote fraud -- by a full release of the data..."

[autorank]

2. And Guess who owns the Nat. Exit Poll Data -- THE NETWORKS

pauldp is right. You can't conclude this discussion unless the data is released and in a way that's verifiable; in other words, it turns out to be the real data collected, not a retrofit.

Then let us decide.

It's like any type of scholarship. For example, the Dead Sea Scrolls were discovered some time ago and are controlled by a committee. Just great, you think we know what's there in as much detail as necessary, not close. The Nag Hamadi find of the Gnostic Gospels was not "controlled" or "juried in private" and we have all of them. Want to translate them yourself, argue with translations, have at it.

Mitofsky treats his data like the Deat Sea Scrolls with the "clergy of American elections," the * wing of the Republican Party (meaning all of it) calling the shots. Mitofsky did the study but he doesn't own the data. Those own it control it's release. Want to guess who owns it?
---------------------------------------------------------------------
TV Networks Officially Refuse to Release Exit Poll Raw Data
Mainstream media finally displays true colors

by Gary Beckwith the Solar Bus http://election.solarbus.org December 22, 2004
http://www.art-science.com/Bush/b11.html

So, Conyers wrote to Warren Mitofsky, who owns the exit poll data, asking for the complete raw data, without the "real" numbers mixed in. Mitofsky balked, saying that the TV Networks actually own it and he was not able to release it without their permission. Conyers then took his inquiry to the leaders of ABC, CBS, NBC, CNN, and Fox.

And they promptly laid an egg. Through a spokesperson who spoke on behalf of all the media companies together, they said they are still analyzing the data and don't want to release it until they're done.

[Time for change]

3. So with the combined resources of all the major TV networks

it takes them 5 months to analyze some data?

[autorank]

6. They're using TI-80 Series machines with an abacus as QA

"It's hard work, hard work." Plus they're also calculating all the revenue they're losing from diminished viewers, particularly on their most profitable show, the nightly news.

We need to have some:sarcasm:compassion for the news sheeple.

They want to give us the data but they just have to make sure it comes out right.

:hi:

[TruthIsAll]

5. NO, and this proves it.

http://www.democraticunderground.com/discuss/duboard.php?az=view_all&address=203x373826

[Time for change]

8. Thank you for the link TIA, could you help explain it to me?

I have two questions:

1. Can you tell me what the percents in your simulations refer to?

2. It looks like your simulation # 5 has Bush winning. Does that mean that you believe that assumptions based on that model are reasonable, and therefore you think that Bush might actually have won the election?

[TruthIsAll]

10. An explanation.

I ran 8 simulations (10,000 per), each assuming a different response curve. The average response rate for all 8 (80,000) was close to 53%.

Kerry's average response rate was 53%.

Simulation #5 is an extreme case, as it assumes a uniform Kerry 50% response rate.

By the way, I'm working on the model to make it more general.

[Time for change]

11. The percents given by precinct in your first table

Are these Kerry response rates or overall response rates? If the former, how do they compare with Bush response rates, and if the latter, how do response rates differ between Kerry and Bush?

[TruthIsAll]

12. They are Kerry response rates relative to Bush ( 50%).

For example, if the Kerry response is 53%, then the ratio (alpha) = 53/50 = 1.06.

[Time for change]

14. Ok, thanks for the clarification n/t

[Febble]

7. Excellent post

A few points:

The reason I chose the word "bias" is precisely because both counting and polling can be biased. It was not easy to choose a neutral word, and, especially since being misunderstood, I have tried to take care always to follow it by the words "whether in poll or count". Biased counting is what we are worried about. Biased polling certainly happens.

A second point is that the mean WPEs in the E-M report for five somewhat arbitrary (and numerically very uneven) categories, and the linear correlation between vote-count margin (or Bush's share of the vote, if you like) both show a slope, with more negative WPEs at the high Bush end.

However, what the new plot shows is that the correlation between "bias" (ln(alpha) and vote-count margin is not significant, suggesting that both slopes may be an artefact of the warp in the WPE as an index. What we don't know yet is what the mean "bias" values are for the five categories - I suspect, eyeballing the plot that the mean "bias" would indeed still be higher in the high Bush category, although not as out of kilter as the WPEs. However, it is also apparent, from eyeballing the plot, that the level of this mean is going to be extremly sensitive to where you draw the category boundary. There is a very low point lurking at 79% Bush that will pull it down drastically; and if you moved the boundary a bit higher you might miss the four very high precincts that are pulling the mean upwards.

In other words, what you find in the means is critically dependent on where you draw the category boundaries, which is why, generally, in statistics we don't categorize continuous variables, and when we do we use percentile splits in order to keep the category Ns the same (N=number of data points) in each category.

I quite agree that Mitofsky's new plot does not disprove fraud, nor prove that response bias was the reason for the net "red shift". It is however, fairly strong evidence against the hypothesis that fraud was greatest in Bush strongholds - or even in apparent "strongholds" that only looked like strongholds because of all the switched votes.

My simulation* here

http://uscountvotes.org/ucvAnalysis/US/exit-polls/simulators/liddle

suggests that randomly distributed fraud should produce a pretty linear slope - and it is hard to model fraud on the scale required to account for the magnitude of the "red shift" without producing a "significant" slope (I have probably underestimated the variance in this model - however, increasing the variance does not hide the slope if fraud is extensive enough to account for the red shift).

So my current thinking is that a fraud hypothesis that is consistent with the data plot shown by Mitofsky at AAPOR must entail greater fraud in KERRY strongholds. This would hide the tell-tale slope - which, it seems to me, is simply not there.

This was not my thinking when I embarked on this whole forensic analysis of the exit poll data - and much of it has derived from off-line discussions with many people, so I am not claiming originality here.

But it now seems to me that the fraud hypothesis most consistent with the data-as-we-know-it now is actually the opposite of that proposed (quite understandably) in the original USCV paper - i.e. greater vote-corruption in Kerry strongholds.

After all, those are the precincts that would give you the biggest bang for your buck.

I'm not saying the new data plot proves this. But that the fraud hypothesis that would be consistent with it would be one that located the fraud greater fraud at the left hand end of the plot, where it would hoist up that end of the regression line.

Finally, it is worth noting that it is not just the suspect high "ouliers", whether in the high Bush category, or in elsewhere, that are leveraging the red-shift - it is pretty obvious from the plot that even if you ignore all those flies around the dense stuff in the middle that the mass of precincts are floating high of zero. In other words, if fraud is responsible for the red shift, it is pretty pervasive.

So to summarise, if I can: I think what the plot Mitofsky presented at AAPOR does is to put some fairly firm parameters on what pattern extensive fraud would have had to have to produce the total red-shift, and also to rule out some patterns that have been suggested.

There are plenty left .



*To use the simulation, choose the "alpha" you want - a value between 0 and 1 will give you "reluctant Kerry responders" and a greater than one will give you "reluctant Bush responders". A value of 1 will give you equally enthusiastic (or reluctant) responders. Then you can choose what proportion of precincts to corrrupt, and the scale of the corruption. Tap the F9 key to get the random number generator to recalculate. You get a slope in the WPE even with random bias, but not with the ln(alpha) plot. But fraud produces a slope in both.

[Time for change]

9. Interpretation of the non-significant slope of bias index vs. partisanship

Thank you for the extensive discussion Febble.

I think that the key issue here is the interpretation (i.e., what does it say about the likelihood of election fraud vs. exit poll bias) of the non-significant slope when your "bias" index is plotted linearly against vote count margin. I argue in my original post to this thread that Mitofsky's scatterplot does little to support rBr and may even argue against it. The main gist of your reply (I believe) is that, irrespective of what the non-significant slope says about rBr, it casts at least some doubt on the likelihood of fraud. This is obviously a central point to this whole argument, and therefore I have done a lot of thinking about that. I have thought of five reasons why the non-significant slope may yet be consistent with fraud, and I would be interested in your opinion of these:

1) Fraud occurring primarily in Kerry strongholds -- You have already agreed that this is a real possibility.

2) Fraud very widely distributed across precincts -- I myself have a big problem with this one, as it doesn't seem very plausible to me -- but who knows?

3) None of the above, but even though the slope is not significantly positive it may still be positive enough to account for the red shift -- I believe that you are saying in your response that your model rules that out, is that correct? But if so, could it be that that is because your model is based on assumptions which may not be valid?

4) I argue in points 1, 2, and 3 of my original post that rBr is not very plausible. If those arguments are valid, then almost by definition fraud is strongly supported, since rBr almost by definition is required to support the red shift if fraud didn't occur on a massive scale.

5) Doesn't your model depend on certain assumptions about a) the slope of the line in the absence of fraud and b) the variance of ln(alpha)? In other words, if the slope was negative in the absence of fraud, then couldn't even large amounts of fraud fail to turn the line significantly positive, or if the variance was large couldn't large amounts of fraud fail to be seen above the noise?

[Febble]

13. Some thoughts....

Your point 2:

I'd like to know the nature of your problem with this one - my problem is that I think it would produce that dratted slope that isn't there. In my simulation, randomly distributed fraud gives a slope. This was counter-intuitive to me at first, but I now see it. So I agree this one is looking implausible for fraud on a scale to account for the red shift.

Your point 3: Yes, it is possible that my model is based on invalid assumptions, and there is one that might be relevant - unfortunately it argues the wrong way. There is also a relevant observation. The observation is that the slight slope looks to me as though it is being leveraged by the very low points at the high Kerry end, not high points at the Bush end. It is important to remember that leverage increases at the extremes. However, before anyone accuses me of saying that the slope means pro-Kerry fraud in high Kerry precincts (although it is of interest to note that any fraud will produce the same slope - who is doing the fraud will affect the intercept of the slope, not its direction), I think there may be one remaining artefact in my bias index, which is to do with the distribtion of probabilities when a candidate's votes are low and the sample size is also low. In essence, where there are only, say 1 or 2 Bush voters in a precinct, there is less room to understate the Bush vote than to overstate it. The distribution rapidly approaches normal as the probability of finding a Bush voter (or a Kerry voter at the other end) increases, and it also rapidly approaches normal as sample size increases. But it does mean that in very extreme precincts, the minority candidate's vote will tend to be overstated more often than understated, and thus give a spurious tilt to the regression line. I think that is probably what is happening in that bottom left hand corner - and I think that is what may be giving the regresssion line its tiny tilt. It will certainly be contributing to it. (There are no equivalent very high Bush precincts.)

If not, your question is: could the tilt be large enough to account for the red shift - my modelling suggests no, and other versions of the model, in which I increase the variance still further, also suggest no. But it could be hidden by noise variance - in other words something could be squashing it, maybe some innocent factor hoisting the line up at the Kerry end - or pulling it down at the Bush end. I can't think of one right now. Or it could be just where random factors happened to fall.

your point 4: if response bias is implausible, yes, fraud is more likely. So what might indicate the implausibility of fraud: your first point in your original post says that it is implausible because response rates weren't lower in Bush precincts. Three factors will influence response rates: miss rates; refusal rates; and the extent to which the interview accurately records either. (this is to leave out total sampling rate - the percentage of each group of voters polled in total, which will also depend on interviewing rate).

The plots we now have include a plot of refusal rates - which ought to be the most sensitive to "reluctance" ("miss" rates ought in theory to be random). This does not, as you say, show that refusal rates were higher in Bush strongholds, which they ought to do, as you argue, if Bush voters are more reluctant in any company. However, statistically, it seems apparent to me that the power to detect a higher refusal rate is not adequate, given the variance in the data. In other words I think we simply do not know whether Bush voters were more likely to refuse from that plot, as the variance was very high. That variance could becoming from random factors (weather is my favorite, but there are others!) that affect both groups of voters and also from misreporting of refusals. In other words, just as random factors may be hiding the "fraud" slope in the other plot, it may be hiding the slope in this plot that would support "rBr". But I agree - it's evidence - however, I think it's weak evidence, and certainly does not make "rBr" "implausible.

I'm not sure I understand your point 2 in the original post - unless you are saying that because the refusal line plot is flat, bias ought to be higher in Kerry precincts because Bush responders must have been more reluctant there? Again, I think the answer to this is "variance" - both tilts even in theory (assuming a mean of alpha of 1.12) would be very small - the absence of either is not very convincing to me, although I take your point. There is a big problem in using the absence of an effect to prove a point. Retaining the null never means the null is proven, it just means that the effect, if it was an effect, was too small to be detected with the statistical power you had. And the statistical power is reduced as variance increases - and there is a heck of a lot of variance.

Regarding your point three in the original post - I think Ron's conclusions depend on a very tight proportion between "response rate" as indicated by misses + refusals on the tally sheet and "sampling rate" which is the proportion of each voter polled, which will also depend on things like interviewing rate. If the tallying was inaccurate, and if smaller interviewing rate allowed for non-random sampling, then these two measures will not be directly proportional. It may well be that you will get "implausible" response rates - but their implausibility would not necessarily mean fraud. It could simply mean that the response bias did not operate only at the level of overt refusals. Which seems to me plausible. My willingness (from bitter experience) to accept this as plausible is probably at the root of my argument with Ron. I know only too well how easy it is to preferentially sample those that look willing, and for those who are really unwilling to avoid being sampled at all. Which is not to say it happened that way - just that the "implausible" response rates are not in themselves refutations of "rBr" - just refutation of the idea that any bias in the poll occurred solely through through differential refusal rates.


Your point 5: I think I might have covered this one above! I think the answer is "possibly". Who knows what invalid assumptions I have made? The trouble is that so far, the ones I keep tripping over turn the line the wrong way! I'd be delighted to hear of one that didn't.

[Time for change]

15. Some more thoughts

1) Clarification of item # 2 in my original post: My point was that rBr predicts a negative slope, not a zero slope. Therefore, to refute rBr, one should not have to show that the actual slope is different than zero, but rather that it is different from the negative slope predicted by rBr -- an easier task, and one that hasn't been attempted, to my knowledge.

2) Clarification of item # 2 of my last response to you. I wasn't speaking of random bias, but rather widespread election fraud in almost all precincts. That would produce a zero slope. My problem with that is that is doesn't seem very plausible to me.

3) Clarification of sampling rates, response rates, miss rates, etc.: I think that this is complicated enough without the need to get into all that. Isn't it just as well to speak of "sampling rate", where the sampling rate is simply those interviewed vs. those attempted to be interviewed (which I believe would be the miss rate plus the refusal rate)?

4) This is what I would most like your response to: You speak of invalid assumptions used in the USCV simulations. I found that one gets similar results, perhaps even more extreme, using algebra to calculate Bush and Kerry response rates (actually, samplying rates), without making any assumptions other than that Mitofsky's data presented in their original report is correct. I've discussed this in some more detail in response to Ron Baiman's response to me in the thread that eomer started:
http://www.democraticunderground.com/discuss/duboard.php?az=show_mesg&forum=203&topic_id=372464&mesg_id=374248&page=
If you're interested I could show you the calculations that I used to come up with this.

[philb]

16. Seems to me Mitofsky is ignoring a huge bias against Kerry caused by

the millions of minority and student voters who were unable to register or vote due to documented manipulation of registrations, number of machines in precincts, identification process, dirty tricks, late polling place changes in minority precincts, inability to get absentee ballots, etc.
As shown by the large numbers of reports by voters to the call in hotlines and hearings in Ohio and Florida. Most of these were illegally prevented from voting, so were not dealt with by the Exit Polls. Is the argument that its too bad about the huge amount of manipulationa and dirty tricks documented in the election, and now we should just forget them and deal with those who were allowed to vote.

[Time for change]

17. You make an important point

The issue that you refer to has nothing to do with the exit polls -- since these people didn't vote they were not exit polled.

But it's still a very important point because, even if the exit polls showed Bush winning, and even if Bush did in fact get more votes among those people who actually voted, it still wouldn't be a fair election.

[Febble]

18. I agree!

And I think the voter suppression issue is huge. I still think that suppression alone could have cost Kerry Ohio.

[TruthIsAll]

19. This exit poll response optimization proves E-M is wrong to float rBr

It's an OPTIMIZATION, NOT a SIMULATION analysis.

http://www.democraticunderground.com/discuss/duboard.php?az=view_all&address=203x374165