Ron Baiman: Sunmmary of the USCV/E-M and Liddle Debate

[eomer]

<note from eomer: this post by Ron Baiman (http://www.democraticunderground.com/discuss/duboard.php?az=view_all&address=203x371726#372424) deserves its own thread>

Dear Democratic Underground Readers,

I have asked to post on Democratic Underground. I hope this "summary of the debate" is useful.

The following summary was written as a response to a post complaining that USCV (www.uscountvotes.org ) was making a fuss about minor discrepancies, implying that the E-M case had been proven by the scatter plot of the WPE-index (Ln(alpha) – the “Liddle” index) shown by Mitofsky at AAPOR.

The response is, almost in full, as follows:

a) USCV initiated the study of partisan exit polling response rates in order to show that implausible partisan response rates are necessary to generate the aggregate tabulations released in the E-M report (see USCV March 31 updated April 12 report). The report develops a methodology to calculate Kerry and Bush voter exit poll response rates (K and B) which when multiplied by their respective reported vote shares (k and b) and added up (assuming negligible independent vote) give an overall exit poll response rate (R = Kk + Bb). Table’s 2 – 4, p. 11 and p. 25 of this report, show that implausible variations in K and B from “representative” (mean valued) precincts would be necessary to generate the mean WPE’s (E) and overall response rates (R) shown in the E-M report.
b) Elizabeth Liddle, who was an active participant in the USCV discussion list at the time discovered that when simulating the logged ratio of possible K/B values – based on the derivations developed in the April 12 USCV report - (Log (K/B) - the log was added – I believe at the suggestion of another participant on the list - to make the ratio symmetric), over partisan precincts, a noticeable “inverted U” asymmetric shaped WPE pattern emerged. This was particularly noticeable when simulating an (implausibly) high 2:1 K/B ratio (see Liddle figure 1 - displayed by Warren Mitofsky as part of his AAPOR presentation).
c) Liddle may have unknowingly reinvented the “wheel” algebraically but her index (the WPE_Index displayed by Mitofsky at his AAPOR presentation) is equal to Log(K/B) where all these terms are based on the derivations developed in the April 12 USCV report (see Appendix C –D in latest, May 12, updated May 21, USCV report). The importance of this is not only one of correct attribution of credit (appreciation of the value of an analysis should not be dependent on the conclusions that are derived from it!), but also to point out that USCV and Liddle have been looking at the same variables (partisan exit poll response rates) derived in a mathematically equivalent fashion.
d) E-M seems to be mistakenly, or deliberately, trying to create the impression that the Liddle analysis is based on some new, hitherto undiscovered “artifact” or “confounding”, that resolves the debate and shows that the mean rbr hypothesis can, after all, in spite of appearances to the contrary, explain the data. This, unfortunately, may partly be a result of the appearance of Liddle’s paper, in which she derives “alpha” (in a rather complicated and convoluted manner) apparently without reference to the equations for K and B presented in earlier USCV reports. (The obvious way to get alpha, once K and B have been derived, is in one or two easy steps as is shown in Appendix C of May 21 USCV report.) In addition, though she refers to the USCV paper in at least one footnote, Liddle unfortunately does not point out that the USCV analysis was based on these very same K and B partisan response rates patterns that she investigates in her paper.
e) In any case, based on her finding of the “asymmetric inverted U” pattern, Liddle came to an opposite conclusion to that of USCV. She surmised that this pattern indicated that the E-M data could have been generated by an unvarying mean partisan response “bias” which she defined as K/B.
f) How could she have come to such a different conclusion? The USCV reports used E-M reported mean WPE and R values to calculate K and B levels in different categories of precincts.. These K and B values diverged markedly in implausible ways across partisan precincts. Liddle, on the other hand, simulated (K/B) to get the suggestive pattern displayed by Mitofsky above. By looking at an exaggerated (by assuming a 2:1 K/B ratio equal to an alpha of 2) simulation of the K/B ratio she produced an “inverted u” shaped WPE graph that seemed to produce an “inverted u” WPE graph that seemed curved enough, and asymmetric enough, so that it appeared that it could approximate the know E-M reported WPE outcomes.
g) In response to Liddle’s pointing out this asymmetric inverted “u” WPE pattern, USCV added Appendix B (which appears in both reports) which derives this pattern from “Differential Partisan Response” (w = B – K). This appendix shows why the “inverted u” pattern appears, why WPE will be at a maximum perfectly competitive districts (k=b=.5), and why differential partisan response (w) will be equal to WPE (E), if k=b=.5 and R = .5. Appendix B also points out that mean calculations that were already done in the USCV (April 12) report show that the “inverted u” finding makes the jump in WPE to -10% in high Bush precincts even more implausible (as highly partisan precincts should have smaller WPE), and that to get the WPE’s for the other categories of precincts, the calculations in the report show that w would have to swing from 40% to an absolute minimum of 20.5%.
h) In addition, Appendix B shows that the “asymmetry” of the “inverted u” WPE curve – which gives a larger WPE in high Republican precincts (see Liddle Table 1 and Mitofsky presentation) that seems to be consistent the (much higher) WPE of high Bush precincts in the E-M data, is a mathematical result of linking an absolute difference (WPE) measure to a ratio measure (alpha). This “mathematical nit” cannot possibly explain the dramatic asymmetry in the E-M data (see the WPE’s generated by a constant alpha = 1.15 in Table 2, p. 19, May 21, USCV report). Moreover, if an absolute difference “differential partisan response” measure is used (w= B-K), even this small asymmetry disappears altogether. Only with highly magnified levels of Alpha (such as a 2: 1 ratio representing alpha=2) will this small effect look significant.
i) This debate with Liddle went on for some weeks. USCV members did further calculations based on means and medians with “alpha” (=K/B) and showed that using either means or medians “alpha” would have to range from below or almost equal to 1 in high Kerry precincts to 55-58% above in high Bush precincts to get the E-M reported WPE outcomes for representative precincts (see Table 1 in Appendix F, p. 19 of most recent May 21 USCV report.)
j) USCV showed that the overall response rate (R) levels that would be generated by representative precincts under the E-M “alpha” hypothesis of K =0.56 and B=.5 were mathematically infeasible in high Bush precincts and highly implausible in high Kerry precincts (R would have to drop to 16.9% - 29.2% in these precincts). Note that Liddle’s index is simply a ratio index that says nothing about overall response rates. Though K and B can be generated from r (=(B+K)/2) and w (see Appendix B) , they cannot be calculated from alpha (=K/B) alone. Another parameter like r is needed to get K and B and overall response rates (see derived equation in Table 4 of Appendix 4.
k) Liddle attempts to address this in her paper by noting that the E-M report states that the overall response rate (R) changes are not significant, implying that the bias (alpha = K/B) is all that needs to be looked at. But this skirts the issue. If a unvarying mean bias (mean constant alpha) hypothesis can only be sustained with radically divergent and implausible overall response rates, the hypothesis cannot explain the data, which as noted, show small (and possibly statistically insignificant) variations in response rates. USCV’s first report also showed that these small response rate variations contradict the E-M hypothesis.
l) None the less, Liddle was not swayed that a constant mean response bias (alpha) could not possibly (with any reasonable degree of chance) explain the E-M data. A key issue at this point was whether “aggregate” calculations using means and medians (such as the ones the USCV had used in its first report and in Appendix B) could adequately represent outcomes from randomly varying precinct level simulations. Recall that Liddle’s analysis at this point was based entirely on hypothetical simulations. In response USCV did some output simulations – trying to match E-M WPE and response outcomes with constant alpha. These also showed that matching E-M reported mean and median WPE levels, and over-all response rates with constant alpha was highly improbable to impossible (see Appendices G and H in the May 21 USCV paper).
m) Liddle remained unconvinced. She went ahead and published her paper, which she had already written about on several web sites. Her paper was hailed as holding the key to saving the constant mean bias hypothesis. The fact that it was based on the very same data (and pretty much the same analysis that USCV had be doing to show the opposite) was overlooked.
n) USCV felt pressured to respond to the Liddle paper by releasing the May 21 paper, and sending representatives to the AAPOR conference. This new USCV paper in addition to the calculations in k) above, also included an input simulation that showed that the E-M hypothesis could not explain the E-M outcomes, even under the most extremely (favorable to the hypothesis) precinct distributions (see Appendix I ). Moreover, these simulators (one of which is in completely transparent Excel form) have been put on the USCV website so that anyone can verify these results.
o) The Liddle and E-M hypothesis had been rejected at this point in at least three different ways.
a) Analysis of “representative” mean and median precincts showed that it was mathematically infeasible or highly implausible (Appendix F, May 21 USCV report).
i. Alpha’s necessary to produce the E-M data change by more than 62% (from mean calculation), or 52% (from median calculation – see Table 1, p. 18, May 21 USCV report).
ii. An alpha of 1.15 (representing an even greater “bias” than the 1.12 hypothesized by E-M and Liddle) is unable to generate the large E-M reported WPE values for high Bush and competitive precincts, and the small WPE for high Kerry precincts (from mean calculations – see Table 2, p. 18).
iii. Under the E-M hypothesis of K=.56 and B=.5 (so that alpha=.56/.5=1.12), the overall response rates (R) for high Bush precincts would have to be a mathematically infeasible -564.4% (from mean calculations) or also infeasible 423.3% (from median calculations). Overall response rates in high Kerry precincts would have to be a highly implausible 16.9% (from means) or 29.2% (from medians), see Table 4, p. 19.
b) “Output simulation” shows that with a high degree of certainty, matching E-M mean and median response rates, and overall response rates, requires significant unexplained changes in K, B, and alpha (Appendices G-H, May 21 USCV report, especially table on p. 20).
i. non-uniform mean alphas that change by at least 31% across partisan precincts.
ii. non-uniform partisan exit poll participation rates (K changes by at least 16%).
iii. High Bush precinct Kerry voter exit poll participation rates that are much higher than Bush voter participation rates (by at least 16% and sometimes up to 40% to 60% higher – p. 21).
c) “Input simulation” shows that it is impossible, under the most extreme favorable to the Hypothesis circumstances, to get E-M reported results from an E-M constant mean K=.56 and mean B=.5 hypothesis (Appendix I, May 21 USCV report). In particular this simulation shows that the E-M reported:
i. High Bush precinct mean WPE
ii. High Bush precinct median WPE
iii. Low Kerry precinct mean WPE
iv. High overall response rates in high (b>.8) and moderately high Bush precincts.
Are all unobtainable under the E-M hypothesis.

p) None the less, E-M, based on the Mitofsky AAPOR presentation and your comments, has embraced the Liddle analysis as providing conclusive evidence for the constant mean bias hypothesis. The key part of this argument appears to be the WPE by LN(alpha) linear correlation analysis presented by Mitofsky at AAPOR that we have been discussing. The question of how such a simple correlation analysis could trump the data already presented by USCV regarding representative precinct mean and median partisan and overall response rates across partisan precinct categories, and the more recent conclusive simulation outcomes, seems to have been lost or deliberately ignored.
q) It should be noted that the E-M report itself simply supplied tabulations and discussions of “factors” that could influence exit poll response and bias and an assertion of the E-M, that partisan response rate of K=.56 and B=.5 could explain all of the WPE error (p. 31 of E-M report). No solid statistical evidence (for example a multiple regression analysis actually showing that the factors can explain the WPE patterns, and that these factors result in .56 and .5 response rates, is offered to support this hypothesis. By the way, I agree with you that the term “reluctant Bush responder” is inaccurate. I was simply using the term as it has been coined by the media. One of the ways to get beyond the simplistic psychological metaphors, to the real factors that influence response rates, is to do the serious multi-factor analysis.
r) Mitofsky claimed, when I queried him about this at the AAPOR conference, to have done the regressions but not released them. This is doubly unacceptable! If they were done, the public has a right to see them! However, I am somewhat skeptical that they have been done, or at least done in a thorough and complete manner, as cursory analysis shows that K=.56 and B=.5 (generating an alpha of 1.12) could not possibly explain the relative magnitudes of WPE shown in the E-M report. See Table 2, p. 19, May 21 USCV report: alpha had to be increased to at least 1.15 to get WPE’s in range of the E-M data. It seems to me that this hypothesis, stated with such certainty on p. 31 of the E-M report, could not have come from an in-depth and serious statistical analysis by some of the “best analysts” in the country!
s) As I stated in my earlier post, the kind of tabulation, and now linear correlation analysis, that E-M has released to the public, would never pass muster as supporting evidence in any kind of serious academic journal (including one that I am an editor of). The gullibility of the media in support of the constant mean bias hypothesis without any serious evidence for it has been a travesty. The notion that “these things take time” etc. is also unacceptable. The credibility of our election system is an extremely important national issue – it should not take six months or more to provide a serious analysis (especially if some of the “nations best” analysts have been looking at it) of such an important issue. Moreover, there is no reason that private business contracts or personal confidentiality should trump critical public interest in this data. There are ways to release this data that protect confidentiality (as has already been done for Ohio). There is no sufficiently important or legitimate reason for E-M not to release the data and very good reasons, relating to a minimal sense of public responsibility and survey ethics, for E-M to immediately release the data without further delay. This is what I meant by “E-M needs to release the data”.
t) It may seem a bit beside the point, after all this to be debating whether a single linear correlation analysis of the E-M data is consistent with the non-varying mean exit poll response bias hypothesis. After all, the statistics (mean, median, and absolute value WPEs, and mean overall response rates) for the different precinct categories have been calculated and reported on. These are clearly highly divergent. Moreover, the influence of precinct partisanship has been eliminated from these data by calculating the direct K and B response rates (all done in the April 12, USCV report) and these show that implausible changes in K-B and K/B (the log of which is the Liddle/Mitofsky WPE-index) are necessary to generate these data from “representative” precincts. Why then should we be debating whether an insignificant linear correlation between this WPE-index and precinct partisanship shows that it unvarying?! We have already done the calculations and the analysis showing that this is not the case! For those concerned about “aggregation bias” in using “representative” precincts, we have shown that this hypothesis is highly implausible, if not mathematically infeasible, with precinct level simulations as well – see point o) above.
u) I submit that whether or not the scatter plot, considered as whole, produces a significant linear correlation or not, is under these circumstances, irrelevant. After all, a zero correlation can be produced with any number of non-linear variations. In this case the range of alpha’s (taking natural logs of mean and median alpha columns in Table 1, p. 19, May 21, USCV report) is (going from low Bush to high Bush quintiles), from means: - 0.0166, 0.1448, 0.1704, 0.1414, 0.4626, and from medians: 0.019, 0.137, 0.168, 0.141, 0.438. This would seem to imply a lot of variation and a positive correlation (with b). However, evidently, because of the very small sample sizes for the highly partisan precincts, the “inverted u” (not flat linear!) shape of the alpha from 90% of the data that is clustered in the less partisan precincts is sufficient to generate a flat zero correlation.
v) This “inverted u” alpha (not evident when drawing a straight line through the scatter plot) suggests that constant alpha is insufficient to generate the large WPE for competitive precincts (-8.5%) relative to the lower WPE levels (-5.9% and -6.1%) for the less partisan districts. This in it self may rule out unvarying alpha. This will depend on the significance levels of these differences – but given the very large sample sizes for these precinct categories, small differences in mean alpha levels (of +0.03 or so) are likely to be significant.
w) You make claim that the only unusual thing about the scatter plot data are four high Bush outliers in the high Bush quintile that are not offset by any high Kerry outliers, and what’s the big deal about four points, though you support investigating these precincts. First, it is curious to have only 40 high Bush precincts compared to 90 high Kerry precincts (measured by reported election outcome) in an election that Bush won by 2.7%? One would expect rough equality or a slightly larger number of high Bush precincts in a representative sample. This looks strange and may indicate that some other high Bush “outliers” have already been dropped from the sample. Second, even if for some reason there were less than half the number of high Bush precincts in the sample, four outliers represents 10% of a sample of 40. If 10% of all of the high Bush precincts in the country were corrupted, this could represent a very serious problem.
x) Moreover, all of the other ways in which these data are not consistent with constant mean alpha cannot be addressed by simply removing the four outliers. Whether or not the high Bush outliers are removed, the USCV reports have shown, for example, that the E-M hypothesis is not consistent with the high Bush median (which presumably would not be greatly affected by removing outliers) with the high Kerry mean, and with the high, and relatively high, Kerry overall response rates.
y) However, I reiterate, the basic point about the correlation analysis of the scatter plot is that it is wholly inadequate. Such an analysis “of the whole” will not provide detailed (or accurate if the variation in alpha is non-linear) information about what??s going on in the most interesting extreme partisan precincts where the constant mean bias hypothesis is really put to the test. As has been shown above, detailed analysis of the E-M constant mean response bias hypothesis breaks down in multiple ways particularly in these kinds of precincts.
z) The constant mean bias conjecture remains an unsupported (and largely inconsistent with the data that has been made public) hypothesis. Six months after the election, we still have no serious explanation for the large exit poll discrepancy. The shoddy (if I could borrow a term) and inadequate analysis (claiming for example that tabulations and linear correlation analysis are sufficient to support the E-M hypothesis) that has been released to the public has just deepened the uncertainty about what happened in the 2004 elections. I don’t see how this could be viewed as anything other than a national disgrace. The volunteer work of USCV, and other citizen activists who are deeply concerned about the credibility and/or integrity of our electoral system, and have refused to be satisfied with this pabulum, may, in fact, be the one glimmer of hope in this mess.

I hope this answers some of your questions and that you can convey, at least some, of my, and my colleagues, frustration and outrage over this situation, to people who have the power to do something about this.

Best,

Ron Baiman