Reply #2: Conclusion

[dzika]

So, are they saying that more data is needed from NEP before the problems with the exit polls can be better understood? AND that NEP is not going to release the necessary data due to political sensitivities?


From "III. What Happened with the Exit Polls?", last paragraph:

What must necessarily become more subjective, having eliminated the most directly testable explanations for the bias, is an attempt to explain what other sources there might be among many unobserved and unmeasured possible explanations. The NEP report concludes that the most likely source of the errors is differential response patterns by Kerry and Bush voters leaving the polls - that is, Kerry voters were more likely to agree to be interviewed while Bush voters were less likely. The authors include a simulation of the likely magnitude of the differential response rates, and then they speculate about contributing factors. These conclusions must be inferred (of necessity, since no information is available on the refusers) from the finding that the average WPE was greatest where younger and more highly educated people were interviewers, irrespective of gender.34 The analysis also suggests that interviewers hired later and who describe themselves as "somewhat" or "not very well" trained also were associated with data that produced higher average WPE's. From these analyses, the NEP leaders conclude they must pay more careful attention to interviewer recruitment, including trying to hire older interviewers, and to their training. These concerns would undoubtedly fall on any researchers' list of ways to improve their data collection if they had additional resources. Given the current information available to NEP, however, it is not possible to estimate what level of error reduction would occur for a given investment in improving any one or all of these features of their studies.

And they conclude the report:

IV. Conclusion

NEP has moved quickly, in conjunction with its partners, to change its procedures for 2008. The first decision is that no results will be provided to their sponsors until 4 p.m., about three hours later than in 2004 (New York Times, November 18).35 It will remain to be seen how effective these changes will be in eliminating leaking. The leaking of early exit poll results is not being done by the NEP staff; it comes from news executives and people working on analysis desks at news organizations who have access to the early data.

The information on the exit poll methodology is still being consumed by independent analysts, and there are now calls for the release of raw and supplementary data from sample precincts. This would include contextual data about the vote history in those areas as well as information about the interviewers.36 This is unlikely to happen, and for justifiable reasons. Such information would be too politically sensitive in that disclosure of the sample sites could subject the exit poll interviewing to manipulation by political organizations and interest groups on Election Day if the same sites are always chosen.