The economics of reporting polls

[DemocratSinceBirth]

The only thing happening in the Meta-Analysis is a slight, slow widening of Obama’s lead. Some of you want to know about individual polls, such as a recent Gallup national poll showing Obama ahead by only +2% (standard likely-voter model) or +6% (high-turnout model). I confess that I tend to ignore individual polls because of the statistical variability. So it didn’t occur to me to care about this particular data point. Obama is still crushing McCain, period.

But there is a lesson to be learned here: It is not in the interest of individual pollsters or media organizations for you to have the most accurate possible picture of the horserace. Here is why.

Uncertainties such as the margin of error can be reduced by taking more samples. An individual pollster can halve the margin of error by surveying 4 times as many people. It’s a square-root relationship: N samples lead to a sqrt(N)-fold reduction in uncertainty. The same is true for combining polls, with the added advantage of reducing the effects of methodological variation. Thus the value of poll-aggregation sites like this one. Meta-Analysis worked extremely well in 2004 and 2006, and is likely to do so again this year.

So why don’t more pollsters or media organizations aggregate polls? The CNN Poll of Polls is a start, but it’s an exception. Two forces encourage bad horserace reporting:

Competition among pollsters. It’s not in the interest of individual pollsters to say “average my results with the others.” It’s also not advantageous to collect a larger sample once the margin of error meets industry standards.

The hungry media beast. With news budgets on the decline, it’s costly to report real news. Why pay for investigative reporting when you can buy a poll and report the horserace? Within the area of poll reporting, market forces discourage high accuracy. For example, commissioning a survey of 4 times as many people would reduce uncertainty by a factor of two. But why pay 4 times as much for data that generate a lower likelihood of an apparent - and reportable - swing?

For these reasons, media organizations aren’t motivated to report polling results with the maximum possible statistical power. The Meta-Analysis of State Polls is pure data reduction, basically a more general version of averaging. As a result, the top-line result is very steady. This a case where a blogger-hobbyist can add value. We use the polling/media system to provide added value.


http://election.princeton.edu/2008/10/17/the-economics-of-reporting-polls/#more-1917

[phrigndumass]

1. This blogger-hobbyist is adding a K+R ... excellent overview!

Thanks for posting this here, DSB! :hi:

[DemocratSinceBirth]

2. Sam Wang Is Great...He's The Equal Of Nate Silver

He nailed 2004 in the pop vote and the EC vote just by averaging (all) the latest polls...

Where he messed up is he took those numbers and added his own expectations of how the undecideds would break... But he had two models...

[phrigndumass]

3. Completely agree, they're both great at what they do :)

Those undecided voters are sneaky, as they don't always follow the flow. Making assumptions about them is difficult, and it can always screw up a great model. I'm hoping I don't screw mine up, lol ... it would be like forgetting to drop the tires before coming in for a landing.

Good morning! :hi:

[DemocratSinceBirth]

4. It's At His Old Site

His straight model had it 286-252 and his model with undecideds breaking the way he thought had it 327-211...

[GrizzlyMan]

6. And they are both smarter than John Zogby n/t

[Undercurrent]

5. Thanks for posting this most interesting article.

Princeton Election Consortium is great site. Crammed with information. It's definitely a new found favorite for me.

[depakid]

7. "An individual pollster can halve the margin of error by surveying 4 times as many people."

That assumes, among other things, that you have a representative random sample- which none of these polling companies has.

If your sample is skewed or you've got bias, size doesn't matter- you'll get the same degree of error with 1,000 as with 10,000.

[Gregorian]

8. Thanks. Interesting stuff. I learned something.

My guess is that Obama will be more attractive to swing voters than McCain. I have a feeling the polls are going to be more accurate because of this than ever before. Don't ask why. It's my truthiness feeling.