Reply #3: 538: Nonrandomness in Research 2000's Presidential Tracking Polls

[mod mom]

6.30.2010
Nonrandomness in Research 2000's Presidential Tracking Polls
by Nate Silver @ 7:00 AM
Share This Content
This is one of the things that I pointed out to Mark Blumenthal had been odd-seeming about Research 2000's polling:

Likewise, take a look at their Presidential tracking numbers
from 2008 (http://www.dailykos.com/dailypoll/2008/11/4).
They published their daily results in addition to their
three-day rolling average ... and the daily results were
remarkably consistent from day to day. At no point, for
instance, in the two months that they published daily results
did Obama's vote share fluctuate by more than a net of 2
points from day to day (to reiterate, this is for the daily
results (n=~360) and not the rolling average). That just
seems extremely unlikely -- there should be more noise than
that.
Let's put some flesh on them bones.

In 2008, Research 2000 published the results of its daily samples in its Presidential tracking poll. To clarify, this means that if they had a tracking poll that ran from Wednesday through Friday, they'd tell you what the individual results were for Wednesday, Thursday and Friday respectively, in addition to the aggregated numbers. I for one appreciated this and actually used the daily numbers rather than the multi-day tracking averages in our forecasting models.

A lot of pollsters would have been reluctant to do this because the sample sizes were quite small -- on average, about 360 persons for each daily sample -- and presumably would have revealed rather striking variation from day to day simply due to sampling error. The margin of error on a sample size of 360 is +/- 5.2 points, so it would be fairly normal for Barack Obama's numbers to careen (for example) from 54 points one day, to 48 points the next, to 52 the day afterward.

But in fact, this didn't happen. In fact, their daily samples showed barely any movement at all. In the 55 days of their tracking poll, Barack Obama's figure never increased by more than 2 points, nor declined by more than 2 points.

-snip

http://www.fivethirtyeight.com/2010/06/nonrandomness-in-research-2000s.html