Democratic Underground Latest Greatest Lobby Journals Search Options Help Login
Google

Nate Silver: My Own Suspicions About Research 2000

Printer-friendly format Printer-friendly format
Printer-friendly format Email this thread to a friend
Printer-friendly format Bookmark this thread
This topic is archived.
Home » Discuss » Archives » General Discussion (1/22-2007 thru 12/14/2010) Donate to DU
 
mod mom Donating Member (1000+ posts) Send PM | Profile | Ignore Wed Jun-30-10 09:40 AM
Original message
Nate Silver: My Own Suspicions About Research 2000
My Own Suspicions About Research 2000

My Own Suspicions About Research 2000
by Nate Silver @ 3:45 PM
Share This Content
Although I expect to proceed fairly carefully with respect to Research 2000, which Daily Kos will be suing for alleged fraud, I have suggested here and to at least one reporter that I had my own suspicions about Research 2000 which paralleled some of the findings in the study by Mark Grebner, Michael Weissman, and Jonathan Weissman. I want to be a bit more explicit about what I mean by that.

This is a copy of two e-mails that I sent to Mark Blumenthal of Pollster.com in the wee hours of the morning on February 4th. Like the examples in the Grebner study, they point toward cases in which Research 2000's data appeared to be other-than-random (although, as I declaim in the e-mails, not necessarily triggered by fraud).

from Nate Silver
to Mark Blumenthal
Mark Blumenthal
date Thu, Feb 4, 2010 at 4:17 AM
subject Research 2000 weirdness
mailed-by gmail.com

Mark,

Not to sound too conspiratorial, but to be honest I'm getting
a little bit suspicious about Research 2000, or at least the
polling they've conducted for Markos over the past two years.
Do you know those guys at all?

I'll keep this pretty brief. In part it's because of the
occasionally really weird result they turn out -- for instance,
they had only 27 percent of Republicans or something in favor
of gays in the military whereas Gallup and ABC/Post have had
those numbers in the 60s. There are two or three other examples
like this I could point to. For another, their contact
information and web presence is pretty sketchy relative to that
of other pollsters and there's not a lot of detail about the
scope of their operations.

But mainly, it's that that their data feels way too clean for
me. Take a look at the attached chart, for example: these are
the age breakdowns in the Democratic vote share for the last
20 contests surveyed by R2K and PPP, respectively. The age
breakdowns in Research 2000's numbers are almost always close
to "perfect" -- in 20 out of 20 cases, for instance, the
Democrat gets a lower vote share from among 30-44 year olds
than among 18-29 year olds. PPP's data, on the other hand,
is *much* messier -- which is what I think we should expect
when comparing small subsamples, particularly subsamples of
lots of different races that are subject to different
demographic patterns.

Likewise, take a look at their Presidential tracking numbers
from 2008 (http://www.dailykos.com/dailypoll/2008/11/4 ).
They published their daily results in addition to their
three-day rolling average ... and the daily results were
remarkably consistent from day to day. At no point, for
instance, in the two months that they published daily results
did Obama's vote share fluctuate by more than a net of 2
points from day to day (to reiterate, this is for the daily
results (n=~360) and not the rolling average). That just
seems extremely unlikely -- there should be more noise than
that.

Maybe/probably they're just using some weighting procedures
that smooth out a lot of the noise that you would ordinarily
expect to see, but it all looks pretty weird to me.

Anyway, let me know your thoughts. If you think there's
enough smoke there, my next step would probably be to bring
this to Markos's attention.

Nate

http://www.fivethirtyeight.com/2010/06/my-own-suspicion...

Getting interesting!
Printer Friendly | Permalink |  | Top
mod mom Donating Member (1000+ posts) Send PM | Profile | Ignore Wed Jun-30-10 09:53 AM
Response to Original message
1. Josh Marshall decides to purge R2K from TPMPolltracker:
Ed.Note on R2K
Josh Marshall | June 29, 2010, 3:56PM
As you've seen we're reported on Markos's charges that at least some of the data provided by Research 2000 was fraudulent and the statistical analysis on which the charges are based. I've read Nate Silver's approving review of the statistical analysis. And I've spoken with Professor Charles Franklin, who is our outside consultant on polling questions.

Based on this review I've decided that there is more than enough question hanging over R2K's data to warrant our purging the firm's numbers from the TPMPolltracker database. That has now been done.

I should add that this is not and we're not in a position to pronounce a verdict of guilty on R2K's data. But that's not the standard. We can only include data the integrity of which is beyond reasonable question. And on that basis we've removed it.

Should subsequent developments remove the cloud of suspicion over R2K's data we would, of course, re-add it to our database.

http://www.talkingpointsmemo.com/archives/2010/06/ednot...
Printer Friendly | Permalink |  | Top
 
sufrommich Donating Member (1000+ posts) Send PM | Profile | Ignore Wed Jun-30-10 09:56 AM
Response to Reply #1
2. They are toast as a polling organization. I expect we'll see
them exit the polling field quickly.
Printer Friendly | Permalink |  | Top
 
mod mom Donating Member (1000+ posts) Send PM | Profile | Ignore Wed Jun-30-10 09:57 AM
Response to Original message
3. 538: Nonrandomness in Research 2000's Presidential Tracking Polls
6.30.2010
Nonrandomness in Research 2000's Presidential Tracking Polls
by Nate Silver @ 7:00 AM
Share This Content
This is one of the things that I pointed out to Mark Blumenthal had been odd-seeming about Research 2000's polling:

Likewise, take a look at their Presidential tracking numbers
from 2008 (http://www.dailykos.com/dailypoll/2008/11/4 ).
They published their daily results in addition to their
three-day rolling average ... and the daily results were
remarkably consistent from day to day. At no point, for
instance, in the two months that they published daily results
did Obama's vote share fluctuate by more than a net of 2
points from day to day (to reiterate, this is for the daily
results (n=~360) and not the rolling average). That just
seems extremely unlikely -- there should be more noise than
that.
Let's put some flesh on them bones.

In 2008, Research 2000 published the results of its daily samples in its Presidential tracking poll. To clarify, this means that if they had a tracking poll that ran from Wednesday through Friday, they'd tell you what the individual results were for Wednesday, Thursday and Friday respectively, in addition to the aggregated numbers. I for one appreciated this and actually used the daily numbers rather than the multi-day tracking averages in our forecasting models.

A lot of pollsters would have been reluctant to do this because the sample sizes were quite small -- on average, about 360 persons for each daily sample -- and presumably would have revealed rather striking variation from day to day simply due to sampling error. The margin of error on a sample size of 360 is +/- 5.2 points, so it would be fairly normal for Barack Obama's numbers to careen (for example) from 54 points one day, to 48 points the next, to 52 the day afterward.

But in fact, this didn't happen. In fact, their daily samples showed barely any movement at all. In the 55 days of their tracking poll, Barack Obama's figure never increased by more than 2 points, nor declined by more than 2 points.

-snip

http://www.fivethirtyeight.com/2010/06/nonrandomness-in...
Printer Friendly | Permalink |  | Top
 
gristy Donating Member (1000+ posts) Send PM | Profile | Ignore Wed Jun-30-10 10:03 AM
Response to Original message
4. I'm surprised this email is so lacking in quantitative analysis
Edited on Wed Jun-30-10 10:05 AM by gristy
At no point, for instance, in the two months that they published daily results did Obama's vote share fluctuate by more than a net of 2 points from day to day (to reiterate, this is for the daily results (n=~360) and not the rolling average). That just seems extremely unlikely -- there should be more noise than that.


Very uncharacteristic of Silver. I would have expected for him to calculate the probability of this happening by chance. Pretty straightforward.
Printer Friendly | Permalink |  | Top
 
BzaDem Donating Member (1000+ posts) Send PM | Profile | Ignore Wed Jun-30-10 10:07 AM
Response to Reply #4
5. Why would that email need more quantitative analysis?
He wasn't trying to make a case in public. Two polling experts were talking to each other. They don't need to convince each other of basic premises. Of course, Nate could have done the actual statistical calculation to show that the chances of not fluctuating more than 2 points per day is less than 1 in 1000000 or whatever. (He in fact did do such calculations in the current post he made.) But why would he have to do that calculation if he is just privately trying to convince someone who already knows the math?
Printer Friendly | Permalink |  | Top
 
mod mom Donating Member (1000+ posts) Send PM | Profile | Ignore Wed Jun-30-10 10:10 AM
Response to Reply #4
6. See thread # 3
Printer Friendly | Permalink |  | Top
 
Robeson Donating Member (1000+ posts) Send PM | Profile | Ignore Wed Jun-30-10 11:22 AM
Response to Original message
7. kick.
Printer Friendly | Permalink |  | Top
 
bleever Donating Member (1000+ posts) Send PM | Profile | Ignore Wed Jun-30-10 12:03 PM
Response to Original message
8. Interesting.
K&R.
Printer Friendly | Permalink |  | Top
 
blogslut Donating Member (1000+ posts) Send PM | Profile | Ignore Wed Jun-30-10 12:27 PM
Response to Original message
9. kick
All this number crunching makes my head hurt but the story is quite intriguing.
Printer Friendly | Permalink |  | Top
 
Las Vegas Mixx Donating Member (57 posts) Send PM | Profile | Ignore Thu Jul-01-10 07:34 AM
Response to Original message
10. Starting in 2009? Suspicions about Research 2000 and Rasmussen
Edited on Thu Jul-01-10 08:12 AM by Las Vegas Mixx
http://eforum.reviewjournal.com/lv/showthread.php?t=425...
http://www.fivethirtyeight.com/2010/06/pollster-ratings...

Daily Kos is suspicious of Research 2000 polling since 2009. That is same starting time frame that Nate Silver of Fivethirtyeight.com finds questionable about Rasmussen's polls.

Nate Silver's Pollster Ratings (June 2008) had both Rasmussen at #3 and Research 2000 at #8 well regarded among the top 32 pollsters studied.

Silver's June 2010 Pollster Ratings for both of these pollsters dropped substantially, Rasmussen dropped all the way down to #15 and Research 2000 landed near the bottom at #38 of 42.

Since 2009, it would appear that both Rasmussen and Research 2000 have been "producing" (manipulating) results to favor their client's built-in bias. "Tell them what they want to hear."

Daily Kos has the integrity to question it's pollster's "results" bias, while the GOP has not shown any inclination to question Rasmussen's own more and more obvious bias. To the contrary, the GOP and conservative groups have instead rewarded Rasmussen's new 'activist' bias with exclusivity contracts and right-wing celebrity status for their new unlikely cult hero, Scott Rasmussen.




Printer Friendly | Permalink |  | Top
 
Las Vegas Mixx Donating Member (57 posts) Send PM | Profile | Ignore Thu Jul-01-10 08:13 AM
Response to Reply #10
11. Nate Silver Pollster Ratings: 2010 vs 2008
Printer Friendly | Permalink |  | Top
 
Las Vegas Mixx Donating Member (57 posts) Send PM | Profile | Ignore Thu Jul-01-10 08:50 AM
Response to Reply #11
12. Caveat from competitor, Pollster.com (Mark Blumenthal)
http://www.pollster.com/blogs/rating_pollster_accuracy_...

Blumenthal:

Second, even if you take the scores at face value, the final scores that Silver reports vary little from pollster to pollster. They provide little real differentiation among most of the pollsters on the list. What is the range of uncertainty, or if you will, the "margin of error" associated with the various scores? Silver told Markos Moulitsas that "the absolute difference in the pollster ratings is not very great. Most of the time, there is no difference at all."

Nate Silver:

We can say with a fair amount of confidence that the pollsters at the top dozen or so positions in the chart are skilled, and the bottom dozen or so are unskilled i.e. "bad". Beyond that, I don't think people should be sweating every detail down to the tenth-of-a-point level.

That information implies, as our commenter jme put it yesterday that "his model is really only useful for classifying pollsters into three groups: Probably good, probably bad and everyone else." And that assumes that this confidence is based on an actual computation of standard errors for the PIE scores.






Printer Friendly | Permalink |  | Top
 
DU AdBot (1000+ posts) Click to send private message to this author Click to view 
this author's profile Click to add 
this author to your buddy list Click to add 
this author to your Ignore list Tue Sep 23rd 2014, 11:44 PM
Response to Original message
Advertisements [?]
 Top

Home » Discuss » Archives » General Discussion (1/22-2007 thru 12/14/2010) Donate to DU

Powered by DCForum+ Version 1.1 Copyright 1997-2002 DCScripts.com
Software has been extensively modified by the DU administrators


Important Notices: By participating on this discussion board, visitors agree to abide by the rules outlined on our Rules page. Messages posted on the Democratic Underground Discussion Forums are the opinions of the individuals who post them, and do not necessarily represent the opinions of Democratic Underground, LLC.

Home  |  Discussion Forums  |  Journals |  Store  |  Donate

About DU  |  Contact Us  |  Privacy Policy

Got a message for Democratic Underground? Click here to send us a message.

© 2001 - 2011 Democratic Underground, LLC