CBS Poll: It's a Tie!

[Teaser]

From Ruy Teixeira's weblog

Well, that is if you weight their data to conform to the 4 point Democratic party ID lead which we have good reason to believe is the underlying distribution in the voting electorate. As many have already heard, the new CBS News/New York Times poll, conducted September 12-16, gives Bush an 8 point lead (50-42) among RVs--but also gives the Republicans a 4 point edge on party ID. Reweight their data to conform to an underlying Democratic 4 point edge and you get a dead-even race, 46-46.

Say, that's familiar. Just like the 46-46 tie in the Pew Research Center poll (which gave the Democrats a 4 point edge on party ID without weighting) and the 48-48 tie in the Gallup poll (once weighted to reflect an underlying Democratic 4 point edge).

Perhaps all this is just a coincidence, but the pattern seems striking. Once you adjust for the apparent overrepresentation of Republican identifiers in some samples, the polls all seem to be saying the same thing: the race is dead-even.

I think they're right.

Excerpt quoted in full. It's a blog, and the author won't mind.

[whistle]

1. Party ID edge? What is that all about?

[buycitgo]

2. based on voters' party registration polling exit info last time

dems were ID'd as having four point registration edge in 2000, which is what the polls this time should reflect, rather than a pug advantage

[Teaser]

3. Where you been man?

It's the reason for all these large Bush leads. Repubs have been included in these samples to a greater degree than they exist in the population.

For example, the Gallup poll had 40% R and 33% D. In 2000, turnout was actually something like 4% higher for Democrats than Republicans. And in more recent polls of voter preference, this ratio has held. So Gallup oversamples Repubs. CBS poll had 36% R and 32% D, still an oversample of R's, but not as bad as Gallup. Accordingly, the Bush lead was smaller in CBS' poll.

If the data were to actually represent the most likely proportion of D to R voters, one sees that the numbers are actually tied.

[whistle]

4. Thanks, I'm glad I asked...

...so just like the employment figures that we get every month from the Department of Labor which put a death/birth factor of new fictitious totally bogus hiring numbers, the pollsters use an edge factor, which may have applied back in 2000, but doesn't mean a thing today, because the democrats have an edge now, because they proved that back in 2000, when everyone thought that the republicans were going to have the edge but they didn't so Jeb Bush decided to cover the odds and cheated in the Florida election which created a big stink so that the real edge came from the Supreme Court Justices who were beholden to George H.W Bush the first and the military industrial complex! I get it now.

[Teaser]

6. not quite, but close.

The Dem's had the edge in 2000. Studies indicate they still have it.
Yet these pollsters are not throwing out polls that have outrageous distributions of Repubs in them.

If you do a random sample, and you find way, way too many R's in it (or D's for that matter) it should be tossed as you have an unrepresentative sample (or at worst, acknowledged as an unlikely sample). But because these organizations want to crank out the polls, they just quick do a poll and don't check the internals against real world data. The pressure is to do a poll because 1 poll = many newspaper stories that a reporter doesn't really have to write. You can just "phone in" a story about a poll. Interview a few pollsters, roll out one result a day for a week, and voila, you can kick back the rest of the time.

[whistle]

7. Listen, there is something called screener questions....

...one of which determines the party affiliation and perhaps also the registration status and likelihood that the respondent will vote on Nov 2nd. So your are saying that a polling company, because they are such cheap bastards and won't discard an interview even if it exceeds the quota margins of a known balance of democrats and republicans, will blindly take the results of a poll and suggest that as being valid? I am not doubting you understand, but that certainly is not the common sense way to conduct random samples in known populations. That is sampling error big time! No kind of adjustment can be made to such samples no matter what the rational is. IMHO.

[Teaser]

11. That's not the only reason it happens, but it does happen

Check out the Columbia Journalism Review article on why reporters love polls:

http://www.campaigndesk.org/archives/000922.asp

[whistle]

14. That helps, thanks

[sonicx]

15. no, you don't adjust the samples

you weight the results by party ID based on an guess for party turnout. If you do a poll and you get more repubs respondents than dems ones, you weight the dems up and weight the repubs down. For the last 3 elections, we have outvoted them and there's little chance of that changing this year. that's the problem with gallup, time, newsweek, and cbs polls. They doesn't represent how the parties are going to vote in Nov. NOWAY will republicans outvote us.

[whistle]

39. I'm a little slow when it comes to conceptualizing number....

...proportions like you are describing. As memory serves me, in the 2000 presidential election, ignoring the votes which Gore failed to receive because of fraud, trickery, miscounting or Supreme Court intervention the popular vote counts were something in the nature of:

Gore....50.9 million
Bush....50.4 million

Our man would have won Florida and the electoral college vote and been president had he been allowed 517 or 527 votes that the Supreme Court denied. But let's just set that aside and use the popular vote advantage that Gore got which was approximately a half million more votes than Bush got. With me so far?

Now, we get a polling company that does its survey for ease of calculation here, on 1,000 completed interviews of registered voters of which 900 say they are likely to vote on November 2, 2004. Among these 1,000 respondents the idea is to get a proportionate random sample of republicans, democrats and independents to represent the distribution of these three groups of voters across the country.

The polling company has the choice to do a poll perfect (which takes time and money) or fast (which meets the needs to get their name published ahead of all others), and since they have limited time, staff and resources, the pollsters do the best that they can. I'm trying to be fair here, but as a consumer I also want to know and understand what goes on with the numbers and the the process to maintain credibility.

Ah event occurs, for sack of my illustration here, it's the recent intensity of resistance fighting in Iraq over the later part of this week, so I as the pollster get my team mobilized to telephone, qualify, screen and interview the 1,000 respondents and I use all of the acceptable procedures and standards of polling and do not cut corners or cheat. I randomize, qualify, monitor my interviewers etc. and in one evening and a day of intense calling I get my 1,000 completes which are immediately processed and reported in the AP before 4:00PM today so it gets the 6:30 national news, the Sunday papers and all of the Sunday morning political programs like Meet the Press and so forth.

What if anything, does the polling company have to do to the results to make them credible when the reporter says: " A new poll just out by Whistle Research says that John Kerry has inched ahead of President Bush 48% to 46% in a poll released just two hours ago,...blah, blah, blah."

The results to the question: "If the election for president of the U.S. were held today who would you vote for:

George W. Bush (rep)...
John F Kerry (dem).....
Ralph Nader (reform)...
Undecided..............

Where does the weighting come in? What legitimate right does the polling company have to use a weighting? What are weightings not disclosed if they in fact have a major impact on the results?

[featherman]

5. For party ID edge see this table from the NES

This is a more sophisticated seven point scale rather than a three (DEM, GOP, IND) that the polls use in summaries but amounts to the same thing.
Probably of most interest is there a 2-3% more INDs that lean DEM than there are who lean GOP.

[whistle]

8. Did you forget the link???

[Oilwellian]

9. Why You Should Ignore The Gallup Poll This Morning

In both polls, Gallup oversamples greatly for the GOP, and undersamples for the Democrats. Worse yet, Gallup just confirmed for me that this is the same sampling methodology they have been using this whole election season, for all their national and state polls. Gallup says that "This (the breakdown between Reeps and Dems) was not a constant. It can differ slightly between surveys" in response to my latest email. Slightly? Does that mean that in all of these national and state polls we have seen from Gallup that they have "slightly" varied between 36%-40% GOP and 32%-36% Democrat? I already know from an email I got from Gallup earlier in the week that in their suspicious Wisconsin and Minnesota polls they seemingly oversampled for the GOP and undersampled for the Dems. For example in Wisconsin, in which they show Bush now with a healthy lead, Gallup used a sample comprised of 38% GOP and 32% Democratic likely voters. In Minnesota where Gallup shows Bush gaining a small lead, their sample reflects a composition of 36% GOP and 34% Democrat likely voters. How realistic is either breakdown in those states on Election Day?

http://www.theleftcoaster.com/archives/002806.html

[whistle]

10. Wasn't George Gallup the pollster who totally miscalled....

...the Dewey/Truman contest in 1948?

[Teaser]

12. Yeah, but he was good after that..

Today's Gallup really isn't the same organization as that Gallup. It's a bunch of people who bought the rights to his name after he died. Continuity in name only.

[whistle]

13. His art and craft must have died with him then, pity.

[shockingelk]

16. I believe many people are confused

The CBS poll is of registered voters not their own definition of "likely voters". They just take a random sample and the R/D/I % is the result of a question asked:

"In most CBS News Polls, Democrats outnumber Republicans among registered voters ... In this poll, where the Republicans hold a significant lead in voter preference and more voters hold negative views about Democratic nominee John Kerry, when voters are asked about their partisan identification at the end of the questionnaire, more identify themselves as Republicans. 36 percent say they are Republican, 32 percent Democrats. The percentage that identifies themselves as Democrats in this poll is lower than it has been in CBS News Polls conducted earlier in the year."

That's an entirely different thing than Gallup determining a sample representative of "likely voters" by including more people who say they're Republican than say they're Democrats.

[DaveinMD]

17. regardless

the sample just isn't representative. Any poll that isn't weighted properly is useless. This happens because the media polls are done on the cheap. Polls paid for by campaigns are more expensive and therefore care is taken to model the actual likely vote. This leads to more accurate results.

[shockingelk]

18. That's not what I was getting at

CBS is not claiming that their sample is representative, but random. It has a margin of error and the answer to "party ID" has a margin of error and confidence level.

"In a nutshell, we choose the people we interview completely at random. We do not choose our respondents based on their age, race, political philosophy, or any other characteristic." http://www.cbsnews.com/stories/2001/07/02/opinion/main299401.shtml

In my opinion, recently registered voters, who are difficult to impossible to poll, will change the outcome of the "Nov 2 poll" drastically, and because there's a lot of anger at Jr - even reaching into pop culture - we'll see results quite different than any of the polls we're seeing.

[DaveinMD]

19. OK

I think a poll conducted like the Pew poll was is likely to be accurate. However, as with all close elections, it will be decided by turnout.

[shockingelk]

21. Yes, that too

The fact that there is not a gubernatorial or senate race in MN, for example, makes it less likely for Repubs to show up on Nov 2. Those with reservations about Jr who would vote for him if they showed up and just going to sit at home and watch Jr lose MN.

Compare to GA - Georgians have a black woman running for Senate. That's going to energize voters who would otherwise be apathetic because "it's a rich white man's world, my vote isn't worth anything."

[ferg]

20. the sample is not statistically random

That's the problem. The 3% margin of error with 95% confidence stuff assumes a random sampling, like if you flip a coin or draw cards from a deck.

But phone call poll responses are not statistically random, because people don't answer the phone, use cell phones, screen calls, etc.

It's not like flipping a coin. A coin can't decide not to tell you whether it landed heads or tails.

So the statement "we choose the people we interview completely at random" is false. They choose the land-line phone numbers at random. They do not choose the people who respond at random.

[shockingelk]

23. I believe you're still mixing up concepts

Here - pretend the main aim of the poll was to determine if people identify as Republicans or Democrats.

You randomly sample people from voter lists and ask "are you a Republican, Democrat or other/independent"?

That's what the CBS poll does.

On the other hand, Gallup thinks "OK, more people that ID as Republicans are going to turn out on Nov 2, so we should weight our random sample towards people who ID as Republican."

[sonicx]

25. no no no

gallup doesn't weight their polls in either 'registered' or 'likely voters'.

when they do 'likely voters,' they just ask questions like 'have you voted before' and 'how interested in the election are you'

[shockingelk]

27. They weight their results

Gallup takes into consideration party ID as well as voting history. That's why the Gallup results differ slightly from the Gallup/CNN/USA Today results. It's different methodologies applied to the same data.

[sonicx]

29. huh?

i emailed gallup and they say they don't weight polls.

also, Gallup and CNN/USA/Gallup are the same poll.

[shockingelk]

31. A sample is different than ...

A sample is different than a methodology used to analyze that sample.

Weighting results is different than weighting a sample.

The very act of determining a "likely voter" is weighting the results according to the definition of "likely voter" you use.

As per Gallup and Gallup/CNN/USA Today - I do believe they use the same data, but apply different methodologies to analyzing it. They are reported separately ande do indeed show different results:

Gallup results listed here now: http://www.pollingreport.com/wh04gen.htm
Gallup/CNN/USA Today listed here: http://www.pollingreport.com/wh04gen2.htm

[sonicx]

33. oh, i never noticed that.

thanks. :)

the likely voter model: i've never heard of gallup using pary id for this. i don't know the questions they ask exactly, but i thought it was just along the lines of 'have you voted before' and stuff like that. then they rank people's 'likeliness' and take the top 55% and that's the sample likely voter sample.

Have you even seen Gallup's 'likely voter' questions before?

[shockingelk]

34. There's an article in a "subscription only" area of gallup.com

Haven't seen it. But this article gives a good idea; http://www.emergingdemocraticmajorityweblog.com/donkeyrising/archives/000635.php

[sonicx]

28. also, we know from 2000's polling...

that polls that used party weights were more accurate.

[ferg]

30. no, the issue has nothing to do with likely voters

You randomly sample people from voter lists and ask "are you a Republican, Democrat or other/independent"? That's what the CBS poll does.

No, that's not what the CBS poll does. That's what the CBS poll would like to do, but fails because the real world is complicated. The point is that no actual telephone poll can do that in truly statistically random sense.

Here's what really happens:

The poll randomly samples phone numbers (this part is truly random). 50% are either not home or refuse to answer. In addition, 10% of the population doesn't have a land-line phone number. (These two are where the statistics goes to hell.)

Of the 40% who answer polls on land-lines, 40% say they're Republicans, 30% say they're Democrats, and 30% say they're Independent.

The problem is that the 40% who answer on land lines aren't random. More Republicans answer than Democrats. So any poll reponse based on the "random" responses are skewed toward the Republicans.

In other words, the percentage of Republicans in the US is not the same as the percentage of people who answer phone calls and say they're Republicans. P(Repub) <> P(Repub|Answer)

Suppose you conducted a telephone poll in 1904 to predict the presidential race by dialing random numbers. It would be utterly worthless because almost no one had telephones, and the people who did were all rich.

[shockingelk]

35. What are you trying to suggest?

That polling is worthless?

[ContinentalOp]

36. And who doesn't answer the phone to take these polls?

Lets see, you have young people who increasingly use cell phones as their only phone number. You have any household where everyone works and in particular, families where people might work more than one job and not be home long enough to talk to pollsters. And you have the technically savvy who either screen their calls or simply hang up when they realize it's not someone they know.

So who is left to answer the phone when these pollsters come calling? Bored, lonely republicans and probably a disproportiate number of old people. The same people who fall prey to telemarketers and con men. Not exactly the Democratic base and not exactly a random sampling.

[sonicx]

22. problem is...

there are probably more dems than repubs registered to vote. I'd be shocked if they were beating us in registering people.

First, there are more dems in the US period. Second, we are pissed and really want bush out. third, we are pissed and really want bush out!!! :)

[shockingelk]

24. Many states don't register voters by party

Like TX. You can't register as a Republican or Democrat. Anybody who says they're a "registered X" in TX is confused.

They only thing you can do is ask your sample, "Are you Democrat, Republican or neither?"

[sonicx]

26. i know, but when they determine party turnout for the election

i don't think they use the state numbers. just exit polls.

and we determine the overall party breakdown in the US by polls too.

[shockingelk]

32. I think we're saying the same thing

But in different ways.

[physaf]

41. there have always been more dems than pubs

this will be borne out in time.

[Zynx]

38. That doesn't mean the poll is not incorrect.

[sofa king]

37. I hope Bush2Barsoom sees this. Pollsters don't call cell phones!

He's a brand-new poster here at DU. I can't find the thread in which he said it, but he pointed out something almost shockingly elementary.

Telephone pollers are prevented by federal law from calling cell phones.

How many of you have dropped your landline entirely in the past four years? Your opinion isn't going to be asked.

The potential ramifications of this recent demographic change are huge--and not historically unprecedented, either.





So here's a special shoutout to Bush2Barsoom, who has a great name and excellent insight. Wellcome to DU!

(Edit: I see that ferg is all over this, too. Excellent explanation!)

[tedzbear]

40. Nobody believes CBS anymore...

...after the forged memos fiasco. Today the Los Angeles Times (Calendar section) said CBS needs to publicly clean house or lose all credibility as a respectable news source.