Weighting Around
Last edited Tue Oct 9, 2012, 10:21 AM - Edit history (5)
Since everyone is noticing the demographics in the Pew poll, this is a good opportunity to talk about weighting and demographics.
One implausible percentage in the raw Pew sample is 56% female.
That's too high. 53% is more like it.
But, as labeled on the poll results, the raw sample size numbers are unweighted. All the responses by women were weighted to equal the usual 53 +/- % female advantage.
Same for race. And age. And region.
This isn't about the Pew poll, but about polls in general. The unweighted demographic sample sizes do not reflect the composition of the reported results.
Whether a poll of 1000 people has an unweighted sample of 100 African-Americans or 200 African-Americans the AA voters will be weighted to make up about 13% the sample. That's because we have a good idea of the AA population and we know that the number doesn't jump around. It isn't like there are millions more or less AA persons in the US from week to week.
If a sample has 500 men and 500 women the answers of the men will be multiplied by something like 0.97 and those of the women by something like 1.03 so that the poll sample will be as in-line as possible with what we know about the population.
Region is another handy way of adding weightedness because we know the total population of whatever states we define as "the west" so all responses from that set of states can be weighted to match their real population. A national poll will have always enough from each quadrant of the country to work with.
And, of course, cell phones. Pollsters weight cell phone versus land-line use.
Weighting the sample to match parts of reality that we know makes polling more accurate. Without weighting every poll would actually be a poll of the opinions of whoever picks up the phone the most.
Now self-reported party and ideology are different. They cannot be weighted because, unlike age, race, gender, etc., we know that they do change a lot. There is no stable "right" answer. Not all poll respondents accurately report their formal party registration, assuming they even have one. Some tend to say they are whatever party they are leaning toward voting for. And enough people do that to make it a volatile category.
Here is an Pew Research piece on the volatility of self-reported party identification written, ironically, to defend against the charge of having over-sampled Democrats in an earlier poll that had Obama up big. When a candidate does well in a poll, the matching party affiliation often also does well.
So if one is so inclined, one can argue that the Pew poll had too many conservatives. That may or may not be true.
One cannot, however, sensibly argue that it had too many southerners, or women, or African-Americans, or young people in the unweighted sample because the results aren't based on the proportions in the unweighted sample.
(I would guess that retired people answer the phone more and take polls more, rather than hanging up. It's a generational difference in both time spent at home and in how we view the telephone. If polls were not weighted to demographics they would just be polls of whoever answers the phone, a trait which is probably not randomly distributed.)
= new reply since forum marked as read
