Welcome to DU! The truly grassroots left-of-center political community where regular people, not algorithms, drive the discussions and set the standards. Join the community: Create a free account Support DU (and get rid of ads!): Become a Star Member Latest Breaking News General Discussion The DU Lounge All Forums Issue Forums Culture Forums Alliance Forums Region Forums Support Forums Help & Search

Silent3

(15,211 posts)
Fri Oct 19, 2012, 07:04 AM Oct 2012

One in 20 polls, simply by random chance, will be outliers

Please keep that in mind when we're seeing so many polls reported from so many places.

Completely apart from sampling techniques, various likely voter models, real or imagined biases and agendas of pollsters, most polling is done based on a 95% confidence interval. That means that even when pollsters do everything else right, when they conduct good, honest polls and process the results with good math, one out of twenty poll results will fall outside of the stated margin of error.

To put that another way, when a pollster says their poll results have a margin of error of +/- 3%, that's never meant as a guarantee that actual voter sentiment is within 3% from a reported poll results, only that 19 out of 20 results will be within 3%.

Taking the confidence interval into account doesn't mean, however, that every time you see a poll you don't like you should decide that that particular poll was the 1 out of 20 outlier. Cherry picking the polls you want to believe is going to produce a more distorted view of the world than the expected level of polling error produces.

Understanding this simply means you have to be patient, look at the big picture, watch results from multiple pollsters and trends over time, and not be so quick to assume that every poll you don't like is either a sign of doom or a deliberate Republican-backed distortion.

19 replies = new reply since forum marked as read
Highlight: NoneDon't highlight anything 5 newestHighlight 5 most recent replies
One in 20 polls, simply by random chance, will be outliers (Original Post) Silent3 Oct 2012 OP
that's why you use poll averages demwing Oct 2012 #1
Use poll medians Shivering Jemmy Oct 2012 #2
But that would make it harder to manipulate. bemildred Oct 2012 #5
Credentials, please. GeorgeGist Oct 2012 #3
Credentials for what? nt bemildred Oct 2012 #4
Credentials for what? Silent3 Oct 2012 #6
Yep, and the word "confidence" is spin, what you have is probabilities, and that is all. bemildred Oct 2012 #8
Well, A Good Poll Is Supposed To Reflect Reality And Not Shape It DemocratSinceBirth Oct 2012 #9
Yeah, I just love that, it cracks me up. bemildred Oct 2012 #10
My Point Is If You Took A Poll In 1962 And Asked Folks If They Would Vote For A Gay Senate Candidate DemocratSinceBirth Oct 2012 #11
And my point is commercial polls are neither. bemildred Oct 2012 #13
The more probable a thing is, the more confidence you can have in that thing. Silent3 Oct 2012 #15
That is not correct. bemildred Oct 2012 #16
I think I'll just step back here... Silent3 Oct 2012 #17
That's what I thought. nt bemildred Oct 2012 #18
That is the definition of a95% confidence interval Shivering Jemmy Oct 2012 #7
If anyone is bored out of their mind davidpdx Oct 2012 #12
I'll bet. bemildred Oct 2012 #14
that's one in 20 within the same polling company titaniumsalute Oct 2012 #19
 

demwing

(16,916 posts)
1. that's why you use poll averages
Fri Oct 19, 2012, 07:19 AM
Oct 2012

To absorb outlier results.

Btw- Nate Silver has Obama back at
70% chance of winning. ; )

Silent3

(15,211 posts)
6. Credentials for what?
Fri Oct 19, 2012, 09:12 AM
Oct 2012

I'm talking about very basic probability and statistics.

http://en.wikipedia.org/wiki/Confidence_interval

"In applied practice, confidence intervals are typically stated at the 95% confidence level."

bemildred

(90,061 posts)
8. Yep, and the word "confidence" is spin, what you have is probabilities, and that is all.
Fri Oct 19, 2012, 09:23 AM
Oct 2012

It has no more predictive power than probabilities with dice or cards, it "estimates" certain statistics about the distribution of the population sampled, based on certain assumptions about the population sampled and the sampling methodology, but it means doodly squat about what will happen today or tomorrow.

Much of the babble on this subject is based on the unstated assumption of sequential correlation, that tomorrow will be much like today, which works great only when nothing much is going on.

So you always wind up predicting something close to the status quo, which is comforting but wrong when you really need to think about what "rough beast, its hour come round at last, Slouches towards Bethlehem to be born".

DemocratSinceBirth

(99,710 posts)
9. Well, A Good Poll Is Supposed To Reflect Reality And Not Shape It
Fri Oct 19, 2012, 09:35 AM
Oct 2012

It's up to us to change reality. When the reality changes the polls change. That's why we have our first African American president.

bemildred

(90,061 posts)
10. Yeah, I just love that, it cracks me up.
Fri Oct 19, 2012, 09:48 AM
Oct 2012

There are a few polls I pay attention to, but none of them are commercial operations for obvious reasons.

You are quite right, we can do whatever we like within the range of our powers, and our powers are great for what sort of creatures we are, and there are billions of us, too.

But we have to learn to work together, or it will continue to be that same old Hobbesian dystopia we have now.

He is not President beause of some poll, he is President because WE made him President. He, Obama, at least, seems to understand that. (But then, not born to privilege, he would know that.)

DemocratSinceBirth

(99,710 posts)
11. My Point Is If You Took A Poll In 1962 And Asked Folks If They Would Vote For A Gay Senate Candidate
Fri Oct 19, 2012, 10:00 AM
Oct 2012

What do you think the results would be?


And now we are on the verge of electing a gay or should I say a openly gay senator.


My point is a reasonably well constructed poll, free from bias, should give you some idea of what folks think on a potpourri of topics. Now, what these people think will change over time, and the polls or surveys will reflect that.


Silent3

(15,211 posts)
15. The more probable a thing is, the more confidence you can have in that thing.
Fri Oct 19, 2012, 10:05 PM
Oct 2012

How's that spin? "Confidence" does not mean "absolutely unshakable guarantee", especially in this common, long-standing statistical usage.

bemildred

(90,061 posts)
16. That is not correct.
Sat Oct 20, 2012, 10:31 AM
Oct 2012

The probability that a roll of your die will be a 6 is 1/6 th, REGARDLESS of any past series of rolls. That is what random MEANS, that there is no correlation from try to try, the past does not predict the future, AT ALL.

It is precisely that which justifies these meta-poll studies, that the polls are each separate tries, and hence (by their randomness) independent as sample data. (This is quite questionable too. Pollsters are poll junkies.)

The technical usage has a technical definition, which is correct. That does not mean the choice of the word "confidence" has the correct non-technical connotations, and as I said, in fact it's wrong, 95% probability is not 95% confidence about any try in particular, each try is ASSUMED independent, non-repeatable, and without replacement, "confidence" suggests "assurance", and that misleads, you have no such thing, you are still gambling. The entire system is built on the premise that each try is independent, that every time you ask the question, it is new.

A 95% confidence interval is a range in which the sample is expected to fall 95% of the time, the other 5% of the time it is expected to fall outside, and you are just as "confident" about the accuracy of the 5% as about the 95%, no more and no less.

At best, what polls provide is a crude heuristic that (done well) a sophisticated observer can interpret with some "confidence". Note that "confidence" is something someone has, an opinion, something one could sample too. I'm pretty "confident" at this point that Barak Obama will be the next President, barring "accidents". But I also know the present does not compel the future, it's just an educated guess based on a structured investigation.

I can go on. This is just scratching the surface, the easy stuff, we can get into Mr. Taleb's bootstrap issues with distributions, which are obvious to anyone with mathematical training and the will to look, for example. They annoyed me when I studied the subject 30 years ago, and they still do. There is nothing normal about the "normal distribution". The real world does not in fact exhibit zeroes or infinities or continuity, it is discrete, quantized, finite, and always something. Even "empty space" is far from empty. Velocity is relative (no absolute rest) and finite (less than the speed of light relative to any particular inertial frame). The uses of such notions in the physical world are all based in the laws of large numbers, probability and combinatorics, and all the physical constants that we know with great precision likewise.

Most of the more responsible pollsters seem to know that some of these issues exist, their rhetoric shows, but all the financial incentives run the other way, toward claims of certainty or some near approximation, or to show some desired "trend", a notion which you will see wanders off completely into the brush of cause and effect thinking.

There is a fundamental contradiction between assuming in one place that people form random populations you can sample randomly, by language no less, and in another, that people's behavior exhibits serial correlation (cause and effect, "confidence" about the future) over time, too. This I have seen mentioned, but I've never seen it really addressed at all.

People want confidence, assurance, security if you will, but that is wishing, we have no such thing and we need to pay attention, not be running around fat, dumb, happy, and full of ourselves because we think we know what's coming next. We don't.

I hope this at least clarifies what I base my views on.

Silent3

(15,211 posts)
17. I think I'll just step back here...
Sat Oct 20, 2012, 12:32 PM
Oct 2012

...and let you get on with whatever argument you're having with yourself or somebody else who isn't me.

davidpdx

(22,000 posts)
12. If anyone is bored out of their mind
Fri Oct 19, 2012, 10:12 AM
Oct 2012

I have access to The Psychology Research Handbook: A Guide for Graduate Students and Research Assistants. There was an interesting chapter about telephone polling.

Latest Discussions»Retired Forums»2016 Postmortem»One in 20 polls, simply b...