Evaluating samples and surveys properly

In short: A survey asks only a part (sample) and draws conclusions about everyone (population). Its validity depends on how the sample was chosen and how big it is.

In more detail: Randomly drawn samples are representative. Self-selected participants (“online poll”) are not.

In Depth

• Representativeness: Every person in the population has a known chance of being selected. Quotas and weighting compensate for distortions.
• Sample size and error: The statistical error falls with the square root of sample size. 1,000 respondents give about ±3 percentage points for a share around 50 %. Four times as many halve the error.
• Biases: Self-selection (only the interested answer), survivorship bias (only successes become visible), non-response, socially desirable answers, leading questions.
• Questions: Word neutrally, no double-barrelled questions, complete answer scales.
• Confidence interval: Gives the range in which the true value lies with high certainty (usually 95 %). “Party A 30 %, party B 28 %” is no difference at ±3 points.
• Online polls: Votes on websites and social media are not representative and can be manipulated.
• Checklist: Who was asked? How many? How were they selected? Who paid? What was the exact question?

See also: correlation and causation, media literacy