Potential Problems with Sampling
AP StatisticsΒ· Unit 3: Collecting Data, Topic 3.2Β· 12 min read
1. Core Definition of Sampling Biasβ β ββββ± 3 min
Sampling Bias
A systematic, repeated skew in sampling results that arises from a flaw in the sampling design, not from random chance variation between samples.
A teacher asks the first 10 students who arrive to class what their current grade in the course is, and calculates an average of 92%. Is this result biased?
- 1
Step 1: Identify the target population: All students in the class.
- 2
Step 2: Note that students who arrive early to class are more likely to have higher grades than students who arrive late.
- 3
Step 3: Confirm the sample systematically overrepresents high-performing students, so the calculated average is biased upwards from the true class average.
Test your basic understanding of sampling bias:
Which of the following is a characteristic of sampling bias?
It is caused by random chance variation
It consistently skews results in one direction
It is impossible to avoid with any sampling method
It only affects samples smaller than 100 people
Reveal answer
It consistently skews results in one direction βBias is a systematic skew, not random variation, and can be avoided with proper sampling design.
2. Four Key Types of Nonsampling Biasβ β β βββ± 4 min
Undercoverage: Entire groups of the population are excluded from the sampling frame
Nonresponse Bias: Selected participants do not return the survey, and their views differ from respondents
Voluntary Response Bias: Participants choose to opt in, overrepresenting strong opinions
Response Bias: Survey questions or interviewer behavior pushes participants to give false answers
A local TV station posts a poll on its Facebook page asking viewers if they support a new public transit tax, and gets 12,000 responses, 78% of which say no. What type of bias is this?
- 1
Step 1: Note that viewers are choosing to respond to the poll on their own, rather than being randomly selected.
- 2
Step 2: Recognize that people who strongly oppose the tax are far more likely to take the time to respond than people who are neutral.
- 3
Step 3: Classify this as voluntary response bias, which makes the 78% no result much higher than the true community opinion.
3. Sampling Error vs Nonsampling Errorβ β β βββ± 2 min
Many students confuse sampling error with bias, but these two concepts are completely distinct:
Sampling Error
Natural random variation between results from different random samples of the same population. It decreases as sample size increases, and is not a flaw in design.
+ Pros: Expected, predictable, and measurable with confidence intervals
Nonsampling Error (Bias)
Systematic skew from a flaw in survey design, sampling frame, or response process. It does not decrease as sample size increases.
β Cons: Cannot be measured from sample data alone, and makes results ungeneralizable
Two random samples of 50 students from the same high school find average GPAs of 3.2 and 3.3. Is this difference caused by sampling error or bias?
- 1
Step 1: Confirm both samples are randomly selected from the full student population.
- 2
Step 2: Note that small differences between random samples are expected, not a sign of a flawed design.
- 3
Step 3: Conclude this difference is normal sampling error, not bias.
4. Evaluating Sampling Flaws for AP Free Responseβ β β β ββ± 3 min
Bias Type | Required Explanation for Full Points |
|---|---|
Undercoverage | Name the excluded group and explain how their opinions differ from the sampled group |
Nonresponse Bias | Note that non-respondents would have answered differently than people who returned the survey |
Voluntary Response Bias | Explain that people with extreme opinions are far more likely to respond than neutral people |
Response Bias | Show that the question wording or interviewer would push participants to lie about their answer |
A restaurant leaves customer satisfaction surveys on every table, and only 8% of diners fill them out. Write a full AP-level justification of the sampling flaw.
- 1
Step 1: Name the bias: This is nonresponse bias.
- 2
Step 2: Link to scenario: The 92% of diners who do not fill out the survey are far more likely to have had a neutral dining experience than the small share of people who had an extremely good or extremely bad meal.
- 3
Step 3: State the skew direction: This means the survey results will overrepresent extreme positive and negative reviews, and will not reflect the average diner's experience.
Exam tip:
To earn full points on AP bias FRQs, you must name the bias, link it to the specific scenario, and explain the direction of the skew, not just state 'the sample is bad'.
5. Common Pitfalls
Wrong move:
Calling any bad survey result 'bias' without naming the specific type
Why:
AP rubrics require explicit identification of the bias category to earn full points
Correct move:
Name the exact bias (e.g. undercoverage) and explain how it systematically skews results
Wrong move:
Confusing sampling variability with bias
Why:
Natural variation between samples is not a flaw, it is expected in random sampling
Correct move:
Note that sampling error describes expected variation, while bias is a systematic, repeated skew
Wrong move:
Claiming a large sample size fixes all sampling bias
Why:
A huge biased sample will still produce systematically wrong results
Correct move:
State that large samples only reduce random sampling error, not existing bias from bad sampling frames
Wrong move:
Mixing up response bias and nonresponse bias
Why:
Nonresponse bias occurs when selected participants refuse to answer, while response bias occurs when participants lie or answer incorrectly
Correct move:
Distinguish the two by checking if the flaw comes from missing data vs inaccurate reported data
Wrong move:
Stating that a convenience sample is 'unbiased because it is easy to collect'
Why:
Convenience samples almost always exclude large segments of the target population, creating undercoverage
Correct move:
Explicitly note that convenience samples cannot produce results generalizable to the full population
6. Quick Reference Cheatsheet
Bias Type | Definition | Common Exam Scenario |
|---|---|---|
Undercoverage | Some members of the population are excluded from the sampling frame | A mall survey excludes people who never visit the mall |
Nonresponse Bias | Selected participants do not complete the survey, differing systematically from respondents | 1% return rate on a mailed tax policy survey |
Voluntary Response Bias | Participants self-select to respond, overrepresenting strong opinions | A radio call-in poll about local public transit |
Response Bias | Survey questions push participants to give inaccurate answers | Asking teens about drug use in front of their parents |
When this came up on past exams
AI-estimated based on syllabus patterns β cross-check with official past papers for accuracy. Use only as revision-focus signals.
- 2023 Β· Paper 1
Sampling bias identification FRQ
- 2021 Β· Paper 2
Bias justification and explanation
- 2019 Β· Paper 1
Sampling flaw analysis scenario
What's Next
Mastering sampling flaws is a critical prerequisite for scoring full points on AP Statistics free response questions focused on data collection, which appear on nearly every AP exam. This skill directly feeds into your ability to critique experimental designs, identify confounding variables, and justify whether study results can be generalized to a larger population or used to prove causal relationships. Next, you will build on this foundation to learn how to design completely randomized experiments, control for lurking variables, and distinguish between observational studies and controlled experiments, a topic that is tested alongside sampling problems in roughly 30% of all AP Stats FRQ sections. Practice applying your new bias identification skills to the linked practice problem sets to lock in your points for exam day.