Potential Errors When Performing Tests
AP Statistics
1. Type I Error Definition and Probability★☆☆☆☆⏱ 6 min
A Type I error occurs exclusively when the null hypothesis is factually true, but your sample data leads you to incorrectly reject it. The maximum allowed probability of this error is set by your pre-chosen significance level, and it is independent of sample size for a valid z-test.
Type I Error
The mistake of concluding there is statistically significant evidence for the alternative hypothesis when the null hypothesis is 100% correct for the target population.
Example:
Rejecting the null that a new vaccine has a 5% adverse reaction rate, when the true adverse reaction rate is exactly 5%.
A city health department runs a one-proportion z-test to check if the share of tap water samples with lead contamination exceeds 2%, using a significance level of 0.02. If the true contamination rate is exactly 2%, what is the probability of a Type I error?
- 1
Step 1: Confirm the null hypothesis states p = 0.02, which matches the given true population proportion.
- 2
Step 2: Recall that Type I error probability equals the pre-specified significance level α when H0 is true.
- 3
Step 3: The probability of incorrectly rejecting the true null hypothesis is 0.02.
Exam tip:
AP graders deduct all points for generic error definitions that do not reference the specific context of the test scenario provided in the question.
2. Type II Error Fundamentals★★☆☆☆⏱ 7 min
A Type II error occurs when the null hypothesis is factually false, but your sample data does not provide enough evidence to reject it. Unlike Type I error, its probability is not a fixed value for a test, and changes based on the exact true value of the population proportion you are testing against.
Type II Error
The mistake of failing to find statistically significant evidence for the alternative hypothesis, even though the null hypothesis is factually incorrect for the target population.
Example:
Failing to detect that a new app increases user retention by 15%, when the true retention lift is real.
A coffee shop tests if over 40% of customers will pay extra for plant-based milk, with n=80 and α=0.05. The true share of customers willing to pay extra is 47%. Describe what a Type II error would look like in this specific context.
- 1
Step 1: State the hypotheses: H0: p=0.4, Ha: p>0.4.
- 2
Step 2: A Type II error means you do not reject the null hypothesis, even though it is false.
- 3
Step 3: The error is concluding there is insufficient evidence that over 40% of customers will pay extra, when the true share is 47%.
3. Statistical Power and Modifying Factors★★★☆☆⏱ 8 min
Power is the core performance metric for a hypothesis test, measuring how likely you are to correctly detect a real effect if it exists. All else equal, higher power is always preferred for research and testing applications.
Increasing the significance level α raises test power
Increasing sample size reduces standard error and raises power
A larger gap between the null proportion and true effect size raises power
Lower population variability reduces sampling noise and raises power
A market researcher calculates power for a one-proportion test and finds their current power is only 0.62. Name three independent adjustments they can make to increase the test power.
- 1
Step 1: First adjustment: Increase the sample size, which reduces standard error and makes it easier to distinguish the true proportion from the null value.
- 2
Step 2: Second adjustment: Raise the significance level α, which lowers the threshold required to reject H0 and reduces β.
- 3
Step 3: Third adjustment: Target a larger minimum detectable difference between the null proportion and the true proportion, making the effect easier to spot.
4. AP Exam Error Phrasing Best Practices★★★☆☆⏱ 5 min
Confirm your understanding before moving forward
If you lower α from 0.05 to 0.01 to reduce Type I error risk, all else held constant, what happens to Type II error probability?
It increases
It decreases
It stays identical
It falls to zero
Reveal answer
It increases —Lowering alpha makes it harder to reject H0, so you are more likely to miss a real effect, raising β.
5. Common Pitfalls
Wrong move:
Defining Type I/II errors generically without referencing the specific test context
Why:
AP rubrics award zero points for decontextualized error definitions
Correct move:
Always tie the error to the given hypotheses, population proportion, and real-world outcome stated in the question.
Wrong move:
Stating Type I error probability equals 1-α
Why:
1-α is the confidence level for a confidence interval, not the Type I error rate
Correct move:
Type I error probability is exactly equal to your pre-specified significance level α.
Wrong move:
Claiming Type II error probability is a fixed universal value for a test
Why:
β changes depending on how far the true population proportion is from the null value
Correct move:
Explicitly note that β depends on the specific true alternative proportion you are evaluating.
Wrong move:
Forgetting that increasing sample size improves both α and β at the same time
Why:
Most adjustments create a tradeoff between α and β, but larger n reduces standard error to improve both
Correct move:
Identify larger sample size as the only intervention that does not force an alpha-beta tradeoff.
Wrong move:
Confusing power with the probability the null hypothesis is true
Why:
Power is a conditional probability of correctly rejecting a false H0, not a measure of null hypothesis likelihood
Correct move:
Always state power = probability of rejecting H0 given that H0 is false.
6. Quick Reference Cheatsheet
Metric | Notation | Definition | Effect on Test Power |
|---|---|---|---|
Type I Error Probability | \alpha | Reject a true null hypothesis | Higher α = Higher Power |
Type II Error Probability | \beta | Fail to reject a false null hypothesis | Lower β = Higher Power |
Statistical Power | 1-\beta | Correctly reject a false null hypothesis | Directly defined as 1 minus β |
Significance Level | \alpha | Pre-set maximum Type I error threshold | Raise to increase power |
When this came up on past exams
AI-estimated based on syllabus patterns — cross-check with official past papers for accuracy. Use only as revision-focus signals.
- 2023 · Paper 1
Free response error context identification
- 2021 · Paper 2
Multiple choice power factor question
- 2019 · Paper 1
Error consequence free response prompt
What's Next
Mastering error types and test power is critical for earning full points on the 10-point AP Statistics hypothesis testing free response question, which appears on nearly every official exam. This concept directly connects your understanding of one-proportion z-tests to real study design tradeoffs that researchers navigate daily, from clinical trial safety testing to public opinion survey analysis. You will next apply these error frameworks to two-proportion hypothesis tests, where you will evaluate error risks when comparing population proportions across two independent groups, and learn how to calculate minimum sample sizes required to hit a pre-specified power target for your inference procedures.