Experimental Design
AP StatisticsΒ· 12 min read
1. Four Core Principles of Experimental Designβ β β βββ± 3 min
Experimental Unit
The individual, object, or subject to which a treatment is assigned in an experiment
Example:
A high school student testing a new study app, a potted plant receiving a new fertilizer
Control: Include at least one comparison group (control or alternative treatment) to isolate the effect of the active treatment
Randomization: Use random assignment to distribute known and unknown confounding variables evenly across all treatment groups
Replication: Assign enough experimental units to each treatment group to reduce random sampling variability
Variability Reduction: Account for known nuisance variables that impact the response to reduce unneeded noise in results
A researcher wants to test if a new energy drink improves exam scores for college students. List all four core principles you would include in this experiment.
- 1
Step 1 (Control): Include a control group that receives an identical-tasting placebo drink with no active ingredients to compare against the energy drink group.
- 2
Step 2 (Randomization): Use a random number generator to assign 100 volunteer students to either the energy drink group or the placebo group, 50 students per group.
- 3
Step 3 (Replication): Use 50 students per group rather than 2 per group to ensure average score differences are not caused by random luck.
- 4
Step 4 (Variability Reduction): Account for pre-existing student GPA as a known nuisance variable to reduce noise in final score comparisons.
Exam tip:
Never skip describing random assignment explicitly: vague references to 'randomly choosing people' will lose points.
2. Common Experimental Design Structuresβ β β βββ± 3 min
Below is a side-by-side breakdown of the three most frequently tested designs:
Completely Randomized Design
All experimental units are assigned to treatment groups entirely at random, with no pre-grouping. This is the default design for most simple experiments.
+ Pros: Simple to implement, easy to analyze with standard t-tests
β Cons: Does not reduce variability from known strong nuisance variables
Randomized Block Design
Experimental units are first grouped into blocks of units that are identical or very similar on a known nuisance variable (e.g. gender, age group, prior test score), then random assignment happens within each block.
+ Pros: Removes variability from the blocking variable from your final error term, making tests more powerful
β Cons: Requires you to identify and measure the blocking variable before running the experiment
Matched Pairs Design
A special case of block design with exactly two treatments: units are paired by near-identical traits, or each single unit receives both treatments in random order to eliminate between-unit variability.
+ Pros: Maximizes test power even with very small sample sizes
β Cons: Order effects can bias results if treatment order is not randomized for each unit
A school wants to test two different math curricula across 8 classes with 30 students each, and knows that pre-existing math ability varies widely across classes. Which design should they use, and why?
- 1
Step 1: Identify the known nuisance variable: pre-existing student math ability across the 8 classes.
- 2
Step 2: Rule out completely randomized design: random assignment across all 240 students would not account for the large pre-existing ability differences between classes.
- 3
Step 3: Select randomized block design: block students by their current class, then randomly assign half of the classes to use curriculum A and half to use curriculum B.
- 4
Step 4: Justify: Blocking by class removes all variability from pre-existing math ability from the final comparison of curriculum effectiveness.
3. Blinding, Placebo, and Bias Mitigationβ β ββββ± 3 min
Placebo Effect
The measurable change in response from a participant who receives an inert treatment, purely from their belief that they are receiving an active intervention
Single Blind: Participants do not know which treatment group they are assigned to, eliminating response bias from the placebo effect
Double Blind: Neither participants nor the researchers collecting or assessing response data know which group is assigned to which treatment, eliminating observer bias from researchers unconsciously favoring one treatment group
A new painkiller drug is tested on 200 patients with chronic back pain. Explain how to implement double blinding for this study.
- 1
Step 1: Manufacture the placebo pill to be identical in size, color, taste, and packaging to the real painkiller pill, so no patient can tell them apart.
- 2
Step 2: Have a third-party researcher not involved in data collection create a random assignment list, and label all pill bottles only with a random ID number.
- 3
Step 3: Give nurses and researchers collecting pain score data no access to the assignment key, so they cannot know if a patient received the real drug or placebo.
- 4
Step 4: Only unblind the groups after all pain score data has been fully collected and entered into the analysis spreadsheet.
Test your understanding of blinding with this quick question:
Which of the following scenarios describes double blinding?
Only participants do not know their treatment assignment
Neither participants nor data collectors know treatment assignment
Only data collectors know treatment assignment
No one at all knows treatment assignment
Reveal answer
Neither participants nor data collectors know treatment assignment βDouble blinding requires that both participants and the team interacting with them and measuring outcomes are unaware of group assignments to eliminate all sources of human bias.
4. AP Exam Phrasing for Experimental Design Questionsβ β β β ββ± 3 min
5. Common Pitfalls
Wrong move:
Claiming an observational study supports causal conclusions
Why:
Only properly randomized controlled experiments can rule out unmeasured confounding variables
Correct move:
Explicitly state that causal conclusions are only justified for randomized comparative experiments, not observational data
Wrong move:
Mixing up random sampling and random assignment
Why:
AP rubrics deduct 50% of available points for confusing these two distinct processes
Correct move:
Clearly distinguish random sampling (generalizing results to a larger population) from random assignment (testing for causal treatment effects)
Wrong move:
Calling a block a control group
Why:
Blocks are used to reduce variability from a nuisance trait, not as a baseline comparison for treatment effects
Correct move:
Label groups that receive no active treatment as control, and groups stratified by a pre-existing trait as blocks
Wrong move:
Forgetting to justify why blocking improves study precision
Why:
AP exam questions on blocking never award points for just naming a blocking variable
Correct move:
Explain that units in each block are similar on the blocking trait, so variation from that trait is removed from the final error term
Wrong move:
Describing a matched pairs design as a simple two-group randomized design
Why:
Matched pairs uses pairing to eliminate between-unit variability, which makes the analysis different from a standard two-sample t-test
Correct move:
Explicitly note that each pair of units is matched on similar traits, or that each unit receives both treatments, to earn full design identification points
6. Quick Reference Cheatsheet
Design Type | Key Feature | Best Use Case | Causal Conclusion Valid? |
|---|---|---|---|
Completely Randomized | Units assigned to treatments entirely at random | No known major nuisance variables | Yes |
Randomized Block | Units grouped by known nuisance trait before random assignment | Nuisance variable strongly impacts response | Yes |
Matched Pairs | Units paired by similarity, or each unit gets both treatments | Small sample size, high between-unit variability | Yes |
Observational Study | No treatment assignment by researcher | Impossible or unethical to assign treatments | No |
When this came up on past exams
AI-estimated based on syllabus patterns β cross-check with official past papers for accuracy. Use only as revision-focus signals.
- 2025 Β· 1
FRQ on new drug trial design
- 2023 Β· 2
MCQ on purpose of blocking
- 2022 Β· 1
FRQ experiment vs observational study
What's Next
Mastering experimental design accounts for 10-15% of all AP Stats exam points, almost always appearing as a standalone free response question on every test. The skills you learn here will directly transfer to upcoming inference units, including two-sample t-tests and chi-square tests for homogeneity, where you will be asked to select the correct hypothesis test based on the study design you identified. You will also apply these principles to critique real-world studies published in peer-reviewed research and popular media, to avoid being misled by misleading causal claims from poorly designed surveys or observational data.