# Experimental Design

> AP Statistics · AP Stats 2024-2026 Curriculum
> Source: https://www.owlsprep.com/study/ap-statistics-u10-experimental-design/

This module covers core experimental design principles, common structure types, bias mitigation, and how to justify causal conclusions aligned to AP Stats assessment standards.

**Prerequisites:** [Observational studies vs experiments distinction](https://www.owlsprep.com/study/ap-statistics-u10-intro-to-data-collection/); [Basic definition of confounding variables](https://www.owlsprep.com/study/ap-statistics-u10-confounding-and-bias/)

## Learning objectives

- Distinguish between observational studies and controlled experiments
- Identify and apply the 4 core principles of valid experimental design
- Classify common designs including completely randomized, randomized block, and matched pairs
- Evaluate sources of bias and confounding that invalidate causal conclusions
- Justify experimental design choices to earn full points on AP free response rubrics

## Four Core Principles of Experimental Design

All valid AP-level experiments follow four non-negotiable rules to produce reliable, interpretable results that support causal claims.

**Experimental Unit** — The individual, object, or subject to which a treatment is assigned in an experiment

*Example:* A high school student testing a new study app, a potted plant receiving a new fertilizer

1. **Control**: Include at least one comparison group (control or alternative treatment) to isolate the effect of the active treatment
2. **Randomization**: Use random assignment to distribute known and unknown confounding variables evenly across all treatment groups
3. **Replication**: Assign enough experimental units to each treatment group to reduce random sampling variability
4. **Variability Reduction**: Account for known nuisance variables that impact the response to reduce unneeded noise in results

**Worked example:** A researcher wants to test if a new energy drink improves exam scores for college students. List all four core principles you would include in this experiment.

1. Step 1 (Control): Include a control group that receives an identical-tasting placebo drink with no active ingredients to compare against the energy drink group.
2. Step 2 (Randomization): Use a random number generator to assign 100 volunteer students to either the energy drink group or the placebo group, 50 students per group.
3. Step 3 (Replication): Use 50 students per group rather than 2 per group to ensure average score differences are not caused by random luck.
4. Step 4 (Variability Reduction): Account for pre-existing student GPA as a known nuisance variable to reduce noise in final score comparisons.

> **tip**
>
> AP exam rubrics almost always award 2 full points for explicitly naming and describing these four principles in a free response design question.

> **Exam tip:** Never skip describing random assignment explicitly: vague references to 'randomly choosing people' will lose points.

## Common Experimental Design Structures

Three standard experimental designs appear on nearly every AP Statistics exam, each optimized for different study constraints.

**Comparing methods**

Below is a side-by-side breakdown of the three most frequently tested designs:

- **Completely Randomized Design** — All experimental units are assigned to treatment groups entirely at random, with no pre-grouping. This is the default design for most simple experiments.
  - Pros: Simple to implement, easy to analyze with standard t-tests
  - Cons: Does not reduce variability from known strong nuisance variables

- **Randomized Block Design** — Experimental units are first grouped into blocks of units that are identical or very similar on a known nuisance variable (e.g. gender, age group, prior test score), then random assignment happens *within each block*.
  - Pros: Removes variability from the blocking variable from your final error term, making tests more powerful
  - Cons: Requires you to identify and measure the blocking variable before running the experiment

- **Matched Pairs Design** — A special case of block design with exactly two treatments: units are paired by near-identical traits, or each single unit receives both treatments in random order to eliminate between-unit variability.
  - Pros: Maximizes test power even with very small sample sizes
  - Cons: Order effects can bias results if treatment order is not randomized for each unit

**Worked example:** A school wants to test two different math curricula across 8 classes with 30 students each, and knows that pre-existing math ability varies widely across classes. Which design should they use, and why?

1. Step 1: Identify the known nuisance variable: pre-existing student math ability across the 8 classes.
2. Step 2: Rule out completely randomized design: random assignment across all 240 students would not account for the large pre-existing ability differences between classes.
3. Step 3: Select randomized block design: block students by their current class, then randomly assign half of the classes to use curriculum A and half to use curriculum B.
4. Step 4: Justify: Blocking by class removes all variability from pre-existing math ability from the final comparison of curriculum effectiveness.

## Blinding, Placebo, and Bias Mitigation

Even perfectly randomized experiments can produce invalid results if human bias distorts participant behavior or data collection.

**Placebo Effect** — The measurable change in response from a participant who receives an inert treatment, purely from their belief that they are receiving an active intervention

- **Single Blind**: Participants do not know which treatment group they are assigned to, eliminating response bias from the placebo effect
- **Double Blind**: Neither participants nor the researchers collecting or assessing response data know which group is assigned to which treatment, eliminating observer bias from researchers unconsciously favoring one treatment group

**Worked example:** A new painkiller drug is tested on 200 patients with chronic back pain. Explain how to implement double blinding for this study.

1. Step 1: Manufacture the placebo pill to be identical in size, color, taste, and packaging to the real painkiller pill, so no patient can tell them apart.
2. Step 2: Have a third-party researcher not involved in data collection create a random assignment list, and label all pill bottles only with a random ID number.
3. Step 3: Give nurses and researchers collecting pain score data no access to the assignment key, so they cannot know if a patient received the real drug or placebo.
4. Step 4: Only unblind the groups after all pain score data has been fully collected and entered into the analysis spreadsheet.

**Check your understanding**

Test your understanding of blinding with this quick question:

1. Which of the following scenarios describes double blinding?

   - Only participants do not know their treatment assignment
   - Neither participants nor data collectors know treatment assignment
   - Only data collectors know treatment assignment
   - No one at all knows treatment assignment

   *Why:* Double blinding requires that both participants and the team interacting with them and measuring outcomes are unaware of group assignments to eliminate all sources of human bias.

## AP Exam Phrasing for Experimental Design Questions

**Exam command terms**

AP Statistics uses very specific command terms for experimental design questions that have strict rubric requirements:

- **Describe an experiment** — You must explicitly list random assignment steps, control groups, and what response variable you will measure to earn full points *("Use a random number generator to assign 50 plants to the new fertilizer group, 50 to the control group, then measure height after 6 weeks")*

- **Explain the purpose of blocking** — You must link the blocking variable directly to reduced variability in the response, not just name the block *("Blocking by plant species removes variability in natural growth rate from the comparison of fertilizer effectiveness")*

- **Why can we conclude causation?** — You must explicitly state that random assignment evenly distributed all confounding variables across groups, which observational studies cannot do

## Common pitfalls

- **Wrong:** Claiming an observational study supports causal conclusions
  - Why it fails: Only properly randomized controlled experiments can rule out unmeasured confounding variables
  - Correct: Explicitly state that causal conclusions are only justified for randomized comparative experiments, not observational data
- **Wrong:** Mixing up random sampling and random assignment
  - Why it fails: AP rubrics deduct 50% of available points for confusing these two distinct processes
  - Correct: Clearly distinguish random sampling (generalizing results to a larger population) from random assignment (testing for causal treatment effects)
- **Wrong:** Calling a block a control group
  - Why it fails: Blocks are used to reduce variability from a nuisance trait, not as a baseline comparison for treatment effects
  - Correct: Label groups that receive no active treatment as control, and groups stratified by a pre-existing trait as blocks
- **Wrong:** Forgetting to justify why blocking improves study precision
  - Why it fails: AP exam questions on blocking never award points for just naming a blocking variable
  - Correct: Explain that units in each block are similar on the blocking trait, so variation from that trait is removed from the final error term
- **Wrong:** Describing a matched pairs design as a simple two-group randomized design
  - Why it fails: Matched pairs uses pairing to eliminate between-unit variability, which makes the analysis different from a standard two-sample t-test
  - Correct: Explicitly note that each pair of units is matched on similar traits, or that each unit receives both treatments, to earn full design identification points

## Cheatsheet

| Design Type | Key Feature | Best Use Case | Causal Conclusion Valid? |
| --- | --- | --- | --- |
| Completely Randomized | Units assigned to treatments entirely at random | No known major nuisance variables | Yes |
| Randomized Block | Units grouped by known nuisance trait before random assignment | Nuisance variable strongly impacts response | Yes |
| Matched Pairs | Units paired by similarity, or each unit gets both treatments | Small sample size, high between-unit variability | Yes |
| Observational Study | No treatment assignment by researcher | Impossible or unethical to assign treatments | No |

## What's next

Mastering experimental design accounts for 10-15% of all AP Stats exam points, almost always appearing as a standalone free response question on every test. The skills you learn here will directly transfer to upcoming inference units, including two-sample t-tests and chi-square tests for homogeneity, where you will be asked to select the correct hypothesis test based on the study design you identified. You will also apply these principles to critique real-world studies published in peer-reviewed research and popular media, to avoid being misled by misleading causal claims from poorly designed surveys or observational data.

---

From [OwlsPrep](https://www.owlsprep.com) — free study guides for A-Level, IB, AP and IGCSE, written against the official syllabus. Canonical page: https://www.owlsprep.com/study/ap-statistics-u10-experimental-design/