Notes · StatisticsUK · A-Levels

Experimental design

This topic covers the principles of designing studies that support valid conclusions: the distinction between observational studies and experiments, control and comparison, randomisation and replication, and blocking and matched pairs. It treats confounding, the main design types, and how sound design sits within the statistical enquiry cycle.

5 sections~13 min reading time3 competenciesLevel Standard 3 · Advanced 2

T·181818 / 18

Exam profile

AO1 · Define the principles of experimental designAO2 · Identify design features and sources of confounding in a studyAO3 · Design and critique an experiment within the statistical enquiry cycle

Operators:designexplaindescribeidentifyassessjustify

basic level

Experimental design is principally full A-Level (Paper 3) content within the statistical-enquiry strand.

higher level

The full A-Level expects the principles of control, randomisation, replication and blocking, the recognition of confounding, and the critique of a design.

Depth

Text

Contents · 5 sections

Experimental design

Observational studies and experiments#

StandardPearson Edexcel 9ST0, Topic 18 (Paper 3)

A study measures the relationship between an explanatory variable and a response. In an OBSERVATIONAL study the explanatory variable is merely observed as it naturally occurs — comparing lung-cancer rates between people who already smoke and those who do not. In an EXPERIMENT the investigator actively IMPOSES the explanatory variable, deliberately assigning units to treatments, which is the decisive difference.

The distinction matters because only a well-designed experiment can establish causation. In an observational study the groups may differ systematically in other ways — smokers may also differ in diet, exercise or occupation — so an observed association could be due to these other variables rather than to smoking. Such a lurking variable that is associated with both the explanatory and the response variable is a confounder.

By assigning treatments itself, ideally at random, an experiment breaks the link between the treatment and any confounders, so that on average the treatment groups are alike in every respect except the treatment. Any difference in response can then be attributed to the treatment. This is why the experiment, not the observational study, is the gold standard for causal claims.

Observational studies are nevertheless valuable and sometimes the only ethical or practical option — one cannot randomly assign people to smoke. They can establish strong, replicated associations and generate hypotheses, but conclusions must be worded as association rather than cause, with confounding explicitly acknowledged. Recognising which kind of study is described, and its consequent limits, is a core AO2 skill.

Worked example

Observational versus experimental

Researchers report that students who eat breakfast score higher in tests. Explain why this does not prove breakfast causes higher scores, and outline an experiment that could.

Identify the study type
Breakfast habits were observed, not assigned, so this is an observational study.
Confounding
Students who eat breakfast may also have more stable home lives or better sleep, which could raise scores — these are confounders.
Design an experiment
Randomly assign comparable students to a breakfast or no-breakfast group and compare test scores, so the groups differ only in breakfast.

Result: The observational link may be confounded; only a randomised experiment assigning breakfast could establish causation.

Typical mistakes

Claiming causation from an observational study.
Confusing a confounder with the response, or overlooking confounding altogether.

Active revision

A survey finds that people who drink coffee sleep less. Explain why this observational finding does not establish that coffee causes reduced sleep, naming a possible confounder.

Active recall

Recall the key points — then reveal.

Sources: Pearson Edexcel Level 3 Advanced GCE in Statistics (9ST0) Specification (Pearson Edexcel)

Control and comparison#

StandardPearson Edexcel 9ST0, Topic 18 (Paper 3)

An experiment needs something to compare its treatment against, which is the role of a control group. The control group receives no treatment, or a standard/existing treatment, so that the treatment group's response can be measured RELATIVE to it. Without a comparison group, a change in the treatment group cannot be distinguished from a change that would have happened anyway.

In studies with human subjects the mere expectation of benefit can produce a real improvement — the placebo effect. To separate the treatment's genuine effect from this psychological response, the control group is given a placebo (an inert treatment indistinguishable from the real one), so both groups share any placebo effect and the difference isolates the treatment.

Bias from expectations is further controlled by blinding. In a single-blind study the subjects do not know which group they are in; in a double-blind study neither the subjects nor the people measuring the response know, which prevents the assessors' expectations from influencing the measurements. Double-blinding is standard in clinical trials for exactly this reason.

Control also means holding other conditions as constant as possible across the groups — the same environment, timing and measurement procedure — so that the treatment is the only systematic difference. Together, a comparison group, placebo control and blinding remove the main non-treatment explanations for any observed difference, which is what makes the comparison fair.

Worked example

Designing the controls

A new drug for headaches is to be trialled. Describe the control features that make the comparison fair.

Comparison group
Include a control group given a placebo pill identical in appearance to the drug.
Placebo effect
The placebo ensures both groups share any psychological improvement, so the difference isolates the drug's true effect.
Blinding
Make the trial double-blind so neither patients nor assessors know who received the drug, preventing expectation bias.

Result: A fair comparison uses a placebo control with double-blinding, so the only systematic difference between groups is the drug itself.

Typical mistakes

Omitting a control/comparison group, so any change cannot be attributed to the treatment.
Confusing the placebo (inert treatment) with blinding (concealing group membership).

Active revision

Design the control arrangements for a trial of a new painkiller, explaining the role of a placebo and of double-blinding.

Active recall

Recall the key points — then reveal.

Sources: Pearson Edexcel Level 3 Advanced GCE in Statistics (9ST0) Specification (Pearson Edexcel)

Randomisation and replication#

StandardPearson Edexcel 9ST0, Topic 18 (Paper 3)

A completely randomised design

Fig. 1Completely randomised design: units are randomly allocated to treatment or control, then responses compared.

Randomisation is the random allocation of experimental units to treatments, and it is the single most important design principle. By assigning at random, the investigator ensures that, on average, the treatment groups are balanced in every characteristic — measured and unmeasured — except the treatment itself. This is what neutralises confounders and justifies attributing a difference in response to the treatment.

The completely randomised design, shown below, applies this directly: the experimental units are allocated at random to the treatment and control groups, and the responses are then compared. Because chance alone decides the allocation, no systematic difference between the groups is built in, and the later analysis (a two-sample test or ANOVA) rests on this randomness.

Replication means applying each treatment to several units, not just one. Replication is essential because it lets the natural variation between units be estimated and averaged out: with more units per treatment the standard error of each group mean,

\dfrac{\sigma}{\sqrt{n}}

, shrinks, so a real treatment effect can be distinguished from random fluctuation. A single unit per treatment gives no way to separate signal from noise.

Randomisation and replication work together. Randomisation removes systematic bias; replication controls random error and provides the estimate of variability that every significance test needs. Adequate sample size is the practical face of replication, and designing a study large enough to detect an effect worth finding links directly to the earlier idea of statistical power.

\text{standard error of a group mean} = \frac{\sigma}{\sqrt{n}}

Why replication helps

More units per treatment ( $n$ ) shrinks the standard error, sharpening the comparison.

Worked example

Randomising and replicating

Thirty seedlings are to be assigned to three fertilisers. Describe the random allocation and the role of replication.

Random allocation
Number the seedlings 1 to 30 and use random numbers to assign 10 to each fertiliser, so no fertiliser systematically gets stronger seedlings.
Why randomise
Randomisation balances confounders such as initial size across the groups on average.
Replication
Ten seedlings per fertiliser (rather than one) let within-group variation be estimated, so a real difference in mean growth can be told from random variation.

Result: Random numbers assign 10 seedlings to each fertiliser; replication of 10 per group provides the variability estimate the analysis needs.

Typical mistakes

Confusing random allocation to treatments (randomisation) with random sampling from a population.
Believing a larger sample removes bias; replication controls random error, randomisation controls bias.

Active revision

Describe how you would use random numbers to allocate 30 plants to three treatments, and explain why replication matters.

Active recall

Recall the key points — then reveal.

Sources: Pearson Edexcel Level 3 Advanced GCE in Statistics (9ST0) Specification (Pearson Edexcel)

Blocking and matched pairs#

AdvancedPearson Edexcel 9ST0, Topic 18 (Paper 3)

A randomised block design

Fig. 2Randomised block design: each block receives all three treatments (A, B, C) in random order across its plots.

When the experimental units vary in a known way that affects the response, that variation can be removed by BLOCKING. Units are grouped into blocks that are internally similar with respect to the nuisance variable — plots of similar soil, patients of similar age — and the treatments are then randomly allocated WITHIN each block. In the randomised block design below, every block contains all three treatments in random order.

Blocking works by comparing treatments within homogeneous groups, so the between-block variation (the effect of the nuisance variable) is separated out and does not inflate the error against which the treatment effect is judged. This makes the experiment more precise and more powerful than a completely randomised design when the blocking variable genuinely matters, following the maxim: block what you can, randomise what you cannot.

The matched-pairs design is the special case of blocks of size two. Units are matched into pairs that are as alike as possible (twins, or the same person before and after), and the two treatments are randomly assigned one to each member of the pair; often each subject receives both treatments in random order. The paired

t

-test or the Wilcoxon signed-rank test on the within-pair differences is the natural analysis, removing the between-pair variation.

Choosing between a completely randomised design and a blocked or matched design is a design judgement. Blocking is worthwhile when a strong source of nuisance variation can be identified and used to form similar groups; if no such variable is available, a completely randomised design is simpler and adequate. Explaining WHY a particular design controls the relevant variation is the higher-order skill assessed.

Worked example

Using a randomised block design

Three fertilisers are compared across a field with a fertility gradient (one end richer than the other). Describe a randomised block design and why it helps.

Form blocks
Divide the field into strips (blocks) across the gradient, so each strip is roughly uniform in fertility.
Randomise within blocks
Within each strip, randomly assign the three fertilisers to three plots (all three appear in every block).
Why it helps
Comparisons are made within uniform strips, so the fertility gradient is removed from the error and the fertiliser effect is estimated more precisely.

Result: Blocking by fertility strip and randomising fertilisers within each strip removes the gradient's variation, sharpening the comparison.

Typical mistakes

Allocating treatments across blocks rather than randomly WITHIN each block.
Analysing matched-pairs data as two independent samples, discarding the pairing.

Active revision

An experiment compares three diets on cattle of widely varying starting weights. Explain how blocking by starting weight would improve the design.

Active recall

Recall the key points — then reveal.

Sources: Pearson Edexcel Level 3 Advanced GCE in Statistics (9ST0) Specification (Pearson Edexcel)

Design types, confounding and the enquiry cycle#

AdvancedPearson Edexcel 9ST0, Topic 18 (Paper 3)

The statistical enquiry cycle

Fig. 3The statistical enquiry cycle: design sits at the planning stage and shapes what later analysis can claim.

The three standard designs answer different needs: the completely randomised design allocates all units at random and is simplest; the randomised block design controls one known nuisance variable by blocking; and the matched-pairs design blocks in pairs for maximum control of between-unit variation. Choosing among them balances the simplicity of full randomisation against the extra precision blocking buys when a strong nuisance variable exists.

Confounding is the thread running through the whole topic: a confounder is a variable linked to both the treatment and the response that can create or mask an apparent effect. Randomisation defends against confounders in general by balancing them on average, blocking removes the effect of a specific known one, and control and blinding remove others; naming the likely confounder in a described study and the design feature that handles it is a recurring AO3 demand.

Good design is one stage of the statistical enquiry cycle shown below: pose a question and plan, collect data, process and present it, analyse and interpret, and conclude and communicate — then, informed by the findings, pose the next question. Design decisions made at the planning stage determine what the later analysis can honestly claim, which is why the cycle emphasises planning before data are collected.

The cycle also carries the ethical dimension of experimentation: informed consent, minimising harm, and honest reporting including of null results and limitations. A complete answer to a design question therefore not only proposes a sound structure but also anticipates confounders, states what the design can and cannot establish, and locates the study within this iterative, ethical process of enquiry.

Worked example

A full design critique

Design a study to test whether a new revision app raises exam scores, addressing confounding and locating it in the enquiry cycle.

Plan
Pose the question and choose a randomised design: randomly allocate comparable students to 'app' and 'no-app' groups (a control).
Confounding
Prior ability is a confounder; randomisation balances it on average, and blocking students by prior grade would control it directly.
Collect and analyse
Collect exam scores, then compare group means with a two-sample test, reporting the effect size.
Conclude with limits
Conclude about the app's effect for this population, noting it cannot generalise beyond the students studied and depends on honest app usage.

Result: A randomised (optionally blocked) design with a control group, analysed by a two-sample test, controls prior-ability confounding within the enquiry cycle, with generalisation stated as a limit.

Typical mistakes

Proposing a design without addressing the specific confounding variable in the context.
Overstating what the study can conclude, ignoring the limits set at the design stage.

Active revision

For an experiment testing whether a revision app improves grades, propose a design, name a likely confounder and the feature that controls it, and state one limitation.

Active recall

Recall the key points — then reveal.

Sources: GCE AS and A level subject content (Statistics) (Department for Education / Ofqual)

References & sources

Sources

Pearson Edexcel

Pearson Edexcel Level 3 Advanced GCE in Statistics (9ST0) Specification

Department for Education / Ofqual

GCE AS and A level subject content (Statistics)

Experimental design

5 sections~13 min reading time3 competenciesLevel Standard 3 · Advanced 2

Experimental design

Key points

Observational versus experimental

Key points

Designing the controls

Key points

Randomising and replicating

Key points

Using a randomised block design

Key points

A full design critique

Sources

Experimental design

Key points

Observational versus experimental

Key points

Designing the controls

Key points

Randomising and replicating

Key points

Using a randomised block design

Key points

A full design critique

Sources