Notes · StatisticsUK · A-Levels

Introduction to hypothesis testing

This topic sets out the logic of significance testing: the null and alternative hypotheses, one- and two-tailed tests, the significance level, the critical region and the p-value. It carries out a hypothesis test for a binomial proportion, and analyses the two kinds of error a test can make and the actual significance level of a discrete test.

5 sections~13 min reading time3 competenciesLevel Standard 2 · Advanced 3

T·111111 / 18

Exam profile

AO1 · Set up and carry out a hypothesis test with a stated significance levelAO2 · Interpret the significance level, critical region and p-valueAO3 · Conclude a test in context and discuss the risk of error

Operators:testfinddeterminestateconcludecomment

basic level

The AS foundation expects a one-tailed binomial hypothesis test, the hypotheses, significance level and conclusion.

higher level

The full A-Level expects two-tailed tests, the p-value approach, critical regions, actual significance levels and the two error types.

Depth

Text

Contents · 5 sections

Introduction to hypothesis testing

The logic of a hypothesis test#

StandardPearson Edexcel 9ST0, Topic 11 (Paper 2)

The hypothesis-testing procedure

Fig. 1The fixed sequence of a hypothesis test, ending always in a contextual conclusion.

A hypothesis test weighs evidence against a specific claim about a population. It begins by assuming the claim — the null hypothesis

H_0

— is true, then asks how surprising the observed data would be under that assumption. If the data would be very unlikely when

H_0

is true, that counts as evidence against

H_0

in favour of the alternative

H_1

; if not, there is insufficient evidence to reject

H_0

The reasoning is deliberately asymmetric, like a trial in which the null is presumed until the evidence is strong enough to overturn it. We never prove

H_0

true; we either reject it or fail to reject it. This is why the correct conclusion is 'there is (in)sufficient evidence to reject

H_0

', never 'accept

H_0

' — the test can only fail to find evidence against the null, not confirm it.

The procedure follows fixed steps, shown in the flow below: state

H_0

and

H_1

; choose a significance level; find the critical region (or the test statistic and its p-value); compare the data with it; decide to reject or not; and state the conclusion in context. Following the steps in order, and never skipping the contextual conclusion, is what full-mark answers do.

Every step has a purpose. The hypotheses fix what is being tested and in which direction; the significance level fixes how much evidence is required; the critical region or p-value measures the evidence; and the contextual conclusion translates the statistical decision back into the language of the problem. Understanding the role of each step prevents the mechanical, context-free answers that lose marks.

Worked example

Framing a test

A manufacturer claims at most 10% of its items are faulty. A retailer suspects the rate is higher. Set up the hypotheses and describe the logic of the test.

Null hypothesis
$H_0$ : $p = 0.10$ (the claimed faulty rate).
Alternative
$H_1$ : $p > 0.10$ (the retailer's suspicion), a one-tailed test.
Logic
Assume $p = 0.10$ ; if the observed number of faults would be very unlikely under this assumption, reject $H_0$ in favour of $H_1$ .

Result: $H_0$ : $p = 0.10$ against $H_1$ : $p > 0.10$ ; reject $H_0$ only if the data are too surprising under $p = 0.10$ .

Typical mistakes

Writing 'accept $H_0$ ' instead of 'insufficient evidence to reject $H_0$ '.
Omitting the conclusion in context, leaving only a statistical decision.

Active revision

Explain, in your own words, why a hypothesis test can never prove the null hypothesis true.

Active recall

Recall the key points — then reveal.

Sources: Pearson Edexcel Level 3 Advanced GCE in Statistics (9ST0) Specification (Pearson Edexcel)

Hypotheses, tails and significance levels#

StandardPearson Edexcel 9ST0, Topic 11 (Paper 2)

The null hypothesis

H_0

specifies a particular value of the parameter (for example

p = 0.5

), and the alternative

H_1

specifies the departure of interest. If

H_1

points in one direction (

p > 0.5

p < 0.5

) the test is one-tailed; if it merely says the value is different (

p \neq 0.5

) the test is two-tailed. The wording of the question — 'has increased', 'has changed', 'is biased' — fixes the tail, and choosing it correctly is essential.

The significance level

\alpha

is the probability threshold below which the data are judged too surprising to attribute to chance, commonly 5%, 1% or 10%. It is fixed BEFORE seeing the data, so the standard of evidence cannot be adjusted to suit the result. A smaller

\alpha

demands stronger evidence to reject

H_0

and so reduces the chance of a false rejection.

For a two-tailed test the significance level is split between the two tails, so a 5% two-tailed test uses

2.5\%

in each tail. This matters because the same data can be significant one-tailed but not two-tailed: a two-tailed test is more conservative, reflecting that it looks for a departure in either direction and so demands more in the direction observed.

The direction of

H_1

must be justified by the context, not chosen after seeing the data. Testing 'has the proportion increased?' one-tailed because the sample happened to be high, when the original question was 'has it changed?', is a form of cheating that inflates the true error rate. Setting the hypotheses and the tail from the wording, before any calculation, is the disciplined approach the examination rewards.

\text{one-tailed: } H_1: p > p_0 \;\text{or}\; p < p_0; \qquad \text{two-tailed: } H_1: p \neq p_0

Hypotheses and tails

A two-tailed test splits $\alpha$ between the tails ( $2.5\%$ each at the 5% level).

Worked example

Setting up a two-tailed test

A die is suspected of being biased towards or against sixes. It is tested at the 5% level. State the hypotheses and the tail allocation.

Parameter
Let $p = P(\text{six})$ ; a fair die has $p = \tfrac{1}{6}$ .
Hypotheses
$H_0$ : $p = \tfrac{1}{6}$ against $H_1$ : $p \neq \tfrac{1}{6}$ (bias in either direction), so two-tailed.
Tail allocation
At the 5% level, each tail carries $2.5\%$ .

Result: $H_0$ : $p = \tfrac{1}{6}$ against $H_1$ : $p \neq \tfrac{1}{6}$ , two-tailed with $2.5\%$ in each tail.

Typical mistakes

Selecting the tail after seeing the data rather than from the question.
Using the full $\alpha$ in one tail of a two-tailed test instead of $\alpha/2$ .

Active revision

A coin is tested for bias (in either direction) at the 5% level. State $H_0$ , $H_1$ and the probability allocated to each tail.

Active recall

Recall the key points — then reveal.

Sources: Pearson Edexcel Level 3 Advanced GCE in Statistics (9ST0) Specification (Pearson Edexcel)

Critical regions and the p-value approach#

AdvancedPearson Edexcel 9ST0, Topic 11 (Paper 2)

There are two equivalent ways to make the decision. The critical-region method finds, before seeing the data, the set of extreme outcomes whose total probability under

H_0

is at most

\alpha

; if the observed value falls in this region,

H_0

is rejected. The p-value method computes, after seeing the data, the probability under

H_0

of a result at least as extreme as the one observed, and rejects

H_0

if this p-value is at most

\alpha

The two methods always agree, because the observed value lies in the critical region exactly when its p-value is at most

\alpha

. The p-value is often preferred because it reports HOW surprising the data are, not merely whether they cross the threshold: a p-value of 0.001 is far stronger evidence against

H_0

than a p-value of 0.049, though both reject at the 5% level.

A critical p-value must be interpreted precisely. It is the probability of the observed (or more extreme) data GIVEN that

H_0

is true — it is NOT the probability that

H_0

is true. A small p-value means the data are surprising under

H_0

, which is evidence against it, but it says nothing directly about the probability of the hypothesis itself.

For a discrete test the critical region is built from cumulative probabilities and rarely has probability exactly

\alpha

, a subtlety pursued in the binomial test next. In every case the decision rule is the same: reject

H_0

when the evidence (the p-value) is at most the pre-set significance level, and otherwise do not reject.

\text{p-value} = P(\text{data at least as extreme} \mid H_0); \quad \text{reject } H_0 \iff \text{p-value} \leq \alpha

The p-value decision rule

The p-value is a conditional probability given $H_0$ , not the probability that $H_0$ is true.

Worked example

Deciding from a p-value

A one-tailed test at the 5% level gives a p-value of 0.032. State the decision and interpret the value.

Compare with alpha
$0.032 \leq 0.05$ , so the data fall in the critical region.
Decision
Reject $H_0$ at the 5% level.
Interpret
If $H_0$ were true, data this extreme would occur with probability 0.032 — surprising enough to reject $H_0$ , though not overwhelming.

Result: Reject $H_0$ : the p-value 0.032 is below 0.05, meaning such data are unlikely (probability 0.032) under $H_0$ .

Typical mistakes

Interpreting the p-value as the probability that $H_0$ is true.
Comparing the p-value with the wrong threshold (for example $\alpha$ rather than $\alpha/2$ in a two-tailed test).

Active revision

In a one-tailed test at the 5% level the p-value is 0.032. State the decision and interpret the p-value.

Active recall

Recall the key points — then reveal.

Sources: Pearson Edexcel Level 3 Advanced GCE in Statistics (9ST0) Specification (Pearson Edexcel)

A test for a binomial proportion#

AdvancedPearson Edexcel 9ST0, Topic 11 (Paper 2)

Critical region on the number of heads

Fig. 2Critical region X >= 15 for testing H0: p = 0.5 against H1: p > 0.5 with n = 20 at the 5% level.

The archetypal test concerns a binomial proportion. To test whether a coin is biased towards heads, take

H_0

p = 0.5

against

H_1

p > 0.5

, model the number of heads

X

n

tosses as

B(n, 0.5)

under

H_0

, and reject

H_0

for values of

X

in the upper tail. The critical region is the smallest set of large values whose total probability under

H_0

does not exceed

\alpha

For

n = 20

tosses at the 5% level, the cumulative binomial gives

P(X \geq 15) = 1 - P(X \leq 14) = 1 - 0.9793 = 0.0207

, which is below 5%, whereas

P(X \geq 14) = 0.0577

exceeds 5%. The critical region is therefore

X \geq 15

, shown on the number line below, and the critical value is 15.

Because the binomial is discrete, the critical region's probability (0.0207) is below the nominal 5% rather than exactly equal to it; this true probability is the ACTUAL significance level of the test. The gap arises because

X

jumps in whole steps, so no region has probability exactly 0.05 — a feature of every discrete test.

The decision then compares the observed count with the region: 15 or more heads falls in the critical region and rejects

H_0

, while 14 or fewer does not. The conclusion must be contextual — 'there is sufficient evidence at the 5% level that the coin is biased towards heads' — and should acknowledge that the actual significance level is 2.07%, not 5%.

\text{Under } H_0,\; X \sim B(20, 0.5); \quad P(X \geq 15) = 0.0207 < 0.05 \leq P(X \geq 14) = 0.0577

Critical region for the binomial test

The critical region is $X \geq 15$ ; its actual probability 0.0207 is the actual significance level.

Worked example

A one-tailed binomial test

A coin tossed 20 times shows 16 heads. Test at the 5% level whether it is biased towards heads, using $P(X \leq 14) = 0.9793$ under $H_0$ .

Hypotheses
$H_0$ : $p = 0.5$ against $H_1$ : $p > 0.5$ ; under $H_0$ , $X \sim B(20, 0.5)$ .
Critical region
$P(X \geq 15) = 1 - 0.9793 = 0.0207 < 0.05$ , and $P(X \geq 14) = 0.0577 > 0.05$ , so the region is $X \geq 15$ .
Compare
The observed 16 lies in the region $X \geq 15$ .
$16 \geq 15 \Rightarrow \text{reject } H_0$
Conclude
Reject $H_0$ : there is sufficient evidence at the 5% level (actual level 2.07%) that the coin is biased towards heads.

Result: Since 16 lies in the critical region $X \geq 15$ , reject $H_0$ ; the coin appears biased towards heads (actual significance level 2.07%).

Typical mistakes

Using $P(X \geq c) \leq \alpha$ but taking $c$ one step too small so the region exceeds $\alpha$ .
Reporting the nominal level (5%) as the actual significance level for a discrete test.

Active revision

Test at the 5% level whether a coin tossed 20 times, landing heads 16 times, is biased towards heads, given $P(X \leq 14) = 0.9793$ .

Active recall

Recall the key points — then reveal.

Sources: Pearson Edexcel Level 3 Advanced GCE in Statistics (9ST0) Specification (Pearson Edexcel)

Type I and Type II errors#

AdvancedPearson Edexcel 9ST0, Topic 11 (Paper 2)

A test can be wrong in two distinct ways. A Type I error is rejecting

H_0

when it is actually true — a false alarm — and a Type II error is failing to reject

H_0

when it is actually false — a missed detection. The two are not symmetric, and which is worse depends on the context: a false alarm on a smoke detector is a nuisance, but a missed fire is a disaster.

The probability of a Type I error is controlled directly: for a discrete test it is the actual significance level, the probability under

H_0

of landing in the critical region. For the binomial test above,

P(\text{Type I}) = P(X \geq 15 \mid p = 0.5) = 0.0207

. Lowering the significance level reduces this probability but makes rejection harder, which increases the chance of a Type II error.

The probability of a Type II error, usually denoted

\beta

, is the probability of NOT landing in the critical region when a specific alternative value of the parameter is true. It can only be computed against a stated alternative, because 'the null is false' is not specific enough. Its complement

1 - \beta

is the power of the test, developed fully in a later topic.

There is an unavoidable trade-off: for a fixed sample size, reducing one error probability increases the other. The only way to reduce both at once is to gather more data. Recognising this trade-off, and choosing a significance level that reflects the relative costs of the two errors, is the mature judgement that hypothesis testing demands.

P(\text{Type I}) = P(\text{reject } H_0 \mid H_0 \text{ true}) = \alpha_{\text{actual}}, \qquad P(\text{Type II}) = \beta

The two error probabilities

$\alpha$ is controlled by the significance level; $\beta$ needs a specific alternative; power is $1 - \beta$ .

Worked example

Probability of a Type I error

For the test $H_0$ : $p = 0.5$ against $H_1$ : $p > 0.5$ with $n = 20$ and critical region $X \geq 15$ , state the probability of a Type I error and describe a Type II error.

Type I probability
$P(\text{Type I}) = P(X \geq 15 \mid p = 0.5) = 0.0207$ .
Meaning
It is the chance of wrongly declaring the coin biased when it is in fact fair.
Type II error
A Type II error here is failing to detect a genuinely biased coin, i.e. observing $X \leq 14$ when in truth $p > 0.5$ .

Result: $P(\text{Type I}) = 0.0207$ ; a Type II error is failing to reject $H_0$ (observing $X \leq 14$ ) when the coin is really biased.

Typical mistakes

Swapping the two error types (false alarm versus missed detection).
Trying to compute a Type II error probability without a specific alternative value.

Active revision

For the binomial test with critical region $X \geq 15$ under $H_0$ : $p = 0.5$ ( $n = 20$ ), state the probability of a Type I error and describe a Type II error in context.

Active recall

Recall the key points — then reveal.

Sources: GCE AS and A level subject content (Statistics) (Department for Education / Ofqual)

References & sources

Sources

Pearson Edexcel

Pearson Edexcel Level 3 Advanced GCE in Statistics (9ST0) Specification

Department for Education / Ofqual

GCE AS and A level subject content (Statistics)

Introduction to hypothesis testing

5 sections~13 min reading time3 competenciesLevel Standard 2 · Advanced 3

Introduction to hypothesis testing

Key points

Framing a test

Key points

Setting up a two-tailed test

Key points

Deciding from a p-value

Key points

A one-tailed binomial test

Key points

Probability of a Type I error

Sources

Introduction to hypothesis testing

Key points

Framing a test

Key points

Setting up a two-tailed test

Key points

Deciding from a p-value

Key points

A one-tailed binomial test

Key points

Probability of a Type I error

Sources