Statistics · Inference

Hypothesis testing & Type I / Type II errors

Every A/B test an analyst runs is a hypothesis test. Interviewers want three things: the 2×2 error table, what a p-value means, and which error is worse in a given business case.

Your win todayDraw the 2×2 error table from memory, put α, β and power in the right boxes, and give a business example of each error.

The setup: a courtroom

H₀: the null hypothesis

"Nothing's going on." The new checkout button changes nothing. Like "innocent until proven guilty". You assume it until the evidence is strong.

H₁: the alternative hypothesis

"Something's going on." The new button changes conversion. It's what you're trying to find evidence for.

You never prove H₀. You either reject H₀ (the evidence is strong) or fail to reject H₀ (not enough evidence). A court says "not guilty", not "innocent".^{[OpenStax 9.1]}

The 2×2 table: memorise this picture

Type I = rejecting a true H₀. Type II = failing to reject a false H₀. α is the probability of a Type I error, β of a Type II, and 1 − β is the power of the test.^{[OpenStax 9.2]} Memory hook: Type I = crIing wolf (false alarm). Type II = sIIence when the wolf is really there.

Why you can't kill both errors

Results to the right of the cut-off lead you to reject H₀. The red sliver is the chance of doing that when H₀ is true (α). The teal area is the chance of not rejecting when H₁ is true (β).

The p-value, said correctly

p-value = the probability of seeing data at least this extreme if H₀ were true. A small p means "this would be surprising if nothing were going on".

p ≤ α (usually 0.05) → reject H₀ ("statistically significant").
p > α → fail to reject H₀.
❌ It is not "the probability that H₀ is true", and it says nothing about how big the effect is.

The 5-step routine

State H₀ and H₁.
Choose α (0.05 is standard. 0.01 when false alarms are costly).
Pick a test and compute the p-value. (Two group means → t-test. 3+ group means → ANOVA. Two categorical variables → chi-square.)
Compare p with α.
Conclude in business words: "the new button lifted conversion. Roll it out."

Which error is worse? It depends on the cost Spam filter: a Type I error (a real email sent to spam) is worse, so use a stricter α.
Cancer screening / fraud detection: a Type II error (a missed case) is worse, so you accept more false alarms.

Say it in the interview"In a hypothesis test, H₀ says there's no effect. A Type I error is rejecting H₀ when it's actually true: a false positive, with probability α, usually 5%. A Type II error is failing to reject H₀ when it's false: a false negative, with probability β. Power is 1 − β. For a fixed sample, lowering one raises the other, so I pick α based on which mistake costs the business more. A larger sample reduces both."

Retrieval check

What is a Type I error?

What does α = 0.05 mean?

p = 0.03 and α = 0.05. What's your decision?

Cancer screening: which error is usually worse?

How do you reduce both α and β together?

Type it

Power = 1 − ?

An A/B test gives p = 0.12 with α = 0.05. What's the decision? (A few words.)

Your turn

On paper, draw the 2×2 table from memory with α, β, 1−α, 1−β. Then check it against this page.
Write one Type I and one Type II example for: an A/B test, a spam filter, a fraud model.
Explain the p-value to an imaginary non-technical manager in 2 sentences, out loud.
Answer out loud: "why don't we just set α = 0.0001?" (Hint: look at the curves.)
Optional: read OpenStax 9.2 and do 2 of its practice problems.

Go deeper

Primary source: OpenStax Introductory Business Statistics §9.2: Outcomes and the Type I and Type II Errors. Then §9.1 on H₀/H₁.

💬 Ask your teacher: "walk me through a t-test with numbers" or "one-tailed vs two-tailed?"