Hypothesis Testing: p-Values, Statistical Significance & Common Misconceptions

Hypothesis testing is how we make decisions from data while controlling the risk of being wrong. It's the foundation of A/B testing, clinical trials, and data-driven product decisions — yet it's one of the most misunderstood concepts in statistics.

The Core Idea

You have a question: "Does this new feature increase conversion?" You observe data. But randomness exists — maybe conversion went up by pure chance. Hypothesis testing gives you a framework to decide: is this result real, or noise?

Step 1 — Define Two Hypotheses

H0 is the default assumption. You assume nothing is happening until the data convinces you otherwise. This is like "innocent until proven guilty" — H0 is innocence, the data is evidence.

Step 2 — Collect Data & Compute a Test Statistic

You run your experiment and compute a test statistic — a number that summarizes how far your observed result is from what H0 would predict.

Example: You test a new checkout button on 1,000 users per group.

The observed difference is +2.5 percentage points. But could this happen by random chance if the button has no real effect?

Step 3 — The p-Value

p-value = the probability of observing a result at least as extreme as yours, assuming H0 is true.

In our example: "If the new button truly has NO effect on conversion (H0 is true), what's the probability of seeing a +2.5 pp difference (or more) just by chance?"

p-value answers: "How surprising is this data IF nothing changed?"

     H0 is true (no effect)
     ↓
     What would random data look like?
     ↓
     How extreme is MY data compared to that?
     ↓
     p-value = probability of being this extreme or more

What p-value IS

• The probability of the data (or more extreme) given H0
• A measure of how incompatible your data is with the null hypothesis
• Written as: P(data | H0)

What p-value IS NOT

Think of it like a fire alarm. The p-value tells you "this alarm would ring by accident only 3 % of the time." It does NOT tell you "there's a 97 % chance there's a fire."

Step 4 — Decision Rule (α threshold)

Before looking at the data, you choose a significance level α (typically 0.05):

"Fail to reject H0" is NOT "H0 is true." It means: we don't have enough evidence to say otherwise.

Types of Errors

Statistical Power & Sample Size

Power = probability of correctly detecting a real effect.

Rule of thumb for A/B tests:

Detecting small effects requires enormous samples. This is why you should always define the minimum effect that matters before running an experiment.

Confidence Intervals > p-Values

A 95 % confidence interval tells you more than a p-value:

Bayesian Alternative

Bayesian hypothesis testing gives you what most people actually want: the probability that the treatment is better.

Bayesian A/B testing is increasingly adopted (Statsig, GrowthBook, VWO) because the results are easier to interpret and communicate.

Common Pitfalls Cheat Sheet

Decision Framework

         Is this a randomized experiment?
         /                              \
       Yes                               No
        |                                 |
  Use hypothesis testing          Use causal inference
  (t-test, chi-square, etc.)     (diff-in-diff, IV, etc.)
        |
  Is your sample large enough?
  (power ≥ 80%)
  /              \
Yes               No
 |                 |
Run test,      Increase sample
interpret        or accept
with CI          lower power

Resources

• Seeing Theory — Interactive visualization of probability and statistics
• Statistics Done Wrong — Common statistical mistakes (free online book)
• Trustworthy Online Controlled Experiments — Kohavi, Tang & Xu (the A/B testing bible)
• Bayesian Methods for Hackers — Practical Bayesian stats
• Khan Academy — Hypothesis Testing — Step-by-step fundamentals

:::