Statistical Significance

Statistical significance is a mathematical measure of how likely it is that the difference between two variants in an A/B test is real and not due to random chance. It is the gatekeeper between a lucky streak and a genuine optimization win.

Definition

Statistical significance in CRO is the probability that the observed difference between a control and a variant is not caused by random variation. It is expressed as a p-value or a confidence level. The industry standard is 95% confidence, meaning there is a 5% or less chance that the result is a false positive. Statistical significance does not tell you whether the result is important. It tells you whether the result is real. Practical significance (the actual business impact) is a separate question.

The 95% Rule and Why It Exists

The 95% confidence threshold is the widely accepted standard for A/B testing. It means that if you ran the same experiment 100 times, the variant would beat the control in at least 95 of those runs. The 5% tolerance for false positives is a balance between being too aggressive (calling winners that are not real) and too conservative (missing real improvements). Lower-traffic sites sometimes use 90% or 80% thresholds, but this increases the risk of implementing changes that do not actually help.

Statistical Power and Sample Size

Statistical significance requires sufficient sample size. A test with too few visitors is underpowered. It cannot detect a real difference even if one exists. Use a sample size calculator before launching any test. Inputs: baseline conversion rate, minimum detectable effect (how much improvement you want to detect), and desired significance level (usually 95%). The output is the number of visitors per variant needed. Testing without calculating sample size is guessing.

Significance vs. Practical Significance

A result can be statistically significant but practically meaningless. A 0.1% lift with 95% confidence is statistically significant. It is also not worth implementing. Practical significance asks: is the improvement large enough to justify the engineering effort, the risk, and the opportunity cost? Always evaluate both significance and effect size before rolling out a change.

Note

The 'peeking problem' is real. Every time you check your test results mid-run, you increase the false positive rate. Use a sequential testing method or a fixed-horizon test with a pre-determined stop date. Bayesian methods are an alternative to frequentist statistics and naturally handle the peeking problem.

In code

<!-- Simplified sample size calculation -->
<script>
  function calculateSampleSize(baselineRate, minimumEffect, significance, power) {
    // Simplified version using z-scores
    const z_significance = significance === 0.95 ? 1.96 : 1.645;
    const z_power = power === 0.8 ? 0.84 : 1.28;

    const p1 = baselineRate;
    const p2 = baselineRate * (1 + minimumEffect);
    const p_avg = (p1 + p2) / 2;

    const numerator = Math.pow(z_significance * Math.sqrt(2 * p_avg * (1 - p_avg))
                     + z_power * Math.sqrt(p1 * (1 - p1) + p2 * (1 - p2)), 2);
    const denominator = Math.pow(p2 - p1, 2);

    return Math.ceil(numerator / denominator);
  }

  // Example: 5% baseline, want to detect 20% improvement, 95% significance, 80% power
  console.log(calculateSampleSize(0.05, 0.2, 0.95, 0.8));
  // Output: ~5,000 visitors per variant
</script>

Common questions

Q: Can I trust results that are 90% significant?

A: 90% confidence means a 10% chance of a false positive. Use it for low-cost, low-risk changes. Use 95% or higher for changes that require significant engineering effort or affect core revenue.

Q: What is the peeking problem?

A: Every time you check your A/B test results and consider stopping early, you increase the chance of making a false positive decision. The more you peek, the more likely you are to see a temporary 'winner' that vanishes with more data.

Q: Do I need statistical significance for every change?

A: No. Use significance for high-impact changes that you plan to implement permanently. For low-risk changes (fixing a broken link, updating a copyright date), you do not need a test.

Key takeaways:

• Statistical significance tells you if the result is real, not if it matters

• 95% confidence is the industry standard for A/B testing

• Calculate sample size before the test. Never skip this step

• Practical significance (effect size) matters as much as statistical significance

• Do not peek at results mid-test. It invalidates the statistics