CRO

Home › Glossary › Cro· 397 ·

Statistical Significance

/stəˈtɪstɪkəl sɪɡˈnɪfɪkəns/noun phrase

Filed underCroAnalytics

In brief · quick answer

Statistical significance is a mathematical measure of how likely it is that the difference between two variants in an A/B test is real and not due to random chance. It is the gatekeeper between a lucky streak and a genuine optimization win.

§ 1 Definition

Statistical significance in CRO is the probability that the observed difference between a control and a variant is not caused by random variation. It is expressed as a p-value or a confidence level. The industry standard is 95% confidence, meaning there is a 5% or less chance that the result is a false positive. Statistical significance does not tell you whether the result is important. It tells you whether the result is real. Practical significance (the actual business impact) is a separate question.

§ 2 The 95% Rule and Why It Exists

The 95% confidence threshold is the widely accepted standard for A/B testing. It means that if you ran the same experiment 100 times, the variant would beat the control in at least 95 of those runs. The 5% tolerance for false positives is a balance between being too aggressive (calling winners that are not real) and too conservative (missing real improvements). Lower-traffic sites sometimes use 90% or 80% thresholds, but this increases the risk of implementing changes that do not actually help.

§ 3 Statistical Power and Sample Size

Statistical significance requires sufficient sample size. A test with too few visitors is underpowered. It cannot detect a real difference even if one exists. Use a sample size calculator before launching any test. Inputs: baseline conversion rate, minimum detectable effect (how much improvement you want to detect), and desired significance level (usually 95%). The output is the number of visitors per variant needed. Testing without calculating sample size is guessing.

§ 4 Significance vs. Practical Significance

A result can be statistically significant but practically meaningless. A 0.1% lift with 95% confidence is statistically significant. It is also not worth implementing. Practical significance asks: is the improvement large enough to justify the engineering effort, the risk, and the opportunity cost? Always evaluate both significance and effect size before rolling out a change.

§ 5 Note

The 'peeking problem' is real. Every time you check your test results mid-run, you increase the false positive rate. Use a sequential testing method or a fixed-horizon test with a pre-determined stop date. Bayesian methods are an alternative to frequentist statistics and naturally handle the peeking problem.

§ 6 In code

<!-- Simplified sample size calculation -->
<script>
  function calculateSampleSize(baselineRate, minimumEffect, significance, power) {
    // Simplified version using z-scores
    const z_significance = significance === 0.95 ? 1.96 : 1.645;
    const z_power = power === 0.8 ? 0.84 : 1.28;

    const p1 = baselineRate;
    const p2 = baselineRate * (1 + minimumEffect);
    const p_avg = (p1 + p2) / 2;

    const numerator = Math.pow(z_significance * Math.sqrt(2 * p_avg * (1 - p_avg))
                     + z_power * Math.sqrt(p1 * (1 - p1) + p2 * (1 - p2)), 2);
    const denominator = Math.pow(p2 - p1, 2);

    return Math.ceil(numerator / denominator);
  }

  // Example: 5% baseline, want to detect 20% improvement, 95% significance, 80% power
  console.log(calculateSampleSize(0.05, 0.2, 0.95, 0.8));
  // Output: ~5,000 visitors per variant
</script>

§ 7 Common questions

Q. Can I trust results that are 90% significant?: A. 90% confidence means a 10% chance of a false positive. Use it for low-cost, low-risk changes. Use 95% or higher for changes that require significant engineering effort or affect core revenue.
Q. What is the peeking problem?: A. Every time you check your A/B test results and consider stopping early, you increase the chance of making a false positive decision. The more you peek, the more likely you are to see a temporary 'winner' that vanishes with more data.
Q. Do I need statistical significance for every change?: A. No. Use significance for high-impact changes that you plan to implement permanently. For low-risk changes (fixing a broken link, updating a copyright date), you do not need a test.

Key takeaways

Statistical significance tells you if the result is real, not if it matters
95% confidence is the industry standard for A/B testing
Calculate sample size before the test. Never skip this step
Practical significance (effect size) matters as much as statistical significance
Do not peek at results mid-test. It invalidates the statistics

How Atomic Glue helps

Atomic Glue handles the statistical rigor of your A/B testing program. We calculate sample sizes, manage multiple testing corrections, and ensure every result is statistically valid before implementation. Analytics & Tracking

Get in touch

# Statistical Significance

Category: Cro (also: Analytics)

Author: Atomic Glue Research Team

## Definition

## The 95% Rule and Why It Exists

## Statistical Power and Sample Size

## Significance vs. Practical Significance

## Note

## In code

<!-- Simplified sample size calculation -->
<script>
  function calculateSampleSize(baselineRate, minimumEffect, significance, power) {
    // Simplified version using z-scores
    const z_significance = significance === 0.95 ? 1.96 : 1.645;
    const z_power = power === 0.8 ? 0.84 : 1.28;

    const p1 = baselineRate;
    const p2 = baselineRate * (1 + minimumEffect);
    const p_avg = (p1 + p2) / 2;

    const numerator = Math.pow(z_significance * Math.sqrt(2 * p_avg * (1 - p_avg))
                     + z_power * Math.sqrt(p1 * (1 - p1) + p2 * (1 - p2)), 2);
    const denominator = Math.pow(p2 - p1, 2);

    return Math.ceil(numerator / denominator);
  }

  // Example: 5% baseline, want to detect 20% improvement, 95% significance, 80% power
  console.log(calculateSampleSize(0.05, 0.2, 0.95, 0.8));
  // Output: ~5,000 visitors per variant
</script>

## Common questions

Q: Can I trust results that are 90% significant?

A: 90% confidence means a 10% chance of a false positive. Use it for low-cost, low-risk changes. Use 95% or higher for changes that require significant engineering effort or affect core revenue.

Q: What is the peeking problem?

A: Every time you check your A/B test results and consider stopping early, you increase the chance of making a false positive decision. The more you peek, the more likely you are to see a temporary 'winner' that vanishes with more data.

Q: Do I need statistical significance for every change?

A: No. Use significance for high-impact changes that you plan to implement permanently. For low-risk changes (fixing a broken link, updating a copyright date), you do not need a test.

## Key takeaways

Statistical significance tells you if the result is real, not if it matters
95% confidence is the industry standard for A/B testing
Calculate sample size before the test. Never skip this step
Practical significance (effect size) matters as much as statistical significance
Do not peek at results mid-test. It invalidates the statistics

## Related entries

Last updated July 2026. Permalink: atomicglue.co/glossary/statistical-significance