CRO

Home › Glossary › Cro· 4 ·

A/B Testing

/eɪ biː ˈtɛstɪŋ/noun phrase

Filed underCroAnalytics

In brief · quick answer

A/B testing is a controlled experiment that compares two versions of a web page or element to determine which performs better against a specific goal. It removes guesswork from optimization. If you are making changes without testing, you are gambling, not optimizing.

§ 1 Definition

A/B testing (also called split testing or bucket testing) is a randomized experiment with two variants: A (the control, usually the current version) and B (the variant, the proposed change). Traffic is split evenly between the two variants. The variant that achieves a statistically significant improvement on the target metric wins. A/B testing is the gold standard of CRO because it isolates the effect of a single change and provides causal evidence that the change caused the difference.

§ 2 The Anatomy of a Valid A/B Test

A valid A/B test requires: (1) A single, measurable hypothesis with a clear success metric. (2) Random assignment of users to variants. (3) A predetermined sample size calculated before the test starts. (4) A significance threshold (usually 95%) set before the test. (5) A fixed duration calculated to account for day-of-week and time-of-day effects. (6) The discipline to not peek at results and stop early. Violating any of these invalidates the test.

§ 3 Common A/B Testing Mistakes

The most common mistakes: stopping the test early because results look promising (peeking fallacy), testing too many changes at once (multivariate confusion), running the test for too few days (not capturing weekly cycles), using insufficient sample size (low statistical power), testing changes that are too small to matter (button color obsessions), and ignoring segmentation (a change that helps mobile may hurt desktop).

§ 4 The 'Button Color' Trap

A/B testing is often trivialized as 'button color testing'. Button color matters, but the impact is usually small (0-5%). The big wins come from testing structural changes: headline copy, value proposition, information architecture, offer structure, navigation removal, and form field reduction. Prioritize high-impact tests. Do not waste your traffic budget on 50 shades of blue.

§ 5 Note

Google Optimize was deprecated in 2023. Do not build new testing infrastructure on it. Modern alternatives: server-side feature flags (LaunchDarkly, Statsig), client-side libraries (GrowthBook, VWO, Optimizely), or a custom statistical engine. The tool matters less than the methodology.

§ 6 In code

<!-- A/B test implementation via Google Optimize (deprecated pattern) -->
<!-- Modern approach: use feature flags or server-side experimentation -->
<script>
  // Hypothetical server-side A/B test configuration
  const experiment = {
    id: 'exp_2026_hero_headline',
    variants: [
      { id: 'control', weight: 0.5, headline: 'Grow Your Revenue' },
      { id: 'variant', weight: 0.5, headline: 'Double Your Revenue in 30 Days' }
    ],
    // Deterministic assignment based on user ID
    getVariant: function(userId) {
      const hash = this.hashString(userId + this.id);
      return hash % 100 < 50 ? this.variants[0] : this.variants[1];
    },
    hashString: function(str) {
      let hash = 0;
      for (let i = 0; i < str.length; i++) {
        hash = ((hash << 5) - hash) + str.charCodeAt(i);
        hash |= 0;
      }
      return Math.abs(hash);
    }
  };
</script>

§ 7 Common questions

Q. How much traffic do I need to A/B test?: A. Enough to reach statistical significance within a reasonable timeframe. Use a sample size calculator. For a 20% relative improvement on a 5% conversion rate, you need roughly 5,000 visitors per variant. Lower traffic sites should use qualitative methods instead.
Q. How long should I run an A/B test?: A. Minimum one full business cycle (usually 7 days). Longer if you have weekly seasonality. Run until you reach the predetermined sample size, not until results look significant.
Q. What if the test shows no winner?: A. A null result is a result. It tells you the change did not matter. Learn from it. Document the hypothesis and move to the next test. Not every test produces a winner.

Key takeaways

A/B testing is the gold standard for causal evidence in CRO
Pre-register your hypothesis, sample size, and duration before the test starts
Do not peek at results and stop early. This invalidates the test
Test structural changes (headlines, offers, navigation) for big wins, not button colors

How Atomic Glue helps

Atomic Glue designs and runs A/B tests end-to-end. We handle the math, the methodology, and the interpretation. No peeking. No bias. Just causally valid results. Conversion Optimization services

Get in touch

# A/B Testing

Category: Cro (also: Analytics)

Author: Atomic Glue Research Team

## Definition

## The Anatomy of a Valid A/B Test

## Common A/B Testing Mistakes

## The 'Button Color' Trap

## Note

## In code

<!-- A/B test implementation via Google Optimize (deprecated pattern) -->
<!-- Modern approach: use feature flags or server-side experimentation -->
<script>
  // Hypothetical server-side A/B test configuration
  const experiment = {
    id: 'exp_2026_hero_headline',
    variants: [
      { id: 'control', weight: 0.5, headline: 'Grow Your Revenue' },
      { id: 'variant', weight: 0.5, headline: 'Double Your Revenue in 30 Days' }
    ],
    // Deterministic assignment based on user ID
    getVariant: function(userId) {
      const hash = this.hashString(userId + this.id);
      return hash % 100 < 50 ? this.variants[0] : this.variants[1];
    },
    hashString: function(str) {
      let hash = 0;
      for (let i = 0; i < str.length; i++) {
        hash = ((hash << 5) - hash) + str.charCodeAt(i);
        hash |= 0;
      }
      return Math.abs(hash);
    }
  };
</script>

## Common questions

Q: How much traffic do I need to A/B test?

A: Enough to reach statistical significance within a reasonable timeframe. Use a sample size calculator. For a 20% relative improvement on a 5% conversion rate, you need roughly 5,000 visitors per variant. Lower traffic sites should use qualitative methods instead.

Q: How long should I run an A/B test?

A: Minimum one full business cycle (usually 7 days). Longer if you have weekly seasonality. Run until you reach the predetermined sample size, not until results look significant.

Q: What if the test shows no winner?

A: A null result is a result. It tells you the change did not matter. Learn from it. Document the hypothesis and move to the next test. Not every test produces a winner.

## Key takeaways

A/B testing is the gold standard for causal evidence in CRO
Pre-register your hypothesis, sample size, and duration before the test starts
Do not peek at results and stop early. This invalidates the test
Test structural changes (headlines, offers, navigation) for big wins, not button colors

## Related entries

Last updated July 2026. Permalink: atomicglue.co/glossary/ab-testing