Incrementality Testing

Incrementality testing measures the causal impact of a marketing campaign by comparing outcomes between a group exposed to the campaign and a randomly selected holdout group that was not exposed. It answers the question: did this campaign actually drive additional conversions that would not have happened anyway?

Category: Analytics (also: Marketing, Cro)

Author: Atomic Glue Analytics Team

## Definition

Incrementality testing is an experimental method that measures the true causal lift generated by a marketing campaign, campaign, or channel. The core approach involves randomly splitting your target audience into a test group (exposed to the campaign) and a control group (not exposed, or a holdout). The difference in conversion rates between the two groups represents the incremental conversions directly caused by the campaign. This is fundamentally different from attribution models, which assign credit among observed touchpoints but cannot distinguish correlation from causation. A campaign may show 'last-click' conversions in an attribution report, but were those conversions incremental (would the user have converted anyway without the ad?), or would they convert through another channel? Incrementality testing isolates the causal signal from the noise of organic behavior.

## How incrementality testing works

A standard geo-based or user-based incrementality test starts with randomization. In user-based tests, eligible users are randomly split into test (see the campaign) and control (do not see the campaign) groups. In geo-based tests, geographic regions are randomized. The control group functions as a counterfactual: what would have happened if the campaign did not run. The difference in conversion rate between test and control is the incremental lift. Results are typically reported as incremental conversions (the extra conversions caused by the campaign) and incremental ROAS (the extra revenue divided by the campaign spend).

## When to use incrementality testing

Use incrementality testing when you need to validate whether a channel or campaign is actually driving new business, especially for upper-funnel channels where attribution models are weakest (TV, podcasts, display, social awareness). Use it before scaling a new channel to confirm the channel is additive, not just cannibalizing existing demand. Use it to calibrate lift to your attribution models and MMM. Do not use it for everything. Incrementality tests are expensive (you must withhold advertising from a control group), take time (2-4 weeks minimum), and work best for high-spend channels where the opportunity cost of testing is justified.

## Incrementality vs attribution vs MMM

Attribution answers: 'which touchpoint got the click that preceded this conversion?' MMM answers: 'historically, how much revenue did each channel category drive?' Incrementality answers: 'what is the causal effect of turning this campaign on or off?' Attribution is observational and correlational. MMM is statistical and aggregate. Incrementality is experimental and causal. Each answers a different question. The most sophisticated measurement frameworks triangulate all three: attribution for daily optimization, MMM for strategic allocation, and incrementality for causal validation.

## Note

Misconception: if an attribution report shows 100 conversions from a campaign, those are all incremental. Not necessarily. Some of those 100 users may have converted without seeing the ad (they were going to buy anyway through a direct visit, organic search, or another channel). Some may have converted through a different channel if the ad had not existed. Incrementality testing measures how many of those 'attributed' conversions are actually new business that would not have occurred without the campaign. Studies across industries typically find that only 30-70% of attributed conversions are actually incremental, depending on the channel and audience saturation.

## Common questions

Q: How is incrementality testing different from A/B testing?

A: A/B testing typically tests changes to your website or app experience (which headline converts better). Incrementality testing specifically tests the causal effect of advertising exposure. Both use randomized control groups, but the treatment differs: A/B testing changes on-site experience; incrementality testing changes ad delivery.

Q: What is a holdout group?

A: A holdout group is a randomly selected portion of your target audience that is intentionally not shown your campaign. By comparing the conversion rate of the holdout group to the exposed group, you isolate the incremental impact of the campaign.

Q: How long should an incrementality test run?

A: At least 2-4 weeks to capture the full conversion cycle and avoid primacy/recency effects. Longer for longer purchase cycles. The test needs to run long enough for the control group's natural behavior to stabilize.

## Key takeaways

• Incrementality testing measures causal lift using randomized holdout groups
• Answers 'did this campaign actually cause conversions that would not have happened anyway?'
• Only 30-70% of attributed conversions are typically incremental
• Use for high-spend channels where attribution uncertainty is highest
• Best used alongside attribution and MMM for a complete measurement strategy

## Related entries

• [Attribution Modeling (First-Click, Last-Click, Multi-Touch, Data-Driven)](atomicglue.co/glossary/attribution-modeling)
• [Marketing Mix Modeling (MMM)](atomicglue.co/glossary/marketing-mix-modeling-mmm)
• [ROAS](atomicglue.co/glossary/roas)

Last updated July 2026. Permalink: atomicglue.co/glossary/incrementality-testing