Imagine a restaurant that pays a commission to whoever is holding the door when a diner walks in. Within a week there would be a crowd of people at the entrance, each claiming to have brought in the customers. The dining room would be no fuller than before.
That is roughly how last-click attribution works. It records which ad a customer touched most recently before buying and gives that ad the credit. It answers the question who was nearest the sale? It does not answer the one you care about: would the sale have happened without the ad?
Why the two answers differ
Some advertising reaches people who were already on their way to you. Three familiar examples:
- Brand search. Someone types your company name into a search engine and clicks the ad above your own organic listing. The ad gets a conversion. Most of those people were coming anyway.
- Retargeting. A shopper with a full basket is shown your banner for a week. When they finally check out, the banner is credited. It may have helped. It may simply have been in the room.
- Coupon sites. A customer at checkout searches for a discount code, finds one, and returns. An affiliate earns commission on a decision that had already been made.
None of these tactics is worthless. All of them look better in an attribution report than they are, because attribution rewards proximity. Meanwhile the video that first made someone aware of you, weeks earlier, often gets no credit at all.
Attribution rewards the last thing a customer touched. Incrementality rewards the thing that changed their mind.
The idea of a holdout
Incrementality borrows its method from medicine. Take a group of people, withhold the treatment from a random portion, and compare outcomes. The difference between the two groups is what the treatment caused. Everything else, such as seasonality, a competitor's sale or the weather, affects both groups equally and cancels out.
In advertising the "treatment" is exposure to a campaign. If the exposed group buys at a noticeably higher rate than the holdout, the campaign is doing real work. If the two groups buy at about the same rate, you have been paying to stand at the door.
Three tests most advertisers can run
1. The email holdout
Pick an automated flow, such as cart recovery. Randomly exclude a small share of eligible recipients for a few weeks. Compare revenue per person between those who received the emails and those who did not. It costs almost nothing, needs no special tools, and regularly surprises people in both directions.
2. The geo test
Choose a set of comparable regions. Switch a channel off, or up, in some and leave the others unchanged. Compare sales trends between the two sets. This works for channels where individual tracking is poor, such as connected TV, audio or out-of-home, because it relies only on where sales happen.
3. The platform lift study
Several large ad platforms offer built-in experiments that withhold ads from a random control group and report the difference. They are convenient and properly randomised. Remember that the platform is marking its own exam, so treat the result as one input and check it against your own sales data.
What good looks like
| Before the test | During | After |
|---|---|---|
| Write down the decision the result will inform, and what outcome would change it. | Leave it alone. Run for at least one full purchase cycle. Do not stop early because the numbers look good. | Report the difference with its uncertainty. A wide range is an honest finding, not a failure. |
Living with both
Attribution is not the enemy. It is fast, cheap and good for choosing between two ads in the same campaign. The mistake is using it to decide how much a whole channel deserves. Use attribution to steer within a channel, and experiments to set the budget between them. When the two disagree, believe the experiment.
If you would like help designing a first test, our measurement practice does exactly this.