LinkedIn
Mailchimp
GA4
Instagram
Google Ads
Definition
A holdout test withholds ads from a comparable group and compares its results with the group that saw them, to estimate the lift the ads caused.
How to use it
The test only works if the groups are comparable and the period is long enough to collect enough orders. Splitting by region or by audience are common designs, and each has its own pitfalls.
Record the split, the dates, the metric and the decision rule before the test starts, so the result cannot be bent afterwards.
