Slack
Instagram
Shopify
Meta Ads
Definition
A hold-out audience is a group of people deliberately kept out of a campaign or an ad, so that their buying can be compared with the buying of people who were reached. The group is the tool, and the comparison it allows is the result, which shows what the ads added.
Relative lift = (rate in the reached group - rate in the hold-out group) / rate in the hold-out groupExample with made-up numbers: Reached group: 50,000 people, 600 purchases, rate 1.2 percent. Hold-out group: 5,000 people, 45 purchases, rate 0.9 percent. Relative lift = (1.2 - 0.9) / 0.9 = 33 percent. The group sizes are made up, and 45 purchases is too few to rely on the figure alone, so the test would need to run longer.
How to use it
The hold-out group must be chosen at random, or by a rule that is applied the same way to everyone, and it must stay excluded for the whole test. If people from the group are added to the campaign later, or if the group is built from people who already bought, the comparison stops meaning anything. Write the rule down before the test starts, not after the numbers come in.
Size the group so that a difference of the size you care about could show up at all. A small group gives noisy results, and a large group gives up sales the business would otherwise have made. Agree the length of the test, the metric and the decision it will drive, and keep the exclusion in the account settings so that it is not removed by accident.
