AMC incrementality test: holdout groups, control group and evaluation

In short: An incrementality test measures how much demand your ads really create. Amazon DSP provides ten customer holdout groups, which you split 50/50 across both sides. The control side is not ad-free: Amazon calls it an order that replicates your current campaign. Both sides see advertising.

Incrementality, attributed performance and lift

An incrementality test answers a causal question: how many purchases would not have happened without your advertising? Attributed performance answers a different question: which purchases can be assigned to your advertising? The two numbers diverge in practice, because a customer would buy without your ad anyway. That difference, the incremental share, is exactly what an incrementality test isolates.

That is where the common error lies: a high ROAS or a positive lift between two periods is no proof of incrementality. Amazon does not recommend this comparison, because market conditions and other factors vary over time. A before-and-after comparison of campaign performance therefore measures more than the effect of your advertising.

StatementWhat it meansIs it causal?
Attributed performancePurchases that Amazon Ads assigns to an ad view or a clickNo, that is an assignment
Observed liftDifference between two periods or between test and control sideOnly with a randomized holdout
IncrementalityShare of purchases that would not have happened without your advertisingYes, provided the holdout was cleanly randomized

The holdout: what the control side really is

The holdout is created on the advertising platform, not in the analysis. Amazon DSP provides ten customer holdout groups. Each group contains 10% of randomly assigned Amazon customers and forms the technical basis for controlled experiments. You split these groups 50/50 across control and test side: five and five groups.

At this point it is decided what your test measures. Amazon describes the control side explicitly as "one control order that replicates your current campaign (i.e., no changes)". The control group therefore sees advertising, and the same advertising as before the test. Both sides run with ads; only the setting under test differs. Amazon compares the effect of that single change, not the effect of advertising versus no advertising at all.

That is the decisive difference to the classic holdout logic that many advertising-error articles describe. A test you set up in Amazon DSP measures the response to a specific change on an already running campaign whose two sides both deliver ads. A statement about what happens without your advertising altogether does not follow from it.

Amazon also states when a holdout is not the right route: "We do not recommend comparing performance before and after implementing bid adjustments as market conditions and other factors vary over time." A before-and-after comparison of campaign performance therefore does not replace the control group.

  • **Ten groups, two sides.** You split users 50/50 between control and test: five and five groups go to one side and the other five to the other. Amazon recommends randomizing the group assignments for each new test to avoid potential cross contamination between experiments or any long-term bias.
  • **Structurally identical sides.** You create two identical orders that differ only in the setting under test, and turn off automated budget optimization so the line items cannot affect each other in delivery.
  • **Assign the audiences cleanly.** In the audience picker you search for "Customer holdout group", add the selected groups to each line item and combine them with OR logic within the groups and AND logic with the rest of your targeting.
  • **Comparable bid reports.** You associate the same bid adjustment terms with the control line item at a value of 1.0. That does not change any bidding, but it yields the same report and therefore a comparable baseline.

The no-ads baseline: retail purchases instead of ads

If you really want to know what happens without your advertising, you need a measure that counts independently of any ad. In AMC that is what the table amazon_retail_purchases from the Amazon Retail Purchases dataset (ARP) is for. Amazon describes its scope as "Entire purchase activity in Amazon store, regardless of advertising" and deliberately sets it apart from the advertising-centric conversion tables, which are "Advertising-centric (can include promoted, brand-halo, and brand-owned ASIN items)".

This separation is the real lever for an incrementality evaluation: the attributed-events tables deliver the advertising-centric view, ARP delivers the retail-side view of the same population. The difference between the two is the incremental quantity you are looking for. Amazon does point out the differing time windows: ARP covers up to 60 months of purchase history, while other data views retain between 13 months and five years. When you join the two worlds you have to account for these windows, otherwise you get inaccurate results.

ARP is a paid feature and therefore not part of every AMC entitlement. Amazon names a 60-day free trial and states that the dataset can be queried at no cost from June 1, 2026 through December 31, 2026 in every region where it is available. Check before you plan whether your marketplace and your entitlement actually include the dataset.

SELECT
  line_item,
  COUNT(DISTINCT user_id) AS reached_users,
  SUM(impressions) AS impressions,
  SUM(clicks) AS clicks,
  SUM(purchases) AS purchases,
  SUM(total_product_sales) AS product_sales
FROM
  amazon_attributed_events_by_conversion_time
WHERE
  conversion_event_date >= '2026-10-01'
  AND conversion_event_date <= '2026-10-31'
GROUP BY
  line_item
ORDER BY
  line_item
Example: separate both sides of the test through the line item key

The query counts the reached users per line item and sums the metrics. user_id is in the query only because COUNT(DISTINCT user_id) aggregates the column without exposing it; it never appears in the final SELECT. The table delivers exclusively the advertising-centric view, that is the side of the test on which the ads run.

**Our assessment:** AMC does not reconstruct the randomization. The statement "this line item was the control group" does not come from the data, it comes from your test setup in DSP. AMC provides the comparable metrics for both sides; the causal interpretation is yours to carry.

Thresholds, intervals and the limits of the evaluation

  • **Aggregation thresholds:** Amazon classifies every field. Level LOW requires at least two distinct users per row, level MEDIUM and HIGH require at least one hundred distinct users. Fields at level VERY HIGH, such as user_id, may never appear in the output, only aggregated.
  • **High-threshold dimensions:** postal_code and city_name sit at level HIGH. They may be output but not filtered with static values; a filter such as WHERE postal_code in (10001, 10011) is rejected by AMC.
  • **Aggregate instead of output:** COUNT(DISTINCT user_id) is the permitted way to count users without exposing the column itself. A query that outputs user_id in the final SELECT is rejected.
  • **Attribution window:** The API exposes the metric intervals _1d, _7d, _14d and _30d. In AMC you aggregate over any interval up to the attribution window, which is 14 days; for an individual interval such as 9 days you set the time difference between events in the WHERE condition. The attribution window is set by the campaign and cannot be changed in AMC.

How this sits next to your existing evaluations

An incrementality test does not replace watching the campaign trajectory, it answers a different question. Comparing both views belongs in the evaluation: an observed lift can arise without a control group, and a holdout without observation says nothing about how the campaign developed.

Before you start, also check whether your question fits the DSP test at all. Amazon describes the procedure for bid adjustments, that is for a concrete change to a running campaign. If you want to measure the entire effect of advertising against an ad-free baseline, the combination of the DSP holdout and the ARP comparison is the path that answers the question.

Step by step

  1. Set up the test. Create two identical orders in Amazon DSP that differ only in the setting under test, and turn off automated budget optimization. The control side replicates your current campaign, so it is not ad-free.
  2. Randomize the groups. Split the ten customer holdout groups 50/50 across control and test side, five and five groups, and randomize the assignment again for each new test.
  3. Assign the audiences. Add the customer holdout groups to each line item in the audience picker, with OR logic within the groups and AND logic with the rest of your targeting.
  4. Secure the runtime. Let the test run until enough budget and time are available for statistically significant results. Amazon names both points explicitly as a prerequisite.
  5. Evaluate in AMC. Read both line items through line_item from the table amazon_attributed_events_by_conversion_time and compare reached users, clicks and purchases.
  6. Add the baseline. Pull the purchases for the same population from amazon_retail_purchases, the ones that occurred independently of any advertising. Comparing the two views yields the incremental quantity.

Frequently asked questions

What is the difference between incrementality and attributed performance?

Attributed performance counts purchases that Amazon Ads assigns to an ad view or click. Incrementality is the share of those purchases that would not have happened without your advertising. The attributed-events tables deliver the first quantity, amazon_retail_purchases the second.

Does the control group in Amazon DSP see no advertising?

No. Amazon describes the control side as an order that replicates your current campaign. Both sides deliver ads, only the setting under test differs. The control side is not ad-free.

Can AMC evaluate the holdout test itself?

AMC evaluates the results of both line items, but the randomization is created in Amazon DSP. Without the ten customer holdout groups, every evaluation remains a description of attributed performance.

Is a before-and-after comparison enough?

No. Amazon does not recommend this comparison because market conditions and other factors vary over time. A before-and-after comparison can show an effect that the advertising did not cause.

What makes the evaluation in AMC fail?

At the aggregation thresholds: level LOW requires at least two distinct users per row, MEDIUM and HIGH at least one hundred. Too few users mean NULL rows instead of a number. Plan the period generously.

Sources: Amazon Ads API: How to test DSP bid adjustments (A/B test with Customer holdout groups) · Amazon Ads API: Amazon Retail Purchases (entire purchase activity regardless of advertising) · Amazon Ads API: AMC data sources overview (arbitrary attribution intervals, attribution window) · Amazon Ads API: Data aggregation thresholds in AMC · Amazon Ads API: AMC SQL basics (supported JOIN clauses) · Amazon Ads API: amazon_attributed_events_by_conversion_time (retrieved 2026-11-15)

Keep reading: AMC cheat sheet · NTB analysis · Audience Segment Insights · Data source ARP