Methodology

How to verify AI ad optimization actually saved you money (use a control group)

Every AI ad tool claims it "improved performance." The honest question is: compared to what? Here is why a self-reported "+X%" is not trustworthy, and how a randomized holdout produces a savings report a CFO can sign.

June 3, 2026·6 min read·DEXUN AdWhiz engineering

Every AI ad tool will tell you it "improved performance / saved you money." There is one question you cannot skip: compared to what?

The problem with a self-reported "+X%"

Without a baseline, you cannot separate the tool’s contribution from seasonality, a rising or falling market, and other changes you made in the same period. The number may look good, but it is not verifiable.

The fix: a randomized holdout

Randomly hold out a set of campaigns the AI never touches, as a control. Then compare the optimized group against the control — that difference is the causal effect of the optimization, not a market tailwind.

What an honest savings report looks like

  • Net savings (not gross) — the real number after costs
  • A confidence interval (how certain that number is)
  • A methodology version (reproducible and traceable)
  • The measurement window (how long the control ran)
This is also why we can offer an ROI money-back guarantee: we measure what we claim — if we cannot measure the savings, we refund the difference.

To see exactly how we measure it, read our savings-methodology doc; to get a report on your own account, start with a free audit.

Want to try what this article describes?

7-day free trial. Cancel anytime.

Start free trial

More like this