How to Test Incrementality and Run Geo Holdouts Without a Data-Science Team

Key Takeaways

• You don't need a data-science team to test incrementality — practical geo and audience holdouts answer the budget questions that matter.
• Incrementality proves causation (did marketing cause sales that wouldn't have happened anyway); attribution only assigns credit and can't prove cause.
• A geo holdout runs a channel in some regions and off in comparable ones, then compares the sales difference; an audience holdout withholds ads from a random slice and compares.
• Validity without data science comes from comparable groups, enough scale to read a difference, running long enough for your sales cycle, and changing only one thing.
• Avoid the pitfalls — non-comparable groups, too little scale, contamination between groups, and stopping too early.
• Read results as directional, not precise, and act on the big clear findings — even simple, imperfect tests teach more than any attribution model.

Incrementality Is Within Reach — It's Not Just for Data Teams

Incrementality is the truest measure of whether your marketing works, and most marketers assume it's out of reach without a data-science team. Both halves of that sentence deserve attention. Incrementality asks the question that actually matters: did your marketing cause sales that would not have happened otherwise? This is different from — and far more valuable than — attribution, which merely assigns credit for sales among channels. Attribution can tell you a channel 'got credit' for conversions; only incrementality can tell you whether those conversions would have happened anyway without the spend. And that distinction is where enormous amounts of marketing budget are wasted or misjudged: a channel can show a great attributed ROAS while being largely non-incremental (taking credit for demand that would have converted regardless), and you'd never know from attribution alone. Incrementality is how you find out what your spend is actually causing.

The assumption that testing incrementality requires data scientists and sophisticated tooling is what stops most companies from ever doing it — and it's largely false. While the most rigorous incrementality measurement can involve advanced statistics, the practical tests that answer the questions you actually have for budget decisions — is this channel causing incremental sales, or would they happen anyway — are well within reach of a company without data scientists. The most accessible of these, geo holdouts and audience holdouts, are conceptually simple: create a control group that doesn't get the marketing, compare it to a group that does, and see whether the marketing made a difference. You don't need a data-science team to run a test like that well enough to learn something true and act on it; you need a clear understanding of the method, sensible design choices, and the discipline to avoid a few pitfalls.

So this guide is about running practical incrementality tests without a data-science team. It explains why incrementality beats attribution and why you don't need perfection to benefit; the two accessible test types (geo and audience holdouts) in plain terms with how to run each; the design choices that make a test valid enough to trust; the pitfalls that invalidate a test and how to avoid them; and how to read and act on results without overclaiming precision. It also flags where the simple approaches suffice and where you genuinely need more sophistication. The goal is to get you running incrementality tests on your biggest spend lines, because even simple, imperfect incrementality tests will teach you more about what's actually working than any attribution model — and the belief that you can't do them without data scientists is the main thing standing in the way.

Why Incrementality Beats Attribution — and Why Imperfect Is Fine

To motivate the effort, be clear on why incrementality is worth testing when you already have attribution. Attribution assigns credit for conversions among the channels that touched them — it answers 'which channel gets credit for this sale.' But it cannot answer the question that actually determines whether spend is worth it: would this sale have happened anyway without the spend? A channel can be assigned lots of credit (high attributed ROAS) while causing very little incremental sales, because it's getting credit for conversions that would have happened regardless — the classic case being retargeting and branded search, which often 'convert' people who were already going to buy. Attribution, by design, cannot distinguish caused sales from credited-but-inevitable sales, so it systematically over-values channels that harvest existing demand. Incrementality can distinguish them, because it compares what happens with the spend to what happens without it — which is the only way to isolate causation.

This is why incrementality testing regularly overturns attribution-based beliefs and saves or redirects significant budget. A company that runs an incrementality test on a channel it believed was highly effective (based on attributed ROAS) sometimes discovers the channel is largely non-incremental — turning it off barely reduces sales, because the sales it was 'driving' would have happened anyway — which means the budget was being wasted on credited-but-inevitable conversions. Conversely, a channel with poor attributed metrics sometimes proves highly incremental — it's creating demand that shows up as sales attributed elsewhere. These discoveries are only possible through incrementality, and they're exactly the kind of finding that justifies major budget reallocation, which is why incrementality is worth the effort despite being harder than reading an attribution dashboard.

Crucially, you don't need perfect incrementality measurement to benefit enormously — and this is what makes it accessible. The budget-relevant questions are usually big and directional: is this major channel causing meaningful incremental sales, or not? Would cutting this spend meaningfully hurt sales, or not? These are questions a simple, imperfect test can answer well enough to act on, because the findings that matter are usually large (a channel is largely non-incremental, or clearly is incremental) rather than requiring precise measurement of exactly how incremental. A test that tells you 'turning this channel off clearly reduced sales substantially' or 'turning it off barely moved sales' answers your budget question, even if it can't tell you the incremental ROAS to two decimal places. So don't let the pursuit of precise incrementality measurement (which does need sophistication) stop you from running the simple, directional tests (which don't) that answer your actual questions. Imperfect but directionally clear beats precise but never done. Ask yourself: do I actually need the exact incremental ROAS, or do I need to know whether this big spend line is causing sales — because the latter is within reach right now.

The Two Accessible Tests: Geo and Audience Holdouts

The two most accessible incrementality tests both work by the same simple logic — create a group that doesn't get the marketing (a control) and compare it to a group that does (a test) — and the difference between them is the incremental effect. The first is the geo holdout: you run a channel or campaign in some geographic regions (test regions) and turn it off in other, comparable regions (control regions), then compare the sales or conversions between them. If the test regions (with the marketing) generate meaningfully more sales than the comparable control regions (without it), that difference is the incremental effect of the marketing — it caused sales that didn't happen in the comparable regions where you didn't run it. Geo holdouts are especially accessible because geography is an easy way to split into groups, many businesses have enough regional spread to form comparable groups, and the platforms and your own sales data can be read by region. For channels where you can control spend geographically, the geo holdout is often the most practical incrementality test available.

The second is the audience holdout: you withhold your ads from a randomly selected slice of your target audience (the control) while showing them to the rest (the test), then compare the conversion rates or sales between the held-out group and the exposed group. If the exposed group converts meaningfully more than the held-out group, that difference is the incremental effect. Some ad platforms offer built-in holdout or 'conversion lift' style tests that implement this for you by randomly withholding ads from a control group and measuring the difference, which makes audience holdouts accessible without you having to engineer the split yourself — worth checking what your platforms offer. Audience holdouts directly isolate the effect of the ads on comparable people, which is conceptually the cleanest form of the test. The main practical requirement, as with geo, is enough scale for the difference to be readable above the noise.

Both tests share the same essential structure and the same accessibility: they don't require statistical modeling to set up or to read directionally — they require the discipline to create a genuine control group, run the comparison at enough scale, and read the difference. The choice between them is practical: use a geo holdout when you can control the channel's spend by region and have enough comparable regions (common for broad channels and businesses with regional spread); use an audience holdout, especially a platform's built-in lift test, when you can randomly withhold from a slice of your audience (common within a single ad platform). Either way, the logic is the same: with-marketing versus without-marketing, and the difference is what the marketing caused. This is the whole conceptual basis of incrementality testing, and it's simple enough to run without data scientists — the sophistication, where it's needed, is in precise measurement and edge cases, not in this core logic. The table below compares the two accessible tests.

Designing a Valid Test and Avoiding the Pitfalls

The difference between an incrementality test you can trust and one that misleads you comes down to a few design choices, none of which require data science — just care. First, comparable groups: your test and control groups (regions or audience slices) must be genuinely comparable, so that the only meaningful difference between them is the marketing. For geo tests, that means choosing test and control regions that are similar in size, customer behavior, seasonality, and baseline sales, so a difference in outcomes reflects the marketing rather than pre-existing differences between the regions. For audience tests, random assignment handles comparability, which is why platform lift tests that randomize are clean. If your groups aren't comparable to begin with, any difference you measure is confounded and the test is invalid — comparability is the foundation.

Second, enough scale and a long enough run. The incremental effect has to be large enough, relative to the natural noise in your sales, to be readable — which requires enough spend and enough volume in the test that the difference stands out above random fluctuation. A test that's too small produces a difference indistinguishable from noise, telling you nothing. And the test must run long enough to capture your sales cycle: if your customers take weeks to convert, a test run for days will miss the sales the marketing caused later, understating incrementality. Run the test at meaningful scale (ideally on a major spend line where the effect is big and the stakes justify it) and for long enough that the caused sales have time to show up. Third, change only the one thing you're testing: hold everything else constant during the test so that the difference between groups is attributable to the marketing you're testing and not to some other change you made simultaneously.

The pitfalls to avoid are the inverses of these design choices, plus a couple more. Non-comparable groups invalidate the test (the difference reflects the groups, not the marketing). Too little scale makes the result unreadable (the effect drowns in noise). Contamination between groups corrupts the test — for example, if your control-region customers are exposed to the marketing anyway (national campaigns bleeding into control regions, or people traveling between regions), the control isn't clean, so ensure your control genuinely doesn't get the marketing. Stopping too early misses delayed conversions and understates the effect. And over-interpreting a noisy or small result — reading precision into a test that only supports a directional conclusion — leads to false confidence. Avoid these, and a simple holdout test gives you a trustworthy directional read on incrementality. The good news is that none of avoiding them requires data science; it requires designing the test with care and being honest about its limits. Ask yourself of any test you run: are my groups genuinely comparable, is the scale big enough to read, is the control genuinely uncontaminated, and did I run it long enough — because those four questions determine whether the test is trustworthy.

Reading the Results, Acting on Them, and Knowing the Limits

Read incrementality test results as directional rather than precise, because that's what a practical test supports and what your budget decisions need. The question to ask of the result is the big, clear one: did the marketing cause a meaningful difference or not? If holding a channel out clearly and substantially reduced sales in the control relative to the test, the channel is meaningfully incremental — it's causing sales, and cutting it would hurt. If holding it out barely moved sales, the channel is largely non-incremental — the sales it was credited with were mostly happening anyway, and the spend is a candidate for reduction or reallocation. These directional conclusions are what you act on, and they're usually clear enough in a well-designed test to be trustworthy even without precise measurement. Resist the temptation to over-read a precise incremental ROAS from a simple test; take the directional finding, which is what matters, and act on the big, clear results while treating small or ambiguous ones as inconclusive rather than forcing a conclusion.

Acting on the results means letting incrementality override attribution where they conflict, because incrementality is the truer signal. When a test shows a channel your attribution loved is largely non-incremental, believe the incrementality test and reallocate the budget — that's exactly the kind of finding incrementality testing exists to produce, and it's where the big budget wins are. When a test confirms a channel is strongly incremental, you can invest in it with confidence that attribution alone couldn't give you. Over time, running incrementality tests on your major spend lines builds a much truer picture of what's actually working than attribution provides, and it's the discipline that separates marketers who know what their spend causes from those who only know what their dashboards credit. You don't need to test everything constantly — periodic incrementality tests on your biggest and most questionable spend lines deliver most of the value, because that's where the budget and the uncertainty are largest.

Finally, know the limits of the simple approaches, because honesty about them keeps you from over-reaching. Simple geo and audience holdouts are excellent for the big directional questions on major channels, but they have limits: they're harder to run for small spend lines (not enough scale), they measure the channel tested rather than untangling complex multi-channel interactions, and precise measurement (exact incremental ROAS, subtle effects, sophisticated multi-cell designs) genuinely does benefit from more statistical sophistication and sometimes a data-science capability. So use the simple tests for what they're good at — answering the big, budget-relevant causation questions on your major spend — and recognize when a question genuinely requires more sophistication (in which case bringing in that capability, whether hiring or partnering, is warranted for the questions that justify it). But don't let the existence of sophisticated methods stop you from running the simple, accessible tests that answer most of your real questions right now. The marketer who runs simple, imperfect incrementality tests on their biggest spend lines is far ahead of the one who runs none while waiting to build a data-science team. If you want help designing and running practical incrementality tests — geo and audience holdouts sized to your business — and acting on what they reveal about your real spend, that is exactly the kind of work our team does with performance-focused companies.