Problem Framing & Decision Design6 min read·By , Independent AI Advisor & Fractional AI Lead·

Your Synthetic Customer Cannot Tell You What You Forgot to Ask

Synthetic customers can test assumptions, but they cannot reveal what your research never asked.

A synthetic customer can give a product team a remarkably coherent account of why someone might choose one offer over another. Coherence is useful when you are examining a hypothesis. It becomes dangerous when the team mistakes the coherence of the answer for evidence about people it has never observed.

The important question is not whether an AI-generated respondent sounds plausible. It is what decision you are prepared to make on the strength of its answer, and what kind of error would change that decision.

Consider a hypothetical pricing exercise. A team describes a product, defines several customer segments, asks a model to respond as each segment and receives a tidy ranking of three prices. Nothing in that sequence establishes what customers would actually pay. The model has produced responses conditional on a description and its own learned patterns. The team has not observed a purchase, a refusal, a trade-off or the moment when a customer interprets the offer differently from the product team. The scenario is illustrative, not a report of an observed company or measured consumer behavior.

This distinction matters because the label synthetic customer hides several very different methods. A loosely prompted persona is a model asked to imagine a type of person. A synthetic segment panel generates multiple responses conditioned on demographic or behavioral descriptors, sometimes with quotas. A data-grounded individual agent is built from information about a particular real participant, potentially including previous responses and interviews. Those methods have different information inputs, and their outputs should not be treated as interchangeable. None is automatically a substitute for observing a new behavior.

What is being validated?

A method can preserve the overall share of people choosing an answer while misrepresenting who chooses it. It can preserve differences between broad groups while failing to predict an individual's answer. It can reproduce an old survey question while failing on a question about a new event. A favorable result on one of those tests does not transfer automatically to the others.

This is why a single accuracy score is an inadequate purchasing criterion for synthetic research. Before selecting a tool, specify the target: marginal distribution, subgroup ordering, individual response, preference ranking, or actual behavior. Then ask for a human collected under comparable conditions. The relevant unit of validation follows from the decision.

Pew Research Center's September 2026 methodological report is instructive precisely because it compared generated survey answers with answers collected from real participants. The researchers used profiles of members of the American Trends Panel and asked an AI model to questions from three survey waves in the first half of 2026. Across nearly 300 questions, synthetic estimates differed from the human survey estimates by about 12 percentage points on average. The report also documented subgroup differences and model sensitivity. This was a study of public-opinion survey replication, not product pricing or purchase conversion; it cannot establish a numerical error rate for commercial customer research. Its more general lesson is methodological: plausible responses and detailed profiles do not eliminate the need for a human comparison.

The observation dates matter. Pew published the report on 30 September 2026, but the human survey waves were fielded in January, March and April 2026; the last synthetic replication was conducted from 27 April to 1 May. As of 7 November, those observations fall outside a strict rolling six-month window. They are included here as a dated of a method, not as evidence of how consumers or current AI products behave today.

There is a serious counterargument to a blanket rejection of synthetic respondents. A synthetic sample may be useful even if it fails to reproduce each person. An October 2026 preprint by D'Ecclesiis, Castellacci and Ahmadi explicitly distinguishes predictive, distributional and substantive validity. Its abstract reports that some model-generated samples preserve demographic ordering and other aggregate relationships despite imperfect individual correspondence. That is a reason to test synthetic methods against the intended analytic purpose, not a reason to promote them to universal substitutes for field research. The preprint's empirical observation period still requires verification before it can support any current-behavior claim.

The unanswered question is usually upstream

Teams often ask synthetic respondents the questions they already know how to formulate. They choose the segments, define the product, describe the alternatives and decide which outcomes count. A model can vary answers within that frame, but it does not independently establish that the frame includes the customer's actual problem. If the product description leaves out a constraint that matters in the field, a hundred additional simulated responses can make the omission look more robust rather than expose it.

That is the difference between sensitivity testing and discovery. Sensitivity testing asks how a conclusion changes when known assumptions change. Discovery asks which assumptions should have been in the model at all. The first can benefit from fast, inexpensive simulation. The second requires access to evidence that was not already embedded in the or training distribution: interviews, direct observation, support conversations, transaction traces or field experiments, with the appropriate consent and privacy safeguards.

The value of direct research is not that every participant is truthful or every interview is representative. Interviews can suffer from recall error, leading questions and social desirability. Behavioral data can omit motives and exclude people who never entered the funnel. Human research needs sampling, triangulation and methodological care. The point is narrower: a synthetic answer cannot independently reveal what it has not been given an opportunity to observe. An appropriately designed study can bring a new constraint into view, even if the eventual estimate needs a different method.

Decide what the simulation is allowed to change

A practical research plan starts with the decision, not the respondent generator. If a team is exploring alternative wordings for an interview guide, synthetic role-play may help surface ambiguous phrases or likely follow-up questions. The result is a candidate question, not evidence that a customer holds the expressed view. If the team is choosing which hypotheses to investigate first, a simulated range of reactions can help challenge its assumptions, provided the assumptions and model dependence are explicit.

If the decision is a price, a market-entry commitment or a change affecting a vulnerable subgroup, the burden is different. The team needs observed evidence relevant to that decision and a predeclared tolerance for error. It should compare the synthetic method with a suitable baseline, examine subgroup failures and preserve uncertainty instead of reporting an unqualified average. It should also ask what would cause the method to fail: an unfamiliar product, a new policy, a rare user need, a context-specific trade-off or a shift in the market.

There is no universal rule that says synthetic methods must always come before human research, or that they must never be used after it. Their role depends on the question. A model grounded in recent, relevant human data may be useful for testing the robustness of a known relationship. That same model should not be trusted to invent evidence about an emerging behavior merely because it can narrate one.

The useful boundary is therefore not AI versus humans. It is generated possibilities versus observed claims. Keep the former available for exploration. Require the latter before a decision depends on what real people supposedly think, want or will do.

Sources and methodological boundaries

The pricing scenario is hypothetical. The decision framework is author synthesis.

Get new essays as they publish.

One email per essay. No noise between.

Berk Bayri

Independent AI Advisor & Fractional AI Lead

I help leadership teams decide where AI belongs, test it before it scales and build the teams that run it, drawing on 25+ years of building digital products, experiences and capabilities.