Why can’t we just build this ourselves?

URL: /academy/articles/why-cant-we-just-build-this-ourselves Type tag: ARTICLE · Second tag: ACADEMY · PART 2 OF 2 Read time: 7 min Author byline: Delta Labs research team First published: planned as the October 2026 newsletter edition (not yet on LinkedIn as of 18 Oct 2026 – publish on the website and LinkedIn together) Source: Elaiia_Academy_Part2.docx Status: READY – Tier A (publish as is) Card teaser: The objection is framed as a capability question. It is not one. Three requirements decide whether a synthetic population can be trusted – and none of them is a modelling problem.

“Why can’t we just build this ourselves?”

Last month we published the results of a like-for-like benchmark: Elaiia’s calibrated AI twins, a standard large language model, and the same model asked to role-play 100 individual respondents, all scored against the European Social Survey, Round 11 (Switzerland) — an academic probability sample of 1,384 people that neither we nor any vendor controls.

The twins deviated from the real population by 1.5%. The persona sample by 14.0%. But the ratio was the least interesting part. Four structural findings mattered more:

• The model was accurate only where the answer has already been published — 6.5% on documented items, 25.2% on personal values, where nobody has published a figure to recall.

• More personas made it worse, not better — so the error is bias, not sampling noise, and no sample size fixes it.

• The value hierarchy collapsed — compressed to 79% of the real spread, and reordered.

• Simulated individuals did not co-vary — 100 unrelated answer sheets rather than 100 coherent people, which quietly breaks segmentation, key-driver and conjoint work.

We then granted every objection we could raise against our own result and re-ran it five times. The twins stayed between 1.2% and 1.6%; the model between 13.4% and 15.5%. The gap never fell below eightfold.

One sentence carries all of it: accuracy is not a property of the model. It is a property of the ground-truth data the model is calibrated against, and of how well that data covers the question being asked.

Which brings us to the question we get asked most often inside large organisations.

Teams are capable. They have the same models we do. Generating synthetic respondents is not technically hard — and that last point is the misunderstanding, not the answer. The hard part was never producing respondents. It is knowing whether they are right.

Three requirements decide whether a synthetic population can be trusted. None of them is a modelling problem.

A synthetic population is only ever as good as the real data it was calibrated against, and there is no shortcut: nothing inside a language model contains the answer to a question about a population nobody has measured. Part 1 was a demonstration of exactly that — competent where a published figure exists, 25% off where none does.

So the scarce asset is not the generator. It is the ground truth: probability-sampled, per market, covering the dimensions the work actually touches, and licensed or commissioned in a form that permits calibration against it.

This is the form the objection almost always takes, and it deserves a precise answer rather than a dismissive one. The data is usually excellent. It is also, for this purpose, the wrong kind — and the mismatch is not one that volume repairs.

What most companies hold about their customers is CRM and transactional data: contract and tariff, tenure, acquisition channel, purchase and basket history, recency and frequency, digital events, service tickets, a campaign-response flag, an NPS score. Four properties of that data matter here.

• It records outcomes, not drivers. It captures what people did inside your own funnel. It contains no measurement of why — values, motivations, trade-offs, identity, the things a person weighs before acting. A purchase log does not contain a value hierarchy, and the value hierarchy is precisely where uncalibrated systems failed hardest in part 1.

• It is endogenous. It describes people who already bought, from the assortment you already offer, through the channels you already run, at the prices you already set. It says nothing about non-customers, lapsed customers, competitor customers, or any counterfactual — which is most of what research gets commissioned to answer.

• It has no sampling frame. No probability design and no weights, so its relationship to the population is unknown and its bias cannot be characterised. That is the difference between a customer base and a benchmark: you cannot calibrate against a dataset when you cannot say what it is skewed toward.

• Its variables are weak predictors of the ones that matter. Age, gender, region, tenure and spend are poor predictors, on their own, of what a person values and how they will choose.

That last point is not something we are asking anyone to take on trust, because the study tested it directly. The persona arm gave the model a demographic profile for each of its 100 respondents — the same class of variable a CRM holds — and asked it to answer as those people. The result was a 14.0% mean deviation with the correlation structure collapsed to r = 0.34. Demographic identification is not calibration.

Which is why scale does not rescue the position. The constraint is not how much data there is; it is that the variables which predict how a person will respond are not in it, and no quantity of the variables that are in it substitutes for them. A hundred million transactions contain zero measurements of what a person values. It is the same shape as the persona finding: there, more personas did not fix a bias; here, more rows do not fill a missing variable.

What would make customer data genuinely valuable is linking it to measured attitudinal data on the same individuals, so that the behavioural record and the psychological structure can be observed together. That is a research programme, not a data export. And in most European organisations there is a prior constraint anyway: data collected to deliver a service generally cannot be repurposed to build a simulation without a fresh legal basis.

A team that wants to validate its personas against reality needs ground-truth data for that population and those questions. If it holds that data, it already holds the answer it was simulating — and the simulation is redundant. If it does not hold it, the personas cannot be scored at all.

An unscored synthetic panel does not fail loudly. It returns confident, well-formatted, plausible numbers, with nothing in the output to signal which of them are 1.5% off and which are 40% off. The error surfaces much later, as a decision taken on a value hierarchy that does not exist in the population.

The lesson: volume is not the constraint. Ask whether the dataset contains any measurement of why people choose — and how anyone would know if the output were wrong.

The most common misconception is that a synthetic population is a static asset: build a Swiss panel, then ask it anything. It does not work that way, and part 1 shows why.

The model’s failure pattern was not really about that model. It is what an uncalibrated dimension looks like. Where the system had been exposed to the relevant measured reality, it was close. Where it had not, it produced a confident fill. That boundary exists for every synthetic population, including a well-built one: a panel calibrated on institutional trust and social values is a valid instrument for those dimensions, and is not automatically valid for category purchase behaviour, price sensitivity, media consumption or brand association.

Each study therefore requires the population to be re-fitted to the dimensions that study is actually asking about, against data that covers them. That is not an inefficiency to be engineered away — it is what distinguishes an instrument from a plausible-text generator.

For anyone building internally, this is the part that changes the shape of the project. The ground-truth requirement is not one dataset acquired once. It is a data supply — per market, per question domain — together with the acquisition relationships needed to keep it arriving.

The lesson: before you ask a population a question from a new domain, ask what dimensions it was calibrated on. Validity does not transfer for free.

Values, attitudes, category behaviour and the language people use to describe all three move over time. The European Social Survey runs in rounds for precisely this reason: a population measured once is a photograph, not a model.

Two clocks run against any calibrated population at once. The population moves — a panel calibrated in one year is accurate for that year and drifts from then on, silently, because a stale panel produces output indistinguishable in tone and format from a current one. And the model gets replaced underneath — updated without notice, so the same prompts return different answers. Neither clock announces itself. Only re-benchmarking against fresh ground truth detects either one.

Calibration is therefore not a build step that completes. It is an operating process with a cadence, an owner and a cost line — and a system without it degrades from the day it ships, while continuing to look exactly as authoritative as it did on day one.

The lesson: ask when the population was last re-benchmarked, and against what. “Built in 2024” is a date, not a status.

• Ground-truth data as a standing supply — not a one-off purchase; per market and per question domain, with the relationships and licences to keep acquiring it.

• Question-level calibration — the population re-fitted to the dimensions each study touches, rather than a general-purpose panel queried about everything.

• Individual-level coherence by construction — so respondents co-vary and multivariate analysis is possible at all. It cannot be requested in a prompt.

• A re-benchmarking cycle against both clocks — population drift and model replacement, on a defined cadence rather than on suspicion.

• A named owner and an audit trail — someone accountable for the error rate, and a record a research buyer, a client or a regulator can inspect.

None of this is beyond a large organisation. All of it is a permanent function rather than a project — a small research operation whose full-time job is keeping a synthetic population true. Teams that decide to build usually discover the distinction after the first study whose answer nobody can check.

Not a model. Models are a commodity and will keep improving for everyone, including for us. Not a prompt — that is an afternoon’s work.

What is worth buying is a standing instrument: the ground-truth data foundation, the calibration of the population to each question, the coherence that makes respondents behave as people rather than as answer sheets, and the re-benchmarking cycle that keeps all of it true as both the population and the models move underneath.

So to the four questions from part 1, add a fifth — and it is the one that separates an instrument from a demo:

5. What ground-truth data is this population calibrated against for my question, and when was it last re-benchmarked?

We publish our number: 1.5%, against a public third-party benchmark anyone holding the ESS data can check line by line. It is not a claim about technology — it is the current reading on an instrument that is calibrated continuously. And it is a number, which is the part that can be checked.

This market currently competes on the breadth of what is claimed. We would rather compete on a figure someone can go and verify — including when they are verifying it against us.

The full validation report is available on request. Elaiia is built by Delta Labs AG in Zürich.