Four matched benchmark cases – AI twins vs. human market research

This validation note benchmarks Delta Labs’ Digital Twins against human market research benchmarks drawn from real client studies. Across four matched-comparison cases, we evaluate agreement on (i) directional decision accuracy (e.g., selecting the same winning option), (ii) mean-value accuracy on standard quantitative endpoints (e.g., purchase likelihood / purchase intent), (iii) distributional accuracy for topline proportions (Top-2 box), and (iv) qualitative topical agreement in open-ended recall.

Human benchmark sample sizes range from n=106 to n=800 where explicitly reported, reflecting standard market research fieldwork appropriate for each method. Digital Twin benchmarks use a fixed n=40 cohort across cases. The Twin sample size was intentionally constrained to test whether benchmark-consistent outcomes can be reproduced under a small cohort; larger Twin cohorts are expected to reduce sampling noise further.

Across quantitative benchmark endpoints (Cases 1, 2 and 4; concept/variant means and Top-2 box), Twins achieve 96.2%–98.9% scale-normalized agreement with human benchmarks. Directional decisions (winner/lowest concept/top-tier) match in all reported cases.

Digital Twins are designed to support market research decisions by generating rapid, target-specific predictions and insights. A core requirement is that Twin outputs are consistent with human research under comparable conditions. The objective of this note is therefore to quantify the degree of agreement between Twin results and human benchmark results under a matched validation protocol, and to document accuracy using transparent, decision-relevant metrics.

Human benchmark provenance (real-world studies)

All benchmarks used in this validation note originate from real market research studies conducted with standard survey/qualitative methods and sample sizes appropriate for the study type (e.g., large-N concept screening; category-buyer product testing; undirected recall verbatims). Digital Twin benchmarks were configured to match the same target definition and were exposed to the same stimuli and question formats to enable a direct comparison of outcomes. Human benchmarks reflect client decision-grade studies (concept screening, product testing, packaging evaluation) with sample sizes aligned to standard practice for each method.

Case selection rationale (coverage across endpoints)

The four validation cases were selected because they span the most common types of deliverables used in consumer market research and cover complementary accuracy modes: (1) multi-stimulus concept screening (mean-level accuracy + decision tiering), (2) A/B product testing (winner direction + mean-level accuracy), (3) open-ended qualitative recall (topic recovery in verbatims), and (4) packaging evaluation using a standard topline KPI (Top-2 box distributional agreement). Together, these cases provide coverage across quantitative means, topline proportions, and qualitative insight extraction under matched measurement conditions.

For each case study, we replicate the benchmark setup along three dimensions:

1. Matched target definition: same market, category-buyer criteria, and quota structure (where applicable).

2. Matched stimuli: identical concepts, product variants, or packaging visuals.

3. Matched measurement: identical question wording and response scales (e.g., 1–11 purchase likelihood; 1–5 purchase intent; Top-2 box; undirected recall).

Human benchmarks represent the external reference. Digital Twin benchmarks were executed in the ELAIIA platform using identical stimuli and identical question wording and response scales to the human benchmark for each reported endpoint, with a fixed Twin cohort per case.

Rationale for Twin cohort size (n=40)

Digital Twin benchmarks were intentionally run with a fixed cohort size of n=40 across cases. The purpose of holding n constant and relatively small is to stress-test whether benchmark-aligned decisions and aggregate metrics can be reproduced under a constrained sample, which is a conservative validation setting. Demonstrating stable agreement under n=40 therefore provides evidence that the approach is not dependent on large simulated samples. In deployment, increasing the Twin cohort size (e.g., 80–200+) is expected to further reduce estimation noise, making convergence to benchmark means easier.

We report complementary accuracy measures to avoid relying on a single summary statistic:

1. Directional decision accuracy

   • Winner agreement (A/B): Twins select the same winning variant as the human benchmark.

   • Bottom-1 accuracy (concept screens): Twins identify the same lowest-performing concept.

   • Top-tier hit rate (concept screens): the Twins’ top-ranked concept falls within the human top tier (allowing ties).

2. Mean-value accuracy (levels) For each stimulus 𝑖, we compute absolute mean deviation |Δ𝑖| = |𝜇𝑖,Twin − 𝜇𝑖,Human|. Deviations are summarized using MAE (mean absolute error) and max deviation. We report normalized agreement as:

   • 1 − MAE/10 for a 1–11 scale (range 10), and

   • 1 − MAE/4 for a 1–5 scale (range 4).

3. Distributional accuracy (topline proportions) For Top-2 box outcomes, agreement is evaluated as absolute deviation |Δ| = |𝑝Twin − 𝑝Human| and normalized distributional agreement 1 − |Δ|, where Δ is expressed as a proportion (e.g., 0.03 for 3pp).

4. Qualitative topical agreement (open-ended recall) Where human open-ended benchmarks are available as verbatim excerpts without coded frequency tables, we evaluate topical agreement as theme presence: a Twin-identified theme is counted as matched if it is directly supported by at least one human verbatim excerpt. Under this evidence format, prevalence-based statistics (theme shares, rank correlations) are not reported.

The validation cases span concept screening, product testing, open-ended recall, and packaging evaluation:

Objective. This study evaluates whether Digital Twins reproduce human outcomes in a multi-stimulus concept screen. We quantify (i) mean-value accuracy (closeness of concept-level mean purchase-likelihood scores) and (ii) directional decision accuracy (identifying the same lowest-performing concept and selecting a top-tier concept for recommendation).

Design & sample (matched setup). The benchmark uses a between-subject design with identical groups. The human benchmark includes n=800 respondents; the Digital Twin benchmark includes n=40 twins. The endpoint is purchase likelihood on a 1–11 scale, measured on the same five concepts under the same question wording and scale.

Methods (agreement metrics).

1. Mean-value accuracy: For each concept 𝑖, we compute |Δ𝑖| = |𝜇𝑖,Twin − 𝜇𝑖,Human|. We summarize deviations using MAE (mean absolute error) and max deviation. We report normalized mean agreement as 1 − MAE/10 (range of a 1–11 scale is 10).

2. Directional decision accuracy: We report two pre-defined decision checks commonly used in concept screening:

   • Bottom-1 accuracy: whether Twins identify the same lowest-performing concept as humans.

   • Top-tier hit rate: whether the Twins’ top-ranked concept falls within the human top tier, allowing ties in the human benchmark.

Results. Twins closely match human concept means across all five stimuli (MAE = 0.11; max deviation = 0.20), corresponding to 98.9% normalized mean-value agreement on the 1–11 scale. Directionally, Twins correctly identify the lowest-performing concept (Bottom-1 accuracy = 1/1: Concept B is lowest in both humans and Twins). For the top recommendation, the human benchmark exhibits a top-tier tie (Concept A and Concept C at 7.10), and Twins’ top-ranked concept is Concept C (7.30), yielding a Top-tier hit rate = 1/1.

Table: Concept-level purchase likelihood agreement (human vs. Digital Twins)

Objective. This case evaluates whether Digital Twins replicate human decision outcomes in a two-variant product test, focusing on directional accuracy (selecting the same winning option) and mean-value accuracy (closeness of purchase-intent means).

Design & sample (matched setup). The human benchmark is a Swiss category-buyer study (n=106) with quotas by gender (51% men / 49% women) and age (18–34: 33%, 35–49: 33%, 50–65: 34%). The Digital Twin benchmark uses a matched cohort (n=40) configured to the same target definition and exposed to the same two stimuli and the same purchase-intent scale (1–5).

Method (quantitative agreement). We compare (i) directional accuracy: whether Twins select the same winning variant as humans, and (ii) mean-value deviation: absolute differences between Twin and human purchase-intent means per variant. Deviation is summarized as MAE (mean absolute error) and max deviation, and normalized by the scale range (1–5 → range 4) as: 1 − MAE/4.

Results. Humans report higher purchase intent for the winning variant (3.6 vs 3.3), and Twins reproduce the same direction and rank order (3.7 vs 3.5). The absolute deviations are 0.10–0.20 points across variants (MAE = 0.15; max deviation = 0.20), corresponding to 96.2% normalized mean-value agreement on a 1–5 scale. Twins and humans select the same winning variant (Winner agreement = 1/1: Variant B > Variant A).

Table: Variant-level purchase intent means and deviations (A/B test; human vs. Digital Twins)

Objective. This case tests whether Digital Twins recover the same spontaneous themes as humans in an undirected recall setting (open-ended question; no pre-defined answer options) when reacting to a protein snack concept.

Design & comparison. The human benchmark consists of open-ended verbatims collected in a standard consumer study (n=150, open question). The Twin benchmark uses a matched cohort (n=40) configured to the same target definition and prompted with the same open-ended question. Because we only have human verbatim excerpts (no respondent-level coded frequencies), this case evaluates topical agreement (theme presence) rather than prevalence statistics.

Method (theme-overlap validation). We performed deterministic theme matching between (i) the five themes extracted by the Twin analysis and (ii) the human verbatim excerpts. A theme is counted as “matched” if at least one human verbatim excerpt provides direct textual evidence for that theme (e.g., explicit mention of price/promotion, protein, ingredients/healthiness, brand/packaging, or healthy snack/fitness framing). No frequency-based metrics (e.g., theme shares, rank correlations) are reported in this case because the necessary human-coded counts are not available.

Result. All five Twin-identified themes are corroborated in the human verbatim excerpts: (1) price/promotion as a purchase barrier and the need for an entry offer, (2) high protein as the primary value driver linked to an active/healthy lifestyle, (3) ingredient transparency and skepticism about whether the product is truly healthy (e.g., sugar/additives) with demand for clearer labeling, (4) brand trust and modern packaging as a positive signal, and (5) positioning as a functional “high-protein snack” (incl. added nutritional benefits) for everyday or sport-related occasions. This demonstrates full topical overlap (5/5) for the dominant themes shown in the benchmark material under an undirected recall setup.

Objective. This study evaluates whether Digital Twins replicate human distributional outcomes on a standard packaging evaluation metric, using Top-2 box (“good/very good”) agreement as the primary endpoint.

Design & sample (matched setup). The benchmark compares a human packaging test against a matched Digital Twin cohort exposed to the same packaging stimulus and asked the same design-liking question on the same response scale. The human benchmark includes n=200 respondents; the Digital Twin benchmark uses n=40 twins.

Methods (agreement metrics). We compute Top-2 box in humans and Twins as the share of respondents rating design liking as “good/very good.” Agreement is reported as (i) absolute percentage-point deviation |Δ| and (ii) normalized distributional agreement 1 − |Δ|, where Δ is expressed as a proportion (e.g., 0.03 for 3pp).

Results. Human Top-2 design liking is 76%, while Twins report 79%. The absolute deviation is 3.0 percentage points (0.03), corresponding to 97.0% distributional agreement on this endpoint.

Table: Top-2 design liking agreement (human vs. Digital Twins)

Across quantitative endpoints where human benchmark values are explicitly reported (Case 1, Case 2, Case 4), Digital Twins show high agreement with human outcomes on both levels and directionality. Mean-value accuracy achieves 98.9% normalized agreement in concept screening (1–11 scale; MAE 0.11) and 96.2% normalized agreement in the A/B product test (1–5 scale; MAE 0.15). For packaging evaluation, Top-2 design liking differs by 3 percentage points (76% human vs 79% Twins), corresponding to 97.0% normalized distributional agreement. In open-ended recall, Twins show full topical overlap with the dominant benchmark themes shown in human verbatims (5/5 themes matched).

These four benchmark comparisons provide evidence that Digital Twins executed in the ELAIIA platform can reproduce key human market research outcomes under matched study conditions. The human benchmarks are drawn from real client decision-grade studies (including large-N concept screening, category-buyer product testing, and packaging evaluation) with human sample sizes up to n=800, reflecting standard industry practice for these methods. Agreement is observed at the decision level (e.g., selecting the same winning option and identifying the lowest-performing concept), at the quantitative level (close alignment of mean purchase likelihood and purchase intent), at the topline distribution level (Top-2 box), and at the qualitative insight level (recovery of dominant open-ended themes). The validation was conducted under a deliberately small Twin cohort (n=40), indicating that benchmark-consistent conclusions can be reproduced under constrained sample sizes; scaling Twin cohort size is expected to further reduce estimation noise.