Methodology.

userken accuracy methodology

userken publishes how well its synthetic audiences predict what real people say. This page defines every number on the accuracy page, exactly as the code computes it (metrics.py, holdout.py, eval_engine.py), so a reader can check the definitions and decide how much weight to give them.

Nothing on the accuracy page is typed in by hand. If no benchmark has been run yet, the page says "first benchmark pending" rather than showing a number.

What is being measured

An evaluation asks the synthetic panel a structured question (single choice or a 1 to 5 scale), collects the panel's answer distribution, and compares it with the true answer distribution from real people. "Real people" are one of:

1. Held-out app reviews. Reviews the panel is not allowed to see, converted

into questions with known answers (star ratings, theme prevalence, app comparisons, platform splits, feature sentiment). This is the bulk of the benchmark and the only part that is refreshed automatically.

2. Published surveys. Toplines from public research reports (Reuters

Institute Digital News Report, Pew Research Center) transcribed with their source URL, year, country, sample size and wording into evals/benchmarks/*.yaml. Only numbers that could be verified on the source page are included.

Every question is scored on its own, and the headline is the plain mean over questions. A question with 40,000 held-out reviews behind it counts the same as one with 40 (subject to the minimum below), so a couple of very large apps cannot dominate the score.

Metrics

All distributions are shares over the question's K answer options and sum to 1. p is the panel's prediction, q is the truth.

1-MAE (industry convention)

1-MAE = 1 - (1/K) * sum_i |p_i - q_i|

This is one minus the mean absolute error in percentage points, divided by

1. It is the number most synthetic-audience vendors publish, so we publish

it too, but it should be read with one thing in mind: it gets more flattering as K grows. The worst possible score is 1 - 2/K. A 5-option question can never score below 0.60; a 10-option question never below 0.80. Putting all of the mass on the wrong option scores 0.0 on a yes/no question and 0.90 on a 20-option question. Comparing 1-MAE across questions with different option counts, or across vendors who use different option counts, is not meaningful.

NDAM (userken definition)

Electric Twin's NDAM is proprietary and not published. userken's "normalised distribution accuracy measure" is defined openly as one minus the total variation distance:

NDAM = 1 - 0.5 * sum_i |p_i - q_i| = sum_i min(p_i, q_i)

It is the share of probability mass the panel placed on the right options. 1.0 means identical distributions, 0.0 means the panel and the truth never overlap. It does not depend on K: adding options nobody picks leaves it unchanged, and a completely wrong answer scores 0.0 whether there are 2 or 20 options. NDAM is the number we consider the honest headline.

Because 1-MAE and NDAM are both linear in the absolute errors, they are related by 1-MAE = 1 - 2 * (1 - NDAM) / K. On a 5-option question, NDAM 0.80 corresponds to 1-MAE 0.92. That is the whole difference between the two numbers.

Secondary metrics (reported per question and per family)

• Jensen-Shannon similarity: 1 - JSD(p, q) in bits, in [0, 1]. Penalises

confident misses more than NDAM does.

• EMD similarity (ordinal questions only): 1 - EMD / (K - 1) where EMD is

the earth mover's distance between the two distributions on the ordered scale. Predicting "4 stars" when the truth is "5 stars" is penalised less than predicting "1 star". Not reported for nominal questions.

• Top-box hit rate: 1 if the panel's most common answer is also the real

most common answer, else 0. Averaged, it is the share of questions where the panel gets the modal answer right.

• Rank correlation: Spearman rho between the predicted and true option

shares (does the panel order the options the way people do).

Human ceiling: split-half agreement

A perfect model cannot agree with a sample of real people any better than two samples of real people agree with each other. For every question we shuffle the real respondents, split them into two random halves, score half A against half B with the same metrics, and repeat 200 times. The mean is reported as the human ceiling next to the panel's score. It is an analogue of the test-retest ceiling other vendors cite (Electric Twin reports 94% NDAM), with two honest differences: it is computed per question from the same held-out data, so it reflects that question's sample size, and each half has n/2 respondents, so it is slightly conservative (lower) compared with a full-size retest. For app-comparison questions the two apps' reviewers are split separately and P(A > B) is recomputed per half.

Baselines

Every question is also scored for three baselines on the same questions:

• uniform: a flat distribution. What you get by knowing nothing.
• corpus prior: the same question answered from the training window

(reviews before the holdout date). "Just repeat history." A synthetic panel is only worth having if it beats this.

• ungrounded LLM: the same model asked the same question with the same

personas but no review retrieval. This isolates what grounding in real reviews adds over the model's priors.

Holdout design and the leakage caveat

The corpus is split by time. Reviews on or after test_from (default: the last 90 days) are the test window; the panel's grounding retrieval is restricted to reviews before train_until (the same date). Questions are answered from test-window reviews only, and each question's evidence count (n) is the number of test-window reviews behind it. Questions with fewer than 30 are dropped.

Caveat. The personas currently in production were clustered and named from all reviews, including the ones now in the test window. The time split stops the panel from reading test reviews during a run, but the persona descriptions may already encode them. The honest number therefore requires rebuilding personas on the train window before the run; until the persona engine accepts a date filter, published runs should say so (the report's notes do). A random id split (a deterministic 20% of review ids by hash) is also supported for quick checks, but it is more optimistic still: train and test reviews interleave in time, so the corpus prior already "knows" the test period. It is never used for the published number.

Question families derived from held-out reviews

Themes are the same regex-tagged themes used across userken; theme labels are mapped to plain English in the questions.

Published surveys: population mismatch

The external benchmarks ask a panel built from app reviewers questions that were put to the general online population of a country. Reviewers are not a representative sample: people who write app reviews skew towards the very happy and the very angry, towards heavier app users, and (for Google Play) towards Android. A gap between the panel and a Reuters Institute or Pew topline is therefore partly a population gap, not only a model error. These questions are reported as their own family and are not mixed into the held-out-review headline. They are there to show the direction and size of the mismatch.

How to read the accuracy page

• The headline NDAM and 1-MAE are equal-weight means over held-out-review

questions in the latest completed run.

• Read the panel's NDAM against the human ceiling (the attainable maximum

for that question set) and the corpus prior (the score you get without a model). The panel's contribution is the distance above the prior; the room left is the distance below the ceiling.

• Per-family and per-audience tables show where the panel is strong and where

it is not. Small n (few questions) means the family number is noisy.

• The holdout box says which split was used and how many reviews were held

out. If it says the personas were not rebuilt on the train window, treat the number as an upper bound.

Reproducing a run

python -m eval_engine --families rating,theme_rating,theme_prevalence,pairwise,platform_split,feature_sentiment \ --holdout time --test-days 90 --samples-per-persona 3 --max-questions 60

--dry-run prints the generated questions and truths without calling the model; --stub runs the whole pipeline with a deterministic fake panel (never published); --report <run_id> prints a stored report; --rebuild-personas makes the run leakage-safe: for every publication or category in the question set it creates (or reuses) an Audience Builder audience named eval-train:<key>:<train_until> whose filters are {apps or categories, date_to: train_until}, builds its personas from that frozen member set only, and answers with those personas. The mapping is stored on the run and stated in the report notes. Runs without the flag use the production personas, which were built on all reviews, and say so.

Respondent sampling runs on the "fast" LLM tier by default (--tier), which is what production surveys use; the report always states the model used for the grounded run and for the ungrounded baseline, plus the run's LLM cost, because an accuracy number only means something next to the model that produced it.

Questions whose audience has no personas yet (at the time of writing, the browser apps) are skipped before the question cap and listed on the run config as skipped_audiences, so a capped run is made of answerable questions.