Measuring accuracy
A synthetic audience is only worth anything if somebody checks it against reality. userken scores its own method against real answers it was not allowed to see, publishes the result at /accuracy, and shows nothing at all rather than an estimate when a benchmark has not run.
The two metrics
Both metrics compare two distributions over the same options: what the synthetic audience said, and what the real people said.
NDAM
Normalised distribution agreement. It is the share of the distribution that lands in the right place: add up the smaller of the two values for each option. One means the two distributions are identical, zero means they have nothing in common. It does not change when a question has more or fewer options, which is why it is the headline number.
1-MAE
One minus the mean absolute error across the options. It is the familiar number and is reported alongside NDAM, but it flatters questions with many options: the more options there are, the smaller each error can be, so the floor rises. Compare 1-MAE only between questions with the same number of options.
For a question with K options, 1-MAE = 1 - 2(1 - NDAM)/K. That is the whole reason both are shown: one is readable, the other is comparable.
The human ceiling
Two halves of the same real sample do not agree perfectly with each other either. userken measures that by repeatedly splitting the held out respondents in half and scoring one half against the other. That is the ceiling: matching it is as good as this question can be answered, and a synthetic number above it means something is wrong, not that the method is superhuman.
The hold-out method
The default split is by time. Reviews from before a cut-off are the training side; reviews from the test window, the most recent 90 days by default, are held out. Questions are generated from the held out reviews, the synthetic audience answers them with retrieval restricted to the training side, and the two are scored. The windows never overlap: a run whose training window runs past the start of the test window is refused.
Question families cover ratings, ratings within a theme, how prevalent a theme is, pairwise comparisons, splits by platform, sentiment about a feature, and external benchmarks. A question needs at least thirty real answers behind it before it is scored.
Production personas were built from all of the reviews, including the test window. The time split stops retrieval seeing the future, but not the clustering. The leakage-safe number is produced by rebuilding personas from the training window only, and the published report says which kind of run it is.
Baselines
A score means nothing on its own. Every question is also answered three other ways:
- Uniform. A flat distribution over the options. The floor: beating this is the minimum bar.
- Corpus prior. The same question answered from the training window, which is to say repeating history. Anything that cannot beat this has told you nothing you did not already have.
- Ungrounded model. The same model with the same personas but no reviews retrieved. The gap between this and the grounded number is what the real data is worth.
Reading the accuracy page
/accuracy is the public page and /app/accuracy is the same report inside the app. What is on it:
- The headline, NDAM and 1-MAE for the latest published run, with the human ceiling beside them.
- The baselines, so you can see the margin.
- By task and by audience, because the method is much better at some questions than others, and that breakdown is the useful part.
- The hold-out: the method, the dates and how many reviews were on each side.
- External benchmarks, scored against published surveys, reported separately and deliberately kept out of the headline: app reviewers are not the general public, and mixing the two populations would make the headline flattering and wrong.
The full write-up is at /accuracy/methodology.
First benchmark pending
Before the first evaluation completes, the page renders in full with every tile marked pending and says so:
userken scores its synthetic audiences against held out real reviews and published surveys. The first evaluation has not completed yet, so there are no accuracy numbers to show and none are estimated in the meantime. Nothing on this page is ever typed in by hand.
That state is deliberate. A pending page is more useful than an invented number, and every figure on the page, when there is one, comes from a stored evaluation run.
On the Team plan you can have an accuracy report run against your own audiences rather than only ours. It is metered like other work; see Plans and credits.