Running a survey
A survey asks an audience structured questions and returns a distribution rather than one averaged opinion. Each persona answers several times, the answers are weighted by persona size, and the result carries intervals, a per-persona split and verbatims in the audience's own words.
Templates
Five templates write the question set for you at /app/surveys/new. Each is also an MCP tool, so Claude can run the same thing.
| Template | Questions it writes |
|---|---|
| Concept test | Appeal on a five point scale, usage intent, an open question about concerns, and willingness to pay in bands. |
| A/B/n test | A forced choice between your variants, an open question about why, and an appeal rating for each variant. |
| Name test | A forced choice between names, a fit rating for each name, and open reactions. Needs at least two names. |
| Message test | Clarity, relevance and motivation ratings for every message, then a forced choice between them. |
| Pricing test | The four Van Westendorp price questions, parsed to numbers, plus a Gabor Granger purchase intent ladder. |
Templates are a starting point. Edit, delete or add questions before you run. You can also skip templates and write your own, or ask userken to suggest questions for a research goal: those suggestions are grounded in the audience's personas, its top themes and ten to fifteen real reviews closest to your goal, then checked for leading wording, double barrelled questions and duplicates before you see them.
Question types
| Type | What a persona returns |
|---|---|
| Single | One option. Reported as shares. |
| Multi | Any number of options. Reported as shares plus the mean number of selections. |
| Likert | One point on an ordered scale of three to seven labels. Reported with a mean, a top two box and a bottom two box. |
| Rank | The options in order. Reported as a mean rank, the resulting order, and how often each option came first. |
| Open | Free text. Reported as themes with shares, plus the verbatims themselves. |
Choice questions take three to six options and Likert scales three to seven labels. Every question also has a purpose, which is how the results page knows which question is the forced choice and which is the appeal rating.
Samples, grounding and cost
Samples per persona
Each persona answers the question more than once, because one draw from one persona is a sample of size one. The wizard offers 1, 3, 5 or 10 samples per persona and defaults to 3. More samples narrow the intervals and cost proportionally more.
Grounding
Grounded is the default and the one to use. Before a persona answers, the question is embedded and the six reviews closest to it from that persona's own segment and star band are retrieved and put in front of it. The review ids are kept on the result, so every answer can be traced to the text behind it.
The alternative, no grounding, is the ungrounded baseline: the same model with no reviews and no persona detail. It exists so you can see what the grounding is worth, and it is the baseline the accuracy evaluation scores against. It is not the setting to use for real work.
What a run costs
Credits are charged when the run starts, from the estimate the wizard shows you. One question at three samples per persona costs one credit; the cost per question rises with the sample count, and theming the answers to an open question adds one credit for that question. If the run fails to start, the charge is refunded. Exact numbers are on Plans and credits.
Runs happen in the background with a heartbeat in the database, so the results page can be closed and reopened, and either host can pick up a run whose owner went away.
Reading the results
The results page for a run shows, per question:
- The distribution, weighted by persona size. The unweighted pooled distribution is kept alongside it, because a large gap between the two is itself a finding.
- Intervals. A 90 percent interval from a two level bootstrap: personas are resampled, then each persona's own draws are resampled, 200 times. It reflects both how few personas there are and how much they disagreed with themselves.
- By persona. Every persona's own distribution, its weight, how many draws it gave and how many were usable. This is where an averaged number falls apart into the segments that made it.
- Verbatims, in each persona's voice, attached to the answer they gave.
- Themes for open questions: named, counted, with a share and a one line summary.
The verdict
The tile at the top is a plain reading of the result, not a separate judgement. For an A/B/n, name or message test it names the leading option from the forced choice. Otherwise it reads the first Likert question: Leans positive when the top two box is at least half and beats the bottom two box, Leans negative in the mirror case, and Mixed otherwise. Alongside it sit the top two box, how many personas lean negative, and for a pricing test an acceptable price point.
Pricing analysis
A pricing test adds the Van Westendorp crossover points and the acceptable range between them, plus a Gabor Granger demand curve with the revenue maximising price on the ladder.
Follow-ups
Any answered question can be drilled into. Ask for follow-ups and userken proposes two or three, written from that question's own distribution, its per-segment split and its verbatims, so they chase the disagreement rather than repeating the question.
A follow-up runs only the new question, and each persona is given its own earlier answer first, so the follow-up is genuinely a second turn with the same respondent rather than a fresh survey.
Exports
Two exports from a run, both untouched by any model:
- JSON at
/app/surveys/{id}/runs/{run_id}/export.json: the whole run exactly as stored. - CSV at
/app/surveys/{id}/runs/{run_id}/export.csv: long format, one row per measurement, with the columnsrun_id, survey_id, question_id, question, question_type, purpose, scope, weight, metric, label, value, text. Overall shares, statistics, per-persona shares, theme shares and every verbatim are all rows in the same file, which drops straight into a pivot table.
Exports are part of the Pro plan and above.