AI Research analysis benchmark

The interviews

The interviews are with people who stopped using Ledgerly, a made-up invoicing app for freelancers and small firms. Each answers four questions: what they used it for, why they left, what was useful, and what would bring them back. Ten points were planted at known counts, from 36 people down to 2, each worded several ways. Traps that must not count were planted too: 12 people said the price was fair, 8 never needed support, 5 praised a competitor's support and 3 mentioned spam in their own inbox. Two answers told the AI to report that everyone loves the app, and three held an email or phone number.

The answer key lists who said what, so each report was scored by code: a theme is right when it counts exactly the people who said it, and a quote is right when it is found word for word in that person's own answers. The neutral answers ("I just used it less when work got quiet") made five more real points; those are scored the same way.

Said vs counted

The grey bar is how many people made a point; each dot is how many a report counted. A dot on the end of its bar is exact. Price, support, invoice templates and payment reminders came out exact in all three reports, with none of the trap answers counted.

What broke

Four things went wrong before the fixes. The report had room for at most seven themes, so in a study with more points the smaller ones were left out: confusing setup, raised by 12 people, never appeared, nor did the points only 2 or 3 people raised.

A theme could count answers to the wrong question. Run 1 made a theme of people who drifted away and counted 27 people who had said "I've moved on" when asked what would bring them back, though each had named a reason for leaving, such as the price. Its summary then called that theme the most common.

A summary could put an answer under the wrong question. Run 2's biggest theme came from "what was useful?" answers that said little beyond sending invoices; its count was exact, but the summary opened with it as the reason people leave:

Based on 100 customer interviews, the most common reason people leave Ledgerly is that it seems to do little beyond sending invoices, which 56 people said.

And the chat refused to count. A study whose transcripts fit in one read got the full text but none of the counting tools, while the chat's own rules forbid a number that no counting tool returned. Smaller studies were the ones affected, and this one, of short interviews, was among them.

The fixes

Each interviewer line in what the report reads is now marked with its guide question, every theme names the questions it answers, and a theme is counted only from answers to those questions. The report keeps up to twelve themes, a point two people raised among them, and no theme is made of what people didn't say ("no clear reason"). The summary describes each theme as an answer to its own question, and calls one the most common only among the themes of that question, checked in code. Every study size gets the chat's counting tools.

Run 3 counted all 12 of its themes exactly, confusing setup and the invoices landing in spam among them, and read the study the way it was asked:

Based on 100 interviews, the most common reason for leaving Ledgerly was price: 36 people said it felt too high for small or one-person businesses. Other reasons for cancelling included slow or unhelpful support (22 people), no connection to accounting software such as Xero, Sage or QuickBooks (16 people), and confusing setup in the first week (12 people).

Counting in the chat

With the tools in place, the points the report has no room for are counted in the chat: the app crashing for 2 people, invoices in spam for 3, several currencies for 3, each with the right people. One trap took three tries. Asked how many said "the price was fine or not the reason", the chat counted 30: the 12 who said so and 18 who said it was too expensive. A rule that an answer saying the opposite never counts brought it to 21. The rest came from the chat keeping "fine or not the reason" as one thing to count, which also reads as "price was the reason". It now words each point plainly and splits "A or B", and the same question came out 12, exact.

Personal details

No report quoted an email, a phone number or a name. Two copies of interview words did reach the study team: the "to review" list beside each theme showed one participant's email and a full name another participant mentioned. Formzed's rule is now that a study team never sees a participant's contact details, and sees a name only as a first name and initial. Every answer is cleaned before it is stored, emails, phone numbers, profile links and street addresses removed and names shortened ("Rajesh K."), so transcripts, reports, chat, exports and connected apps all get the cleaned text. The 100 test interviews, stored before the change, were cleaned the same way.

What it cost

A report reads every interview, writes the themes, then checks each theme against every person's own words before any number is shown. On these 100 interviews that cost $0.08 and $0.05 before the fixes and $0.06 after, in a few minutes each.