Every research study has a pilot in it whether you planned one or not. Somewhere in the run you find out that a question you were proud of doesn’t work — it means two things at once, or it leads the witness, or it dead-ends into a shrug — and you quietly rewrite it and carry on. That moment is the pilot. The only question is when it happens, and who pays for it.
In classic user research it happens on the clock, in front of a real person, and it is the most expensive form the mistake can take. You recruited someone, scheduled them, sat down across a table or a video call, and spent one of the few hours you’ll get with a real user discovering that your third question is broken. You can fix the wording for the next interview — but you can’t un-ask it of the person in front of you, and you can’t go back and put the corrected version to the five people you’ve already run. The instrument changed halfway through the study, and the earliest, most careful sessions were run on a version of it you no longer trust.
The first three interviews are your pilot. You just paid full price for them, and you can never re-run them on the questions you learned to ask.
A dead question doesn’t announce itself
The failure is quiet, which is what makes it costly. A broken question rarely throws an error — it returns an answer. It returns a confident answer, and you write it down, and it sits in your notes looking exactly like a finding. “How easy and fast was checkout?” gets a cheerful “pretty good,” and only weeks later, when you’re trying to act on it, does it matter that you can’t tell whether they meant easy or fast, or which one the “pretty” was hedging. The question was double-barreled from the start. It didn’t fail loudly. It failed by giving you something usable-looking that you can’t actually use.
You can catch some of these by reading your script aloud, and good researchers do. But the ones that survive a read-aloud are precisely the ones you can’t see from the inside — the wordings that are perfectly clear to the person who wrote them and ambiguous to everyone else. The whole problem with a question you misjudged is that you’re the last person who can tell it’s broken. You need it read back to you by someone who isn’t you.
What a dry run actually buys
That’s the job we built AI User Interviews to do — not to replace the humans, but to rehearse the instrument before it reaches them. You give it your script; it runs the whole thing, moderator and participants both, and hands you back transcripts. The output is not “findings.” It would be a mistake to treat a synthetic participant’s opinion as a real user’s, and we don’t ask you to. The output is something narrower and more honest: a debugged instrument.
Run your five questions against a batch of AI participants and the dead one gives itself away, because a dead question scatters. A working question gathers answers around a shared reading of what was asked; a broken one sprays them across every reading it permits. When half your synthetic participants answer “How easy and fast was checkout?” about speed and half about clarity, that split is the tell — and you saw it in minutes, for the cost of a few tokens, instead of in week three, for the cost of three recruited humans. You rewrite the question, run it again, watch the answers pull together, and only then do you spend a real hour.
Rehearsing the study, not faking it
It’s worth being precise about what changes hands here, because the tempting overclaim — “AI can do your research for you” — is both wrong and beneath the actual value.
What an AI participant cannot give you is the finding. It has never used your product, missed a deadline because of it, or felt the specific irritation of your onboarding flow at 11pm. Its answer to “what frustrated you” is a plausible fiction, and a study that trusts it is a study that learned nothing.
What an AI participant can give you is everything about the study that isn’t the finding — and that turns out to be most of what goes wrong. Whether the question is answerable. Whether it means one thing. Whether the moderator’s follow-up actually digs or just nods along. Whether the order of your questions primes the later ones. Whether “on a scale of one to five” is doing any work or whether everyone parks on four. None of that requires a real user to surface. It requires someone to run the script and answer honestly-shaped answers, enough times, fast enough that you can fix and re-run. That is a rehearsal, and rehearsals are how every other discipline that performs live keeps from breaking on stage.
An AI participant can’t tell you what your users think. It can tell you whether your questions are capable of finding out — which is the part you were going to get wrong.
The economics were the whole point
Underneath all of this is a cost curve, and it bends the same way it bends across everything we build: drop the price of a rehearsal far enough and the rehearsal stops being a luxury you skip under deadline and becomes the default first step. When piloting a study meant recruiting a second set of humans just to test the questions you’d ask the first set, almost nobody piloted; the “pilot” was whatever you could salvage from the early real sessions. When a pilot costs minutes and a handful of tokens, there’s no reason to send an untested instrument into a room with a real person. You wouldn’t ship code without running it once. Research is the same kind of artifact — a thing that behaves differently than you imagined the moment it meets reality — and it deserves the same courtesy of being run before it’s shipped.
The five interviews you were always going to burn finding out your questions were wrong are still yours to spend. We’d just rather you spent them learning what your users actually think — with an instrument that has already been read back to you, out loud, by someone who isn’t you.