Ask a person, cold, how satisfied they are with something, and you’ll get a number. Say a seven. Now rewind, and before you ask that same question, ask them to tell you about the last time the thing let them down — the export that failed, the sync that stalled, the setting they couldn’t find. Let them sit in that memory for a minute. Then ask how satisfied they are overall. The number comes back lower. A five, maybe a four.
Nothing about the product moved between the two runs. The person didn’t change. The words of the satisfaction question were identical to the letter. The only thing that changed was what they had in their head when they answered — and what they had in their head was put there by the question you asked just before.
This is an order effect, and it is one of the oldest and most thoroughly documented findings in the study of asking people things. Researchers have been measuring it for the better part of a century, under names like priming, anchoring, and assimilation. The specific-then-general pattern above is a classic: a pointed question about one part quietly reweights the general question that follows it, because the respondent, reasonably, assumes you don’t want to hear about the frustration twice and subtracts it from the overall — or, just as often, keeps it front of mind and lets it color everything. Either way, the general answer is now standing on the specific one.
The order is doing work, and you can’t see it
Here is the uncomfortable part. It isn’t that one order is biased and the other is clean. Every order is an order. Ask the general question first and it anchors the specific ones that follow. Ask the specific ones first and they anchor the general. There is no neutral sequence that asks nothing of the questions around it — the moment you put two questions in a row, the first one is furniture in the room where the second gets answered.
Which means the number you write down is never just the answer to the question. It’s the answer to the question in that position, after those other questions. Move the question and the answer moves with it. You have measured the product and the running order together, and stapled them into a single figure that looks like it’s only about the product.
The number you write down is never just the answer to the question. It’s the answer to the question in that position, after everything that came before it.
Now picture how a normal round of interviews gets run. You’ve recruited eight people. Scheduling them was the hard part, so you are not about to burn half of them on a variant. You write one discussion guide — a sensible, logical flow, warm-up to specifics to the big summary question at the end — and you run that same guide, in that same order, with all eight. You do this precisely because consistency feels like rigor: same questions, same order, every time, so the answers are comparable.
And they are comparable — to each other. What they are not is comparable to what those same people would have said under a different order, because you never ran a different order. The order effect didn’t disappear. It got applied, identically, to all eight transcripts, which is the one thing that makes it impossible to spot. A bias that hits every response the same way doesn’t look like a bias. It looks like the answer. You’ll read your eight fives, feel the reassuring agreement between them, and file a satisfaction score that a different — equally defensible — running order would have put a point and a half higher.
The fix is old; only the price was ever the problem
Survey methodologists have known the remedy for as long as they’ve known the disease. You counterbalance: you write the interview in more than one order and rotate which respondent gets which, randomly. Half your people get specific-then-general, half get general-then-specific. Now the order effect isn’t a constant hiding inside every answer — it’s a difference between two groups, and a difference is a thing you can see, size, and reason about. The gap between the arms is the order’s contribution. What’s left, once you’ve averaged across orders, is the closest thing to an answer that doesn’t depend on where you happened to stand when you asked.
The reason this stays in the textbook and out of most real studies is not that people forgot it. It’s arithmetic. Counterbalancing splits your sample. With eight interviews, running two orders leaves four people per arm, and four-versus-four tells you almost nothing — the noise swamps the effect you were trying to measure. So the honest small study makes the only choice it can afford: it picks one order, runs it clean, and quietly hopes the order wasn’t doing much. Sometimes that hope is fine. Sometimes it’s a point and a half of satisfaction that was never really there.
This is the whole case for making interviews cheap. Not faster, exactly — cheap enough that you stop having to spend your sample on a single arrangement of the questions. AI User Interviews runs the same interview with a hundred people about as easily as with ten, which means the running order stops being a thing you commit to and starts being a thing you vary on purpose. Randomize the sequence across respondents. Hold every word of every question fixed and let only the order move. Then read the answer next to how much the order was moving it — the two numbers the small study had to hand you fused into one.
You still have to decide what the satisfaction score really is. But now you get to make that call looking at the spread, instead of at a single figure that a different, equally reasonable order would have quietly rewritten. The question before the question was always doing work. The only thing that ever changed is whether you could afford to watch it.