You shipped the new onboarding on Monday morning. By lunch you had the dashboard open, and there it was: sign-ups up six percent against last week. You told the team. Somebody added a rocket emoji. On Tuesday it was four. By Friday the line had wandered back down to about where it started, and the rocket looked a little silly. Nobody had touched the code since Monday. So which number was real — the six on Monday or the one on Friday?
The uncomfortable answer is that both of them were honest reports of the data available at the time, and neither of them was the effect of your change. The effect of your change is a single number you never actually get to see. What you get instead is an estimate of it, and every estimate arrives wearing an interval — a band of “it’s somewhere in here” — and on Monday that band was wide enough to hold a triumph and a dud at the same time.
The number is never a point
We talk about experiment results as if they were points. “Sign-ups went up six percent.” But six percent is the middle of a cloud, not a dot. The honest way to say it on Monday is: the effect is somewhere between minus four and plus sixteen, and our best single guess is six. Written out like that, nobody adds a rocket. Minus four means the change might have quietly hurt. Plus sixteen means it might be the best thing you shipped all year. A range that contains both of those isn’t a finding. It’s a shrug with a decimal point.
The width of that shrug is set by how much data you’ve collected, and on Monday you had one morning. A handful of visitors, an ordinary Monday’s worth of noise — the weather of a normal day is louder than the signal you’re straining to hear. The point estimate bounced to six because a few extra people happened to sign up before lunch, the way a few extra people happen to do anything on any given morning. It would have bounced to two, or nine, on a different Monday with the same code.
The early number isn’t wrong. It’s just mostly made of the day, and only a little bit made of your change.
What changes over the week isn’t the effect. It’s the band around the effect. Every additional day of visitors is another few thousand coin flips averaged into the estimate, and averages of more flips sit closer to the truth. The band narrows. It doesn’t narrow evenly — it closes fast at first and then grinds, because it shrinks with the square root of the sample, so the fourth day of data buys you far less certainty than the first. But it closes. And as it closes, the estimate stops being free to wander. By Friday the band was tight enough that the line had nowhere to go but where it actually was: a small nudge, maybe real, maybe nothing, but nowhere near a win.
Why looking early is worse than not looking
Here’s the part that turns a statistics footnote into an operational rule. It isn’t just that the early number is noisy. It’s that watching it is a trap, and the trap gets worse the more you care.
Leave a wide-banded estimate wobbling on a screen in front of a team that shipped something they’re proud of, and someone will look at the exact moment it’s high. Not because they’re dishonest — because they check more often when they’re excited, and the check that lands on a peak is the one that gets screenshotted. You’ve turned a single measurement into a search: you keep glancing until the noise arranges itself into the answer you wanted, then you stop. Statisticians call it peeking, and it has a brutal property. If you let yourself call a winner the first time the line crosses the win threshold, a change that does nothing at all will still eventually cross it, just from wobbling, if you watch long enough. Enough peeks and you can “prove” anything, because you’ve stopped measuring the change and started measuring your own patience.
The fix isn’t more discipline about when you glance. Discipline fails exactly when it matters, on the morning of the launch you believe in. The fix is structural: decide the rules before you have any data to be seduced by.
Name the window before you ship
So code-results makes you do the boring thing first. Before a deploy goes out, you set the window — how long you’ll collect before you read the result — and, if you want a clean decision, the threshold that counts as a win. Both of those get chosen while you still have no idea which way the numbers will break, which is the only moment you can choose them honestly. Then the tool holds the line. During the window it will happily show you the estimate and its interval, so you can watch the band close, but it won’t let a fat-banded Monday reading masquerade as a verdict. The verdict is a thing that happens once, when the window closes.
Naming the window in advance does something quietly powerful: it converts “keep looking until it looks good” into “look once, at a time you picked for reasons that had nothing to do with the outcome.” A result read at a pre-committed moment means what it says. A result harvested from whenever the line peaked means only that the line peaked.
It also, mercifully, kills the Monday-versus-Friday argument before it starts. There’s no debate about which read was real, because there was only ever going to be one read, and everybody agreed to its date before anyone knew what it would say.
What the window is really buying you
The window isn’t a delay you tolerate for the sake of rigor. It’s the thing that makes the answer mean something. A one-day read and a one-week read aren’t the same measurement at different speeds; they’re different measurements. The first is mostly a report on Monday. The second is mostly a report on your change. The only way to move from the first to the second is to spend days, and no amount of staring at the dashboard on day one substitutes for the days you haven’t collected yet.
There’s a version of speed that’s just impatience wearing a lab coat — shipping fast, then grabbing the first number that flatters the ship. The whole point of measuring deploys is to be faster in the way that compounds: you learn what actually worked, so the next change is aimed better. That only happens if the numbers you learn from are true, and a number is true when it’s read at a moment you committed to before you could be tempted, with a band tight enough that the estimate has stopped being free to wander.
The line went up on Monday. It came back down by Friday. Nothing about the deploy changed in between — only how long you let the noise average out. The win that wouldn’t hold still was never a win. It was a wide interval, caught at its widest, by someone who wanted good news at lunch. Name the window. Read it once. Let the band decide.