The dashboard had its worst week in months. Sign-ups down a third, and not for any reason anyone could name — no outage, no bad deploy, just a run of thin days stacked into a bad number. So you did what a good team does: you found something plausible to fix, tightened the onboarding step everyone had grumbled about, and shipped it Friday. The next week the number came most of the way back. Monday’s standup gave the fix the credit, someone moved the card to Done, and the onboarding change went into the deck as the thing that turned the quarter around.
Here is the part that should have been in the deck and wasn’t. The week you chose to act was not a random week. You shipped because the number was at its worst — nobody scrambles a fix during a good stretch. And “the worst week” is the one week most likely to be followed by a better one, for a reason that has nothing to do with your code and everything to do with what “worst” means.
Extremes are made partly of luck, and luck doesn’t repeat
Any number the business watches is two things added together: a real level, and a week’s worth of noise on top of it — weather, a holiday, a big customer’s billing cycle, plain randomness. Most weeks the noise is small and the number sits near its true level. Once in a while the noise piles up in one direction and you get an outlier: a week that looks far better or far worse than the business actually is.
The worst week you can point to is, almost by construction, a week where the bad luck stacked. And bad luck, unlike the underlying level, does not carry over to next week — next week gets its own fresh draw of noise, which on average is nothing. So the number drifts back toward its true level whether or not anyone touches it. Not because anything healed. Because the thing that made the week extreme was the part that was never going to stay.
The recovery was loaded into the timing the moment you decided to act on the worst week. Your code shipped into a rebound that was already on its way.
This is regression to the mean, and it is the single most reliable way for an intervention to steal credit. It doesn’t require a bug or a bad metric. It requires only that you acted because things were extreme — which is exactly when anyone acts.
The perfect track record that means nothing
The trap has a nasty second half: it makes fixes look like they always work. Think about when you ship reactive changes. You ship them after bad weeks, never after good ones. Every one of those changes is therefore followed, on average, by an improvement — because every bad week is followed, on average, by an improvement. String a year of them together and you get an unbroken record of fixes that “worked,” a wall of green cards, and a growing, wrong belief that the team can pull any number out of a hole by shipping at it.
The mirror image is just as costly. Ship an experiment during a freakishly good week and the number will sag back toward normal afterward — and a genuinely good change gets recorded as a flop, because it happened to launch at a peak that was always going to give some of itself back. The bias doesn’t just manufacture wins. It buries real ones that had the bad manners to ship on a high note.
You cannot think your way out of this by staring harder at the line. The rebound and a real effect look identical from inside the treated group, because they are identical there: the number went down, you shipped, the number went up. Every reactive fix you have ever celebrated has this exact shape. The shape is not evidence.
The only witness is the week you didn’t touch
There is one thing that can separate the fix from the bounce, and it isn’t a cleverer chart. It’s a comparison — a slice of traffic that lived through the same bad week and got no fix at all. A holdout. If sign-ups were going to regress toward their level on their own, the untouched group regresses too, right alongside the treated one, and the two rebounds sit on top of each other. The part of the treated group’s recovery that your code actually caused is the gap between the two lines — and if there’s no gap, there’s no effect, no matter how far the number climbed off the floor.
That gap is the whole ballgame. “The number went up after we shipped” measures the deploy and the weather welded together. “The number went up more than the group we didn’t ship to” measures the deploy. The first is the story that writes itself on a Monday. The second is the one that’s true.
This is why code-results doesn’t just draw the metric after your deploy and call the rise your win. A deploy tied to a number that moved is a deploy tied to regression, seasonality, and every other thing that was moving that week anyway. What tells you the code did something is the counterfactual held beside it — what the same users would have done without the change. Ship at the bottom all you like; shipping at the bottom is often the right call, because the bottom is where the pain is. Just don’t let the timing hand your code a medal the rebound earned. Hold the control beside it and read the gap. If the fix did real work, the gap is where you’ll find it — and if the week was always going to recover on its own, the gap is where you’ll find that out too, before it goes in the deck.