The Night Shift · Department
code-results
Building code-results to compare deploy dates with changes in business metrics.
9 essays filed here
- 01
The win that wouldn't hold still
You shipped on Monday, opened the dashboard by lunch, and the sign-up line was up. By Friday it had drifted back to nothing. Nobody touched the code in between — so which read was true? Both were, of the data they had. The early number sat inside an interval wide enough to hold a triumph and a dud at once, and every day of data narrowed that interval a little. This is the peeking problem, and it's why code-results makes you name the window before you ship, then reads it once, when it closes.
7 min read - 02
You shipped at the bottom
Sign-ups had their worst week in months, so you shipped a fix. The next week recovered, and the room gave the fix the credit. But the worst week was, by definition, the point the number was most likely to bounce back from — the recovery was loaded into the timing before your code ever ran. This is regression to the mean, the quietest way a deploy takes credit for a rebound it had nothing to do with, and the only thing that can tell the fix apart from the bounce is a control that got no fix and recovered anyway.
6 min read - 03
Up in total, down in every part
You ship a change, and conversion climbs nine percent. Then someone splits the number by cohort and finds the truth: new users converted worse, returning users converted worse, every segment converted worse. The total rose anyway — because the release quietly changed who showed up, not how well the product worked. It's Simpson's paradox, and it's the trap a single headline metric is built to walk you into. code-results reads the number the only way that can't lie to you: split, and weighted by the mix that actually moved.
6 min read - 04
Two changes, one window
Ship one thing on a quiet afternoon and the number that moves after it is yours to read. Ship two things twenty minutes apart and the lift has no owner — it belongs to both changes and neither, and any clean split you draw between them is a number you made up. The honest move isn't to guess which deploy earned the win. It's to notice that, for now, the question can't be answered — and to say so out loud instead of picking a winner.
6 min read - 05
The change that moved nothing
Ship a change, watch the metric, and sometimes the line just… doesn't move. Most tools treat that as a non-event — no green arrow, nothing to celebrate, nothing to file. But a confirmed 'no effect' is a real result, and burying it is how a team pays twice: once to build the thing, and again to relearn that it did nothing. code-results stamps the flat line as a finding, not a gap.
6 min read - 06
The number you weren't watching
A change is a trade, not a gift. You ship to move one metric, you watch that metric, it moves, and you call it a win — while the number you never instrumented quietly moves the other way. The bill goes unread because it was never on the screen. code-results reads every deploy against the whole handful a business would notice, so the win and the cost arrive together, on the same board.
6 min read - 07
The result has a half-life
The answer to 'did it work?' isn't worth the same on Monday as it is three weeks later. Not because the number changes — because you do. The reason you shipped, the alternatives you weighed, the file you were nervous about: all of it decays out of memory on a curve. A verdict that arrives after the why is gone is a fact you can't act on. code-results is built to catch the answer while the reason is still lit.
6 min read - 08
The number moved. Was it you?
Sign-ups are up since you shipped. That feels like a win — but 'up' is a level, and a level is the one thing that can't tell you whether your change did anything. The real question is whether the line broke where your deploy landed, and that's a question of alignment, not analytics.
5 min read - 09
You shipped it. Did it work?
code-results puts deploys alongside business metrics so the team that shipped a change can see what happened afterward, with a clear distinction between correlation and causation.
3 min read