trolls.dev
05 The workbench — case study

code-results

See what changed after your code shipped.

code-results puts each deploy alongside business metrics such as sign-ups and revenue. The team that shipped a change can compare the numbers before and after, then investigate what might explain the difference.

Status
Live
Platform
Web
Field
Product analytics · Engineering
Cadence
Every deploy
Open code-results ↗ All work
code-results.com Live
The ship
deploying… ✓ deployed a1f3c92 · ship checkout redesign
The result
Sign-ups
+18%
Activation
+12%
Revenue
+9%
a1f3c92 → +18% sign-ups after deploy
dashboards opened · 0 release metrics
After a release, the team should be able to check what changed. The bet behind it

Why we built it

The change shipped. Did the numbers move?

A release can pass its tests and run without errors while still missing its business goal. Engineers need a way to check what happened after the deploy, especially once they have moved on to the next ticket.

The relevant numbers may live in another team’s analytics tool or appear in a monthly review. By then, several releases can be mixed together, making an individual change harder to investigate.

code-results connects the release history to those metrics. It gives the people who wrote the code a view of what moved and a link back to the change.

A change in a metric is a reason to investigate. It does not, by itself, prove that a particular deploy caused it.

What we made

Review a release alongside its metrics.

  1. 01

    Mark each deploy

    code-results records when a change goes live so you can compare metrics before and after it. You can look at Tuesday's checkout rewrite separately from the rest of the quarter.

  2. 02

    Track business metrics

    The tool follows outcomes such as sign-ups, activation, revenue, and retention. You can see whether those numbers changed after a release.

  3. 03

    Compare against a baseline

    Each change is shown against the trend before it shipped, with a link back to the commit. That gives the team a starting point for investigating a rise or drop.

  4. 04

    Get results to the team

    The readout is designed for the engineers who shipped the change, so they can review it while the implementation is still fresh in their minds.

What we decided

Keep the readout useful and checkable.

We kept the summaries focused and made their limits explicit.

  1. 01

    Label correlation clearly

    We show what moved after a deploy. That alone cannot establish what caused the movement; other releases, seasonal changes, or outside events may also matter.

  2. 02

    Give each release a summary

    Each release gets a focused readout. The team can check it before deciding what needs a closer look.

  3. 03

    Focus on business outcomes

    We chose metrics such as activation and revenue to help teams assess the goal of a change after it ships.

Further reading from the journal

We wrote about running our own products in Why we only build what we run, and about keeping their scope manageable in In defense of small software.

code-results reports changes in metrics after a deploy. Teams still need to investigate the cause before attributing a result to a release.

Colophon End of Case Study 05

Check the numbers after your last deploy.

Review the metrics alongside a release while the team still remembers the changes it made.

Open code-results ↗ More case studies →