A traffic camera does not send you a car. It sends you frames — thirty a second on a good feed — and an object detector, doing exactly its job, draws a box around the car on every single one of them. One sedan easing through an intersection over four seconds is not one detection. It is a hundred and twenty boxes, each landing a few pixels from the last, each an honest report that right now, here, is a car. Pin every one of those to the map and you have invented a traffic jam out of a single quiet Tuesday commuter.
This is the problem that hides behind the demo. Detection is the part that looks hard and turns out to be mostly solved: a model spots the car, draws the box, labels it, and the box is genuinely correct. The part nobody films for the demo is the question that comes after every box is correct — which of these hundred and twenty correct boxes are the same car? That question is called tracking, or data association, and it is where a live map either tells the truth or dissolves into noise.
The detector has no memory
It helps to be precise about what a frame-by-frame detector actually knows, because it is less than you’d think. Run detection on a single frame and you get a list of boxes with labels and confidences. Run it on the next frame and you get another list. The detector does not tell you that box #3 in this frame is box #3 from before, one car-length further along. It doesn’t tell you anything about before. Each frame is a fresh, amnesiac glance. The model that is superhumanly good at what is in this picture has, by construction, nothing at all to say about what was in the last one.
So the boxes arrive as an unlabeled pile. Thirty times a second the map is handed a fistful of rectangles and asked, with no identifiers attached, to decide how many things they represent. Get that decision wrong in the generous direction and every car becomes a swarm of phantom cars flickering in place. Get it wrong the other way and two real cars passing each other merge into one and half your traffic vanishes. The map’s accuracy is set almost entirely by how well it answers a question the detector never touches.
The model is brilliant at the frame and blind to the film. Everything a map cares about — how many, which way, how fast — lives in the film.
Stitching the boxes back into a car
The way you answer it is to give the boxes the memory the detector lacks. You carry forward a small set of tracks — running guesses about the real objects on the scene — and each new frame you match the fresh boxes to the tracks you already hold.
Matching is a prediction problem before it is a bookkeeping one. A track isn’t just “the last place I saw this car.” It carries a little model of motion: this car was here, moving this fast, this direction, so in a thirtieth of a second it should be about there. When the next frame’s boxes come in, you compare each one to where each track predicted it would be. A box that lands almost exactly on a prediction is almost certainly that car, one step on, and it extends the track. A box that lands nowhere near any prediction is something new — a car that just entered the frame — and it starts a track of its own. A track whose predicted box fails to show up for a few frames is a car that has left the scene, or ducked behind a truck, and after a short grace period you retire it.
None of this needs the model to be cleverer. It’s ordinary geometry and a stopwatch: where should this thing be by now, and which of the new boxes is closest to there. But it converts the amnesiac pile into something with continuity. The hundred and twenty boxes stop being a hundred and twenty things and become one thing, observed a hundred and twenty times.
The count was never the real prize
Collapsing the pile to a true count is the obvious win, and it matters — a map that double-counts is worse than no map, because it double-counts most exactly where traffic is heaviest and the reader most needs it to be right. But the association buys you something you could not have gotten any other way, and it’s easy to miss because the detector’s output looks so complete without it.
A single box is a location and nothing else. It cannot tell you which way the car is going, because direction is a comparison between two moments and a box is one moment. It cannot tell you how fast, for the same reason. Speed and heading are not properties of a car in a photograph; they are properties of a car between photographs. The instant you’ve decided that this box and that box are the same vehicle a fraction of a second apart, you can subtract their map positions and divide by the time, and out falls a velocity — a heading and a speed — that was never present in either frame on its own.
That is why the track is worth the trouble even when nobody’s counting. A pile of correct boxes tells you a car is there. A track tells you a car is going somewhere, at a pace — and a map of vehicles that knows where each one is headed is a different, far more useful object than a map of anonymous dots blinking on and off. The heading and the speed are not a feature we bolted on. They are the change in position over time, which is precisely the thing that association reconstructs and a frame-by-frame detector throws away thirty times a second.
Where it gets hard, and stays honest
The clean version above is the version that works on a quiet lane. Real feeds are less polite, and it’s worth being straight about the seams, because they’re the reason this stays interesting.
Two cars that pass close together swap the predictions they’re competing for, and a naive matcher can hand car A’s track to car B and vice versa — an identity switch that leaves the count right but the trajectories crossed and wrong. A car that disappears behind a bus for a full second violates the “should be about there” assumption, so the grace period has to be long enough to forgive an occlusion but short enough not to keep a ghost alive after the car has genuinely gone. Cameras that pan or zoom — the ones we’ve written about before — shift every box at once, so the motion model has to be told the difference between the world moving and the lens moving before its predictions mean anything. And frame rates on public feeds are neither high nor steady; when the feed stutters from thirty frames a second to three, a car covers ten times the distance between glances and “about there” becomes a much larger, much less certain circle.
We don’t pretend those are solved. We pick conservative defaults — err toward starting a new track rather than forcing a bad match, err toward retiring a stale one rather than teleporting it across the frame — and we tune them against the failure that actually shows up on the map, which is a phantom crowd, not a missed car. A vehicle we drop for a second and re-acquire as a new track costs the map a little continuity. A vehicle we invent by pinning every box costs the map its credibility. Given the choice, we spend the continuity.
The demo is the detector drawing a crisp box around a car, and it’s a good demo. The product is everything that has to be true so that the box, drawn thirty times a second, adds up to one car with somewhere to be — instead of thirty cars with nowhere.