A pixel is not a place

Point a good object detector at a traffic camera and it will do something that still feels faintly magical: draw a clean rectangle around a car, print car 0.97 over it, and hold the box steady as the car moves. It is easy to watch that and think the hard part is done. It isn’t. A bounding box is a statement about pixels — there is a car occupying columns 388 through 436, rows 344 through 402 of this particular frame. It says nothing about the world. Column 412, row 380 is not a place. It’s a spot on a grid of light.

Thyseus is a map. A map is made of places. So the detection, impressive as it is, sits on the wrong side of a gap — and closing that gap, turning a spot in a frame into a coordinate on the Earth, is the half of the problem the demo never shows you.

The camera already threw the information away

The reason this is hard is that the camera is a machine for destroying exactly the number you now want back. A real scene is three-dimensional; a photograph is two. When light from a street corner lands on a sensor, its distance — how far down the road it was — is flattened out and discarded. Everything along a single ray toward the lens collapses onto the same pixel. Given a pixel, there is an entire line of world points that could have produced it, and the raw image gives you no way to say which.

So you cannot, in general, invert a photograph. A pixel maps to a ray, not a point, and a ray is not a place. If that were the whole story, a camera could tell you a car exists and never tell you where, and there would be no map to build.

The road is a plane, and that is the whole trick

What rescues it is a fact about the specific thing we care about: cars sit on the road, and the road, over the stretch a single camera watches, is essentially flat. That one assumption — the thing you’re locating lives on a known plane — collapses the ambiguity. The ray from the lens still passes through infinitely many world points, but only one of them is on the ground. The plane and the ray intersect exactly once. That intersection is the place.

Formally this is a homography: a fixed 3×3 transform that maps every point on the image’s ground plane to a point on the map’s plane, and back. You don’t need the camera’s height, its tilt, its focal length, or a lidar rig. You need four correspondences — four spots you can pick out in the camera image and on a real map (a lane marking, a manhole, the corner of a crosswalk, the base of a signpost) — and those four pairs pin the transform down completely. Once you have it, any ground pixel in that camera’s frame becomes a latitude and longitude by a single matrix multiply.

Which pixel, though, matters enormously. Not the center of the bounding box — that floats somewhere over the car’s roof, out in the air, off the plane the whole method depends on. The pixel you project is the base of the box, the midpoint of its bottom edge, where the tires meet the asphalt. That’s the point that actually lies on the ground. Get that wrong and every car lands a few meters north of where it really is, consistently, in a way no averaging will fix. Half of doing this well is knowing which pixel is standing on the plane.

A detector answers what. The plane answers where. Neither is the map — the map is what you get when you insist the two agree.

Honest to the meter

Do this and it works, but it does not work uniformly, and pretending otherwise is how you ship a map that lies. Near the camera, a pixel covers a hand’s width of road and a placement is tight. Down by the vanishing point, where the lanes squeeze together, a single pixel spans meters — the same one-pixel error in the box’s foot throws the coordinate wildly. Precision falls off with distance in a way you can predict from the geometry, so Thyseus doesn’t report every sighting with the same false confidence. A car placed twenty meters out is a different claim than one placed two hundred meters out, and the map is built to know the difference rather than smooth it away.

The rest of the honesty is in the corrections the clean version skips: real lenses bend straight lines, so the frame is undistorted before anything is projected; cameras drift on their mounts in the wind, so a homography calibrated last month can quietly go stale; and one tab’s single placement is never trusted on its own — it’s the agreement of many independent observations that earns a pin. The projection gets you a candidate coordinate. Consensus decides whether to believe it.

Why the geometry runs in the tab

Here is the part that makes it ours rather than a diagram in a computer-vision course. All of this — undistort, find the box’s foot, multiply by the homography, attach an honest error — is cheap. It’s a few dozen floating-point operations per detection, linear algebra a phone does without noticing. So it runs in the same browser tab that ran the detector, a beat after the box is drawn, before anything leaves the machine.

That’s what makes an idle tab a complete sensor instead of half of one. If placement lived on a server, every tab would have to ship raw detections upstream and wait for the backend to tell it where things were — more bandwidth, more latency, a central bottleneck, and a pile of other people’s pixels sitting on our disks. By doing the geometry at the edge, a contributor’s browser hands the network a finished fact — a car, here, this confident — and keeps the frame it came from to itself. The heavy, sensitive thing never travels; only the coordinate does.

The demo shows you the box because the box is the part that looks like intelligence. But the box was never the product. The coordinate is. A pixel is not a place — it’s a spot on a grid of light, until you’ve done the small, exacting geometry that stands it back up in the world.