The nicest thing a data source can do for you is ring. A webhook, a push channel, a long-lived socket that stays open and whispers changed the instant something does — any of these turns the hard problem inside out. You stop asking and start being told. Your app sleeps until the world pokes it, spends nothing while nothing happens, and is never more than a few milliseconds stale. Most of the apps on your phone live this charmed life. They are told.

The Drudge Report tells no one anything.

There is no push. There is no webhook. There is no RSS to poll politely, no API with a since cursor, no Last-Event-ID to resume from, no queue you can subscribe to. There is a page — hand-set HTML, re-edited dozens of times a day by a human with an eye — and the page has exactly one way of telling you it changed, which is that it changed. If you want to know, you have to go and look. And since you can’t stand there staring at it forever, “go and look” becomes: knock, wait, knock again.

The two bad numbers

The moment you accept that you’re polling, you’re handed a single dial — how long to wait between knocks — and both ends of it are wrong.

Turn it down, knock every few seconds, and you catch every change almost the instant it lands. You also become a machine hammering a page you don’t own, hundreds of times an hour, the overwhelming majority of those requests fetching a wall that hasn’t moved a pixel since the last one. It’s rude, it’s fragile — the fastest way to get an IP quietly throttled is to behave like something that needs throttling — and, worst of all, it barely helps. The reader is not refreshing their eyes every four seconds. The news being four seconds old instead of ninety changes nothing about their morning.

Turn it up, knock every ten minutes to be a good guest, and you’ve solved the rudeness by manufacturing staleness. A story breaks, the wall reorganizes around it, the siren goes up — and for up to ten minutes DReader is calmly serving the arrangement from before, with no idea it’s holding a stale paper open in front of someone. On the one kind of day the whole thing exists for, it’s asleep.

A single fixed interval is a bet that the news breaks at a constant rate. It never has.

That’s the trap under every naive poller: you’re forced to pick one cadence for a page whose whole character is that its cadence isn’t constant. Three in the morning and the middle of a breaking story are not the same page arriving at two speeds. They’re two different rhythms, and any one number is wrong for at least one of them.

The page is already telling you how fast to go

Here’s the turn. The signal you need to set the interval is the same signal you’re polling for.

When the Drudge Report is being edited heavily — a headline swapped, resized, moved, a second and third link added underneath — that flurry is not just the thing worth refreshing. It’s also a near-certain sign that more edits are coming in the next few minutes, because that’s how a breaking story behaves: it doesn’t land once, it develops. And when a poll comes back byte-for-byte identical to the last several, that’s the page telling you, in the only language it has, that nothing is happening and nothing is about to.

So DReader doesn’t hold the dial still. It reads its own recent history and moves it:

  • A poll that found a change tightens the interval — sharply. The wall just moved; the odds it moves again soon just jumped; get closer.
  • A run of polls that found nothing relaxes it — geometrically. Each identical answer roughly doubles the wait, the way anything sensible backs off: quick to suspect, slow to fully stand down.

The result is a cadence that breathes with the page. During a breaking story it collapses to a knock every twenty or thirty seconds, riding the edits almost as they happen. Through a dead overnight hour it stretches out toward one knock every several minutes, sitting quietly on a page that has nothing to say. Nobody configured either number for the moment; the page’s own volatility chose them.

Both ends are fenced, of course — a floor so an active story can’t drive the interval to zero and turn DReader back into the hammer, and a ceiling so a genuinely quiet stretch never lets it drift so far out that it sleeps through the next break. Between those two rails, the rhythm is the page’s to set, not ours.

The knock that costs nothing

Adaptive timing decides when to knock. There’s a second, quieter discipline about what each knock costs — because even a well-timed poll that drags the whole page down the wire on every check is a poll you should be ashamed of.

The web has a built-in answer for exactly this, and it predates the whole push era: the conditional request. On each fetch DReader sends back what the server last told it — an ETag, a Last-Modified — and the server, if nothing has changed, answers 304 Not Modified with an empty body. No markup, no rebuild, a few dozen bytes over the wire to confirm still nothing. The great majority of DReader’s knocks — all the quiet ones, which are most of them — cost almost nothing at both ends. Only the poll that actually catches a change pays for the page.

That’s what makes the fast end of the adaptive range affordable to a stranger’s server. A knock every twenty seconds during a breaking story sounds aggressive until you notice that nineteen of every twenty are 304s that never touch the page’s real work. You’re not asking it to serve you twenty times a minute; you’re asking it has anything changed? twenty times a minute, and paying the full price only when the honest answer is yes. Add a little jitter so a fleet of readers never falls into lockstep and knocks in a thundering unison, and the whole scheme is a considerate guest: present exactly as often as the news warrants, invisible the rest of the time.

The rhythm was never yours

It’s tempting to think of a reader like this as a thing you tune — that somewhere there’s a correct polling interval, and the craft is in finding it. There isn’t. The page you’re mirroring is edited by a person on no schedule, fast when the world is fast and still when it’s still, and the only honest cadence for reading it is one that is fast when it’s fast and still when it’s still. DReader doesn’t own that rhythm. It listens for it, on a page that will never once ring to announce it, and keeps time.