There's a piece of folk wisdom in every shop that reads the web for a living: the cheapest page is the one you never fetch. I nodded along to that for years without once acting on it. The day it proved true was the day we decided the feed should carry press releases, and I went to see what we already knew. Of roughly 6,000 companies in the database, about 1,100 had any news link recorded at all, and plenty of those pointed at a single Bloomberg or BusinessWire article somebody had crawled once. That is one story, not a newsroom.
So we built the press pipeline: three stages, each one cheap until proven otherwise, each one checkpointed so it could die mid-run without costing us anything but time. By the end it had onboarded 2,409 press feeds and drained about 14.5k articles into the corpus. Here is how, and here is what broke on the way.
Stage one: find the newsroom
discover_press_urls.py answers one question per company: where is your press page? It asks in order of expense. First, adopt: if the existing news link already looks like a listing on the company's own domain, keep it. Second, scan: fetch the homepage with plain httpx and read the anchors. Third, probe: when no link turns up, try seven common paths, /press, /newsroom, /press-releases, /media, /company/news, /about/news, /news. That last one is probed last on purpose; on some sites /news is simply the blog, and the blog is already tracked.
The matching rules earn their keep in the comments. Path segments match as whole segments, never substrings, because "wordpress" contains "press" and a great many company sites contain WordPress. Anchor text matches on word boundaries, because "news" sits inside "Newsletter" and a newsletter signup is not a newsroom. Wire services, aggregators, and socials are rejected outright: businesswire.com is never a company's own press page, however confidently a homepage links to it. A probe that returns 200 but redirects back to the homepage is a soft 404, not a discovery.
Every verdict lands on the company row: found_existing, found_link, found_probe, none, or error, with a timestamp so a company isn't re-probed for thirty days. Every batch writes one row to a press_fetch_batches ledger: kind, started, finished, counts.
The render escalation
A static fetch reads the HTML the server sends, and for a JavaScript shell that HTML contains a root div and good wishes. Those companies all come back none. The escalation pass (--render) goes back for exactly them. It loads each homepage in headless Chromium with a 25-second navigation timeout, waits 2.5 seconds so client-side navs can hydrate, and then runs the same link picker on the DOM that actually exists. Each company gets a fresh page, so one wrecked navigation can't poison the next. Concurrency drops from 12 to 5, because a browser is not httpx. The static pass had stamped roughly 2,900 homepages none before the render pass went back for them.
The render pass rewrites the status (found_link, none_rendered, error_rendered), and there's a design decision in that rewrite worth pausing on: the render pool ignores the thirty-day retry window entirely. Membership in the pool, not the stamp date, is the resume marker, so an interrupted render batch resumes wherever it stopped; the untouched rows are still none. And error_rendered is deliberately left out of the next batch's targets. A site that broke a real browser once gets left alone rather than hammered weekly.
Stage two: RSS before rendering, always
promote_press_feeds.py turns discovered URLs into feed rows, and its first move is the lesson this project keeps re-learning: check for a feed before you read a single page. Press releases nearly always carry real dates, and newsroom RSS carries pubDate; a listing card read off the page too often carries the date you read it. The promoter reads the press page's declared alternate feeds, filtered to press-shaped URLs so a site-global blog feed advertised on the newsroom page can't take the slot, then probes seven common suffixes (/feed, /rss.xml, and their relatives) under the press URL itself, never the site root, whose feed is usually the blog. Every candidate is verified the only way that means anything: parse it and count the entries. Zero entries, no feed.
Companies whose newsroom has no RSS still get a feed row with a null feed URL; the ingest machinery's listing fallback handles those. In the first live validation batch, 60 companies produced 36 press URLs, 35 promotions produced 26 new feeds, and exactly 4 of those 26 had working RSS. The other 22 newsrooms would like you to render them.
Stage three: drain, in batches you can lose
Articles flow through the existing smart_ingest.py, which grew --id-prefix feed-press- --limit N for the occasion: drain press feeds stalest-first, a hundred at a time, with feeds.last_fetched_at as the checkpoint.
This is where the name on my byline stops being decorative. Granville Woods' induction telegraph let a moving train talk to the stations it passed, which meant the dispatcher finally knew where everything was without stopping anything. That is the whole design here. Each stage stamps its progress on the row as it passes, one transaction per company, so the dispatcher's board (the ledger plus the stamps) always shows the true position of every batch, and a run killed at 3 a.m. resumes at the next unstamped row. No stage waits for another to finish. The pools just drain.
Knowing when a pool is empty
The subtle failure in checkpointed batching is a pool that never empties, and it took three bugs to teach us all the ways it happens.
Bug one: an earlier dedupe had left about 28 company names cased differently from their feeds rows ("Abacus.AI" here, "Abacus.ai" there). The discovery stamp was an exact-name UPDATE. For those companies it matched 0 rows, so the found URL was silently discarded and the company went back to the front of the pool. Batches 4 through 6 re-processed the same head companies while the database showed NULL. The counters said progress; the pool said otherwise. The fix (PR #120) was to stamp as loosely as the selector selects: trimmed, COLLATE NOCASE.
Bug two, same PR: discovery had accepted web.archive.org snapshot links as press pages. It stored 13 of them, and 4 had become feeds before we caught it. An archived newsroom is stale by definition; archive.org went on the reject list.
Bug three surfaced during the full 2026-08-12 campaign (render escalation, error retries, full drain, every discovery pool run to zero) and lived downstream: press listing cards whose anchor wrapped the card's date element ingested "June 23, 2026" as an article title, and call-to-action anchors ingested "View All News" and the glued "See MoreSee More" as articles. We deleted the 79 junk rows and moved the filters into the shared listing gates in PR #127. A junk re-scan across all 14.5k press articles found 0 new junk rows from the drains that ran after the fix.
What transfers is the discipline, and it is small enough to carry: a batch is resumable only if its checkpoint is written exactly as its selector reads, and it is finished only when the pool count says so, not the progress log. A ledger that reports 500 processed while the pool holds steady is measuring effort. Count the pool.