Here is a bet I would have lost. Pick a tag at random from this site's database (there are 88,223 of them) and tell me the odds it names a real topic, something multiple publications are actually writing about. Before I ran the numbers I'd have guessed one in three. Tags come from articles; articles are real; how bad could it be?
Seventeen to one against. Of the 85,551 tags attached to at least one article, 5,048 clear the bar we set for "real topic." The other 94% are noise. This post is about how a corpus gets that way, what we do about it, and why the answer is not what you'd expect.
Where 80,000 junk tags come from
Our tags are not the product of a language model reading each article and judging its themes. They come from keyword-frequency extraction: a script counts the words in a title and abstract, drops stopwords, and keeps what recurs. It is fast and free, and it is wrong in every way that counting words is wrong.
The distribution tells you what that buys. Of 85,551 tags in use:
- 49,126 (57%) are attached to exactly one article, ever.
- 69,326 (81%) appear five times or fewer.
- 5,092 appear in twenty or more distinct sources.
That first number deserves a pause. The majority of the tag table is words that occurred once, in one headline, and will never occur again: misspellings, model numbers, fragments of company names. A tag used once is not a topic. It's a typo with a database row.
The failure modes were sometimes structural. For over a year the two most important tags in a tech-news corpus, ai and ml, did not exist at all, because both ingest paths dropped tags shorter than three characters. Repairing that took an allowlist and a backfill: 37,733 articles gained ai, another 635 gained ml, and the backfill had to exclude French feeds, where "j'ai" tokenizes into a false ai on every other sentence. A singular/plural merge cleaned up roughly 5,300 duplicate pairs, and the first version of the merge script had its own base-rate lesson, preferring the singular form so aggressively that a typo fragment could swallow a real term. It took kubernetes down to kubernete before we caught it.
The gate: filter at read time, not write time
The obvious fix is to clean the table, delete the junk, keep the real topics. We didn't, and the reason is statistical as much as operational. Junk tags are not misbehaving; they're inert. A tag used once costs a row and harms nobody unless a chart or a page treats it as signal.
So the filtering happens where signal is claimed. The tag heatmap on the dataviz page admits a tag only if it appears across at least 20 distinct sources with no single source supplying more than 65% of its uses. That's a diversity requirement: a topic is something the field is discussing, not something one blog says often.
Run the gate over the whole table and 5,092 tags meet the source-count bar; 5,048 also pass the concentration test. Those numbers are worth staring at: the second filter only removes 44 tags. Once a term shows up in twenty independent sources, it is almost never one outlet's private vocabulary. Breadth was doing nearly all the work.
And the gate is a screen, not a proof. Look closely at the heatmap and you'll find actually in the top rows, 361 taggings across 211 sources in a month. Two hundred eleven publications do not share a topic called "actually"; they share a verbal tic our stopword list missed. A diversity gate catches single-source noise. It cannot catch noise the whole field emits together. That one goes on the denylist, and the denominator goes on my list of things to re-check.
The other cleanup: whole articles
Tags are the small version of a bigger question: how much of a quarter-million-article "tech news" corpus is actually tech news? My prior here was more embarrassing. I'd have said 95%; we ingest from curated engineering blogs and newsrooms; how much drift can there be?
The screening pipeline priced it at 22.6%. When we ran a moderation-and- relevance dry run over the backlog, roughly one article in four-and-a-half didn't belong: finance and VC content from business wires, sales and HR posts from company blogs that publish everything, health-supplement marketing from fringe aggregators, and pages whose whole title was "Full Article." The production corpus now stands at about 200,000 visible articles, with the screened-out remainder hidden rather than deleted, so the decision stays reversible and every row can be re-counted when the criteria change.
The screening itself is three gates in sequence, cheapest first: an emoji denylist that catches innuendo headlines no moderation model flags, a moderation API for the content it does flag, and a small-model relevance check that runs only when a keyword fast-path can't already tell the article is tech. Every gate fails open: an API outage means an article stays visible until the next pass, because a screening system that fails closed would quietly empty the site.
Update accordingly
Two priors, both wrong in the same direction. I assumed a corpus assembled from good sources would be mostly signal; the measured rates were 6% signal in the tag table and 77% signal in the article table. Curation at the source does not compose into cleanliness at the aggregate: every scraper, tokenizer, and feed adds its own residue, and the residue compounds quietly until you count it.
The transferable habit is the counting. Before the cleanup work started, someone had to ask what fraction of the tags were real and accept an answer with a denominator in it. If you run a corpus of anything (articles, products, tickets, logs), I'd offer even odds that your equivalent of the seventeen-to-one number is sitting in a table you haven't grouped yet.