Under the hood

Same story, twenty sources

This section is currently available in English only.

When a species comes back from the brink, every nature outlet covers it. Twenty versions of the same event land in our pipeline within hours. Which one do you keep? And what do the other nineteen tell you?

These are two different questions. The first is about removing repetition. The second is harder than it looks, and the honest answer today is: less than you might hope.

The dedup problem

Our pipeline collects from outlets around the world. Every four hours, new articles arrive. When something genuinely newsworthy happens, a dozen outlets publish within the same day. Showing all of them wastes your attention.

The naive approach is keyword matching: if two titles share enough words, they're the same story. We tried this. Title Jaccard similarity (the overlap between the sets of words in two titles, divided by their union) seemed promising. In practice, it barely works.

Why naive approaches fail

We analyzed duplicate pairs across several weeks of data. The Jaccard scores between titles that humans would call "the same story" ranged from 0.00 to 0.33. That's essentially random.

Why so low? Because different outlets frame the same event with entirely different vocabulary. One writes "Wolf population rebounds in Germany." Another writes "Predator numbers surge in Central Europe." Same story. Zero word overlap after removing stop words.

Multilingual sources make this worse. The same event appears in English, Dutch, German, and Spanish with no lexical overlap at all. That is why the grouping now compares meaning, not words: a multilingual model that compares meaning places the wolf headline and the predator headline close together, in any language.

What we actually do

Three steps, in three services:

Collection (FluxusSource) drops exact copies: the same text arriving from several outlets. A wire story that an editor lightly rewrote is no longer an exact copy, so it passes through as a separate article.

Scoring (NexusMind) groups articles about the same story by meaning, across languages, and picks one article to represent each group.

The site (ovr.news) shows one version per group. A title-comparison rule (Jaccard similarity) also exists in our editorial layer, but it is switched off: paraphrased titles defeated it.

It errs both ways. Sometimes one story stays split into two cards. Sometimes two different stories on the same topic land in one group: in one check in October 2026 (rated by an AI model, not by people), about four in ten pairs we flagged as the same story coming back within a week were in fact different stories. When two stories land in one group, only one is shown and the other is hidden. We accept that cost for now.

What the source count tells you

Coverage by several outlets looks like evidence. A reef recovery story covered by one blog could be aspirational; the same story covered by a news agency, a research institute and a local paper seems firmer. But we cannot yet tell independent reports apart from one agency story republished by many outlets. So the count says how many outlets in our collection were grouped with this story, not that they confirmed it independently. It can be too low (exact copies were dropped earlier) and too high (a different story sometimes lands in the group).

When several outlets report the same story (grouped by NexusMind's cross-source analysis), we show one version. Once a version has been shown, it stays the one we show; otherwise it is the version NexusMind picked to represent the story, if we have a summary for it, or else the highest-ranked version we can show. Being covered by several outlets does not move a story up the feed.

The result: you see one version of each story. The article page shows how many sources were grouped into the story, counting the one you are reading. It does not name them: when we checked whether a named outlet really covered the same story, it was right somewhere between about half and six times in ten, whether we picked the most credible outlet or one at random. That is not reliable enough to name an organisation, so we stopped showing names.

Grouping vs. the count

Grouping and the source count use the same raw material for different jobs:

What we haven't solved

Independence. Nothing in our pipeline checks whether two outlets reported a story separately or both republished the same agency copy, so the source count cannot mean corroboration in the journalistic sense.

Precision. Grouping by meaning sometimes joins different stories on one topic, and sometimes splits one story. The grouping is being worked on upstream; until then we keep removal conservative.

Last updated: November 2026