Media / news · News media startup
Articles from many feeds are grouped into stories by meaning, each story gets an AI title and summary.
The challenge
The client is a news media startup. It ran a self-hosted aggregator that pulled articles from dozens of feeds, and editors wrote posts about the stories those articles covered.
One story arrives as many articles from different outlets, hours apart and with different headlines. Editors had to find which ones belonged together, read them all and write the post by hand.
The articles came from many news sources. Some of them protect their pages from bots, so a plain request from the app got no text.
The approach
An editor picks a time window (all unclustered articles, the last 3 or 24 hours, or a custom range) and creates a cluster group. Once every article in it has an embedding, a worker compares them by cosine similarity: each article joins the closest cluster at or above the threshold, or starts a new one. The threshold is a slider in the UI, 0.78 by default, and each group keeps the value it was clustered with, so a group can be re-run at another setting or split up again.
Some sources protect their pages from scraping: the article only appears after the page's scripts run in a real browser, so a plain HTTP request from a bot gets nothing. I got around it by emulating a browser: a small Node.js sidecar keeps one headless Chromium running with Playwright, opens each link with images, fonts and styles blocked and returns the page it lands on. The app then fetches the full text from there and drops links that lead to an article it already has.
Each cluster with two or more articles gets an AI-written title and summary on the enrichment queue, with retries when a model returns broken JSON. The Trending page shows the groups and their stories, with single-article clusters folded away under their own toggle.
The editor picks a cluster and a guideline, and the post is generated in the background. When the full text of every article fits in 128,000 characters, the model writes the post in one pass. When it doesn't, each article is first condensed to a factual extract and the post is composed from those, so no article is cut off. The editor gets a status update over Mercure and then edits the post in a rich-text editor.
AI guidelines are kept in the app, not in the code: tone and voice, article structure, things to avoid, things to verify and a word limit. Editors keep several, switch them on and off and preview each one before using it in a post.
Feed fetches, embeddings, clustering and post generation, AI enrichment, full-text fetches and protected-page fetches run on four Symfony Messenger queues spread over three workers. A Queue Depths panel shows the backlog of each queue, and translation and search indexing can be switched off in the settings when they aren't needed.
In motion
Story bundles from the feeds go in first as fixed clusters. Every other article is compared with the clusters so far: at 0.78 or above it joins the closest one, below that it starts a new story.
Each protected link is opened in headless Chromium, which runs the page's scripts like a reader's browser and lands on the publisher's article. The full text is then fetched from there, and a link that leads to an article already saved is skipped.
Six long articles add up to more than 128,000 characters, so each one is condensed to an extract first. The post is then written from the six extracts, and the status flips to Complete.
See it in action
The outcome
Send a short brief about your project. Within 24 hours you get an estimate or a clear next step, even if we don't work together.