SEO / developer tools · Own product (open source)
An open-source MCP server and CLI that crawls a site, finds technical SEO problems, suggests internal links from local embeddings and shows what changed between two crawls.
The challenge
SEO crawlers produce tables with thousands of rows. Someone still has to read them, decide what matters and turn them into a list of fixes, and that is the slow part of every audit.
AI clients like Claude and Cursor are good at that kind of reasoning, but they cannot see a site's structure. They can read a page or two, not crawl a thousand pages, follow every redirect, count inlinks and compare this week with last week.
Internal linking is the most manual part of the work: which older articles should link to a page, in which paragraph and with what words. Crawlers report orphan pages but do not say where the missing links should come from.
The approach
Crawlmind follows links and sitemaps, obeys robots.txt and Crawl-delay, and pauses between requests to the same host. For every page it records the redirect chain, response time, title, canonical, headings and each link with its place on the page: menu, footer or content. It reads HTML as the server sends it and does not run JavaScript.
After each crawl, 28 checks run over the same data: broken links, redirect chains, canonical errors, duplicate titles, thin content, orphan pages, slow responses and more. Each issue comes with examples and a recommendation, and the link graph adds click depth and internal PageRank.
A local multilingual model embeds every page and every paragraph of its main content. For pages on the same topic that do not link to each other yet, Crawlmind returns the paragraph that fits best and anchor phrases taken from it, matched by word stems so inflected forms count. The AI client writes the final sentence.
Every crawl is kept in a local SQLite database, so any two can be compared: new 404s and 500s, new broken links, pages that lost inlinks, slower responses, changed titles and canonicals.
Sixteen MCP tools return short text an agent can act on, with structured data alongside. Long crawls run in the background and report progress while the agent waits. Two ready-made prompts run a full audit or build a linking plan, and the embeddings add-on installs itself after the user agrees, without restarting the client.
A crawled page can hide instructions for the AI. Crawlmind refuses private and local addresses on every connection and redirect, DNS rebinding included, and writes reports only to its own folder unless the user allows more. A hidden prompt cannot turn the client into a scanner of the local network or make it overwrite files.
In motion
One question starts a crawl, waits for it, reads the summary and the issues, compares the result with last week and comes back with a ranked list of fixes.
The terminal counts pages as they arrive, then prints the summary: status codes, links, response times and the top issues with examples.
A suggestion names the source page, the paragraph and anchor phrases from it. The same words in the page become a link, and the target gains a content inlink.
See it in action
The outcome
Send a short brief about your project. Within 24 hours you get an estimate or a clear next step, even if we don't work together.