We shipped llms.txt - here's exactly what's in it, and what isn't

llms.txt has become a thing you're supposed to have, the same way robots.txt was in 2005 - a file everyone adds and almost nobody checks the correctness of. A lot of the ones we've looked at were hand-written once, and left to quietly rot behind the site they describe.

Ours is different in one specific way: /llms-full.txt isn't a static file we have to remember to update. It's generated per request from the same content sources that power the site - every blog post and docs page, in English and Spanish, pulled live. If a page's title changes, the index changes with it. No stale second copy of the sitemap to forget about.

We're also not going to pretend it's finished. Right now /llms-full.txt covers editorial content - blog and docs - and nothing else. It does not list individual MCP servers or bundles. If an agent asks "what MCP servers does this marketplace have," the honest answer today is "read /llms.txt, follow the link to Discover, and crawl from there" - not "here's a machine-readable list of every listing." That's the next thing we're closing, not a hypothetical.

The more interesting bugs, honestly, weren't in the llms.txt work - they were in structured data we'd already shipped and hadn't re-checked. An MCP listing's rating lives in our database on a 0-10 scale; schema.org's AggregateRating.ratingValue defaults to a 0-5 scale. Nothing crashed. Nothing looked wrong on the page. It would have just quietly told every crawler pulling structured data that a 7.4-out-of-10 listing was actually 7.4-out-of-5 - a rating that isn't just wrong, it's impossible, and validators either reject it outright or silently drop the field. Structured data bugs like that don't throw errors. They just make your site slightly less legible to the systems increasingly doing the reading.

Prediction, since that's the more interesting question than "do you have an llms.txt": the sites that end up actually discoverable by agents won't be the ones with the prettiest llms.txt file. They'll be the ones whose structured data was correct before anyone was checking it with an LLM instead of a human eye.