llms.txt Explained: The Complete Guide + Free Generator

The llms.txt specification, exactly

llms.txt (see the proposal at llmstxt.org) is a Markdown file served at /llms.txt. The format is simple and specific:

1. An optional byte-order mark.
2. An H1 heading, the only required section. This is your site or project name, written as a single Markdown H1 line (# Your Site Name).
3. An optional blockquote directly after the H1: a one-line summary of what the site is.
4. Optional free-text body content.
5. Optional H2 sections, each containing a bullet list of Markdown links in the form - [page title](url): short note, grouping your key pages (docs, pricing, API reference, etc).

That's the entire spec. It's intentionally minimal: the H1 is the only hard requirement, everything else is there to help an AI system navigate your site faster than parsing full HTML. Our AI Visibility Checker checks for exactly this: does the file exist, and does it start with a valid H1.

Full reference: every AI crawler this tool checks

Most llms.txt and AI-crawler guides list three bots. Here are all 16 this tool checks, individually, against your robots.txt file, with what each one is used for and whether it's documented to respect a Disallow rule.

The robots.txt block to allow all of them

If you want to explicitly allow every AI crawler in the table above, add this to your robots.txt (adjust if you want to keep specific ones out, for example if you don't want your content used for a particular company's model training):

User-agent: GPTBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: CCBot
Allow: /

User-agent: Meta-ExternalAgent
Allow: /

User-agent: Amazonbot
Allow: /

Run the AI Visibility Checker afterward to confirm the change took effect and see which, if any, bots are still blocked by a broader wildcard rule elsewhere in your file.

GPTBot and OpenAI's crawlers specifically

Because it's the single most-searched AI crawler by name, a quick note on GPTBot specifically: it's OpenAI's crawler for gathering training data for future GPT models, separate from OAI-SearchBot (which indexes pages for ChatGPT's search feature) and ChatGPT-User (which fetches a page live, on demand, when a ChatGPT user clicks through a citation). All three are documented by OpenAI to respect robots.txt, and all three can be allowed or blocked independently, since they're separate user-agent strings, not one combined bot. See OpenAI's own documentation for the current, authoritative list.

For the broader technical foundation these crawlers depend on (indexability, schema, sitemap), see our GEO guide, or check your own site directly on the homepage.

Generate your own llms.txt

Fill in your site name and a one-line summary below to generate a spec-valid llms.txt file you can copy and publish at your site root.

Common llms.txt mistakes

The most common mistake is publishing the file without the required H1 line, for example starting directly with a blockquote or a paragraph. Per the spec, the H1 is the only mandatory section, and a file missing it is treated as invalid by this checker (a yellow, not red, signal, since the file at least exists). Other frequent issues: linking to pages that return a 404, listing so many links that the file loses its 'concise summary' purpose, and forgetting to update it after a major site restructure, since nothing will automatically flag a stale llms.txt the way a broken sitemap link sometimes does.