How to Write an llms.txt File: Making Your Site Readable to AI Crawlers

You’ve probably seen the recommendation by now: add an llms.txt file to your site so AI models can “understand” it better. It’s become a common suggestion in SEO circles, repeated often enough that it can feel like a settled best practice.

Here’s the honest answer before the detail: an llms.txt file is a plain-text Markdown file, placed at the root of your domain, that summarizes your site and links to your most important pages in a format language models can parse quickly. It’s cheap to create, it can’t hurt, and it might help a language model orient itself once it’s already looking at your site — but there’s no confirmed mechanism by which it gets you crawled, indexed, or cited more often. On the surface it looks like robots.txt or sitemap.xml — plain text, sitting at your domain root — but it serves a different function. It is not a ranking lever.

That’s the whole bet. What follows is how to write one properly, where it fits next to the other files already doing real work on your site, and what the current evidence actually shows — not what the hype cycle implies.

What Is llms.txt, and Where Did It Come From?

llms.txt is a proposed standard: a Markdown file hosted at yourdomain.com/llms.txt that gives large language models a condensed, structured overview of a website — what it is, what it offers, and which pages matter most. The idea was introduced by Jeremy Howard, co-founder of answer.ai, in September 2024.

The reasoning behind it is straightforward. Language models have limited context windows, and full HTML pages are noisy — full of navigation menus, ads, scripts, and boilerplate that eat up tokens without adding meaning. A clean Markdown summary, the thinking goes, lets a model get the gist of a site fast, without wading through markup it doesn’t need. It’s the same instinct behind robots.txt in 1994: give automated visitors a predictable, machine-readable file at a known location, rather than making them guess.

The comparison to robots.txt is useful, but only up to a point, and it’s worth being precise about where it breaks down — which is the next section.

llms.txt File Format and Structure

The proposed format is deliberately simple. It’s a Markdown file, not JSON or XML, which means it’s readable by a human glancing at it in a browser and parseable by a model without special tooling. The recommended structure looks like this:

1. H1 title. The first line is a single # heading with your project or site name.

2. A blockquote summary. Directly below the title, a short blockquote (using >) gives a one- or two-sentence description of what the site or product is.

3. Optional context paragraphs. Additional short paragraphs can add detail — audience, scope, key facts — without turning into marketing copy.

4. Linked sections with Markdown lists. The bulk of the file is a series of ## headings (like “Docs,” “Guides,” “API Reference”) each followed by a Markdown-formatted list of links, where each link includes the page title, the URL, and an optional short description.

A minimal example might look like this:

# Acme Analytics

> Acme Analytics is a web analytics tool for small e-commerce teams.

Acme focuses on privacy-first tracking without cookie banners.

## Docs

- [Getting Started](https://acmeanalytics.com/docs/start): Install the tracking script
- [Dashboard Guide](https://acmeanalytics.com/docs/dashboard): Reading your traffic data

## Company

- [About](https://acmeanalytics.com/about): Team and mission
- [Pricing](https://acmeanalytics.com/pricing): Plans and billing

There’s also a companion file some sites publish alongside it: llms-full.txt, which contains the full text content of key pages rather than just links to them, intended for cases where a model needs the complete material in one fetch rather than a summary with pointers.

Two things are worth noting honestly. First, there’s no enforcement mechanism — nothing checks that your llms.txt is accurate, and no crawler is obligated to read it at all. Second, keep it genuinely short and genuinely current. A stale or bloated llms.txt is worse than none, because it either wastes the small amount of attention a model gives it or actively misleads whatever does read it.

How to Create an llms.txt File for Your Site

You don’t need special software to write one — it’s a text file, and you can create it in any editor. But there are a few practical paths depending on the size of your site.

Manually, for small sites. If you have a handful of core pages, write the file by hand following the structure above. Prioritize your most important pages — documentation, pricing, core product pages, key guides — rather than listing everything. Save it as llms.txt and upload it to the root of your domain, the same location as robots.txt.

With a generator tool, for larger sites. For sites with too many pages to catalog by hand, tools exist that crawl a website and generate an llms.txt automatically. Firecrawl’s llmstxt.firecrawl.dev is one such tool: you point it at a domain, it crawls the site’s structure, and it outputs a generated llms.txt (and can produce an llms-full.txt) via a hosted tool or API. This is useful for sites with deep documentation or large content libraries where manual curation isn’t realistic.

Validate before you publish. However you generate it, check that every link actually resolves, that the summary reflects what the site currently does, and that you haven’t buried the handful of pages that matter under a long undifferentiated list. A good test: if a person unfamiliar with your site read only the llms.txt, would they understand what you do and where to go next? If not, trim it and rewrite the summary.

Once it’s live, add it to whatever process you use to catch broken links or stale documentation, since nothing will alert you if it goes out of date.

llms.txt vs. robots.txt vs. sitemap.xml

This is where confusion causes real mistakes, because the three files look similar — plain text, root of domain — but do completely different jobs.

robots.txt is a directive file. It uses Allow and Disallow rules to tell crawlers which parts of a site they may or may not access, and it’s been part of the web since long before AI crawlers existed. It’s about permission and crawl control, not explanation.

sitemap.xml is a discovery file. It lists every indexable URL on a site — sometimes with metadata like last-modified date — so search engines can find pages efficiently, especially ones that are hard to reach through internal links alone. It’s exhaustive by design.

llms.txt is a curated summary. It doesn’t list every URL, and it carries no permission or directive weight at all — a crawler can’t be blocked or allowed by it, because that’s not what it does. It’s editorial: a hand-picked (or generated) shortlist of what matters most, written for something trying to understand the site quickly rather than index it completely.

File	Purpose	Audience	Governs access?
robots.txt	Crawl permissions (Allow/Disallow)	All crawlers	Yes — it’s a directive
sitemap.xml	Exhaustive URL discovery	Search engine crawlers	No — informational only
llms.txt	Curated summary + key links	Language models	No — informational, and voluntary to even read

The practical implication: llms.txt doesn’t replace either of the other two, and it doesn’t do their job worse or better — it does a different job entirely. You still need a correct robots.txt and an accurate sitemap.xml regardless of whether you add an llms.txt. Think of it less as infrastructure and more as a note left on the counter — worth writing if you have five minutes, not something to lose sleep over.

Does llms.txt Actually Work? The Adoption Evidence

This is the part that separates a useful answer from a hyped one, and it’s the least comfortable to write because it doesn’t resolve cleanly.

The core problem: no major AI company has publicly committed to crawling, parsing, or prioritizing llms.txt files as part of how their models retrieve or cite web content. There is no confirmed mechanism connecting “having an llms.txt” to “being cited more by an AI model.” What evidence exists for its effectiveness is anecdotal rather than demonstrated at scale, and the honest assessment is that this is an unsettled question — the file may be a low-cost addition worth having, or it may do nothing measurable at all. Nobody has shown otherwise yet.

Why the uncertainty persists is worth understanding rather than just accepting. No model provider has confirmed that llms.txt is a signal it actually uses. That doesn’t mean it’s definitely inert. It means nobody outside the model providers can currently confirm it’s load-bearing, and the honest position is to treat it as unproven rather than either dismiss it or oversell it.

So should you build one? Given that it costs little more than an hour of focused writing and cannot plausibly hurt anything, a reasonable position is: yes, as a low-priority, low-cost addition — not as a strategy, and not instead of the things with clearer evidence behind them. If you’re choosing between spending an afternoon on llms.txt versus spending it on structured data, clean page markup, or fixing a blocked crawler, the latter three should win every time.

What Actually Has More Evidence Behind It

If llms.txt is a contested bet, it’s worth being clear about what isn’t contested — the groundwork that every credible AEO/GEO discussion agrees matters.

Audit robots.txt for AI crawler access. Before writing anything new, check what you might already be blocking. Many robots.txt files, especially older ones, were written before AI crawlers existed and may inadvertently disallow bots like GPTBot, ClaudeBot, or PerplexityBot through broad rules aimed at something else. If a crawler can’t reach your content, no amount of AI-specific tooling elsewhere will matter. Review your Disallow rules line by line and confirm you’re not blocking the crawlers you actually want visiting, while being mindful of crawl budget if you run a very large site.

Consider content-usage signals beyond robots.txt. Whether and how a site should signal usage rights or permissions to AI systems is a separate question from what llms.txt does — llms.txt is a curated summary for AI tools, not a permissions mechanism.

Serve clean, structured content. Some sites offer a Markdown version of a page alongside the HTML version, or use HTTP content negotiation via the Accept header so a requesting client can receive a .md version directly. The underlying principle — strip the noise, keep the substance — is the same instinct behind llms.txt, just applied at the page level instead of a single root file.

Use structured data. Schema.org markup in JSON-LD format — types like Article, FAQPage, Organization, or Product — gives machines explicit, unambiguous labels for what’s on a page, rather than making them infer it from prose. It’s a long-standing structured-data vocabulary that’s been used for machine-readable labeling in search for years.

Get the HTML semantics right. Proper heading structure, semantic HTML, a working sitemap, an RSS feed where relevant, and correct canonical URLs form the discovery layer underneath everything else. If these are broken, an llms.txt file is decoration on a shaky foundation.

Taken together, this is the actual landscape of what’s sometimes called Generative Engine Optimization or Answer Engine Optimization — the practice of shaping a site to be understood and cited by AI Overviews, Perplexity, and retrieval-augmented systems generally, of which llms.txt is one small, unproven tactic among several with firmer footing.

How to Tell If Any of This Is Working

Two practical checks help:

Server log analysis. Review your raw server or CDN logs for requests from known AI crawler user agents to see whether they’re visiting at all, how often, and which pages they fetch. This tells you about crawl activity — not citations — but it’s the closest thing to ground truth available.

Track referral patterns and mentions. Watch your analytics for referral traffic patterns from AI platforms, and periodically check whether your content is being cited or summarized in AI answers for relevant queries. This is slower and less precise than traditional rank tracking, but it’s the current state of the art for measuring AI visibility.

Neither of these will tell you definitively whether your llms.txt file specifically moved the needle — there’s no clean attribution path for that yet. What they will tell you is whether your site is being crawled and cited at all, which is the more important question regardless.

The Bottom Line

Write an llms.txt file if you have a spare hour: a clear H1, a short honest summary, and a curated list of the pages that actually matter, kept current. Don’t confuse it with robots.txt or sitemap.xml, which do real access-control and discovery work that llms.txt doesn’t touch. And don’t mistake adding it for a strategy — the evidence doesn’t support that yet, and the things that do have evidence behind them (clean crawler access, structured data, solid HTML semantics) deserve the bulk of your attention first.

FAQ

Do I need an llms.txt file for my site? No AI provider requires or has committed to using it, so nothing breaks if you skip it. It’s a low-cost, optional addition — reasonable to add, not urgent to add.

Where does llms.txt go on my site? At the root of your domain, the same location as robots.txt — for example, yourdomain.com/llms.txt.

What’s the difference between llms.txt and llms-full.txt? llms.txt is a short summary with links out to key pages; llms-full.txt contains the fuller text content of those pages directly in the file, for cases where a complete fetch in one place is more useful than a set of pointers.

Will llms.txt improve my rankings in Google or my visibility in AI Overviews? There’s no established mechanism connecting llms.txt to ranking or citation outcomes, and no major crawler has publicly confirmed it uses the file that way. Treat any claim otherwise as unproven.

Can I generate an llms.txt automatically instead of writing it by hand? Yes — tools exist that crawl a site and generate a summary file for you, which is more practical than manual writing once a site has more than a handful of key pages.

Does llms.txt block or allow AI crawlers like robots.txt does? No. It carries no directive weight — it can’t permit or restrict access. Crawler permissions are still controlled entirely through robots.txt.