Should You Use llms.txt? The Robots.txt for Large Language Models

•

by AEO Checker Team

•

5 min read

The llms.txt file is a newly proposed standard that aims to guide large language models (LLMs) when they crawl your website. In this article, we explain what llms.txt is, who created the proposal and when, what kind of content or rules go inside it, and whether implementing it is worthwhile today.

Origin of the llms.txt Proposal

The idea of llms.txt emerged in late 2024 as AI chatbots began pulling information from websites in real time. The concept was first proposed by Jeremy Howard (co-founder of Answer.ai and fast.ai), who published an initial spec on September 3, 2024. Howard and collaborators envisioned llms.txt as a way to provide LLMs with a curated "map" of a site's important content for inference, similar in spirit to how robots.txt guides search engine crawlers. Early momentum came in mid-November 2024, when developer documentation platforms like Mintlify started auto-generating llms.txt files for thousands of sites. Howard described the goal succinctly: "Site owners should decide what an LLM reads, no more random scraping", emphasizing that site authors want some control or guidance over what AI models see.

Notably, the llms.txt proposal coincided with discussions about a related file, llm.txt (without the "s"). What's the difference? The singular llm.txt has been suggested as a "permissions" file to declare rules for AI crawlers (similar to robots.txt directives like allow, disallow, no-index or no-train for AI). In contrast, llms.txt (plural) is a content manifest – a markdown-formatted roadmap of your site's key pages for LLMs to read at inference time. Both ideas were introduced together, but llms.txt (the manifest) is our focus here as it's more often likened to a "treasure map" for AI rather than a strict crawler rule file.

Key timeline:

  • Nov 2024: Jeremy Howard publishes specs for both llm.txt and llms.txt. Companies like Mintlify and Anthropic begin supporting automatic generation of these files.
  • Dec 2024: Community-run directories (e.g. llmstxt.site) start cataloguing sites that have adopted llms.txt.
  • Mar 2025: AI developer tools (LangChain, etc.) publish their own llms.txt and llms-full.txt for documentation, and some IDEs let developers feed these manifests into LLM assistants.
  • Apr 2025: Google's John Mueller weighs in, comparing llms.txt to the deprecated <meta name="keywords" /> tag, implying it currently has "zero ranking impact" and limited usefulness.

What Exactly Is llms.txt?

In essence, llms.txt is a special text file (formatted in Markdown) placed at the root of your site (e.g. https://yourdomain.com/llms.txt) that summarizes and highlights your most important content for AI. Think of it as a cheat sheet for LLMs. While a normal human visitor navigates via your menus and pages, an LLM with limited context length would benefit from a concise guide pointing to the key pages or sections of your site.

"Imagine walking into a large store and looking for socks. Instead of wandering every aisle, you get a store map highlighting the sock section and other key departments. You don't have to use the map, but it makes finding what you need much easier." This is how Yoast describes the role of llms.txt for AI models on a website.

Unlike robots.txt which tells bots what not to access, llms.txt suggests what AI should read. It's about guidance and curation rather than exclusion. Notably, llms.txt is not intended for training (the proposal assumes models are mostly trained already) but for real-time retrieval (inference) when an AI agent is answering questions and needs to quickly fetch relevant info from your site.

Syntax and Format Examples

The llms.txt file uses a strict Markdown structure that is both human-readable and machine-parseable.A valid llms.txt typically contains:

  • H1 title: The name of your site or project (first line, beginning with #).
  • Short description: A one-paragraph summary, often in a Markdown blockquote (> ), highlighting the site's purpose or key info.
  • Details (optional): Additional context or instructions in plain text or bullet points, but no further headings at this point (to avoid breaking the structure).
  • Section headings (H2): One or more ## headings that categorize important links (for example, "## Docs", "## Products", "## Support").
  • Link lists under each section: Under each H2, a markdown list of important pages. Each list item is a hyperlink with an optional brief description. For example: "Pricing: Latest pricing plans | API Guide: Reference for developers". These should point to content optimized for LLM consumption, ideally Markdown or plain text versions of pages (notice the ".md" extension in the
  • Optional section: A special optional section can be included at the end (as an H2 titled "Optional"). Links listed under this section are considered lower priority or supplementary; an AI agent might skip these if short on context space.

For instance, a minimal llms.txt might look like this:

# MySite

> A one-line description of MySite's purpose or content focus.

Extra details or context can be provided here in plain text.

## Documentation
- [Getting Started](https://mysite.com/docs/getting-started.md): Quick setup guide
- [API Reference](https://mysite.com/docs/api.md): Full API docs for developers

## Optional
- [About Us](https://mysite.com/about.md)

This example shows the basic structure: title, summary, one section with two key links, and an optional section with a less-critical page. The use of Markdown (e.g. - Link: description) means an LLM can parse it easily to find link titles and URLs, while a developer could also use simple scripts or regex to extract the info. Allow/Disallow Rules: Despite the "robots.txt for LLMs" nickname, llms.txt does not use crawl directives like "Disallow" or "User-agent", that's actually the realm of the proposed llm.txt (singular) permissions file. If your goal is to forbid AI bots from crawling or training on your content, you'd use robots.txt or potentially llm.txt with lines like

User-agent: GPTBot 
Disallow: /private/ 
NoTrain: /premium/

llms.txt, on the other hand, assumes the AI is allowed and is trying to help it find the most relevant content quickly, free of navigation noise and ads. In fact, Google's John Mueller clarified that "LLMs.txt is not a way to control AI bots... it's a way to show the main content to AI bots"

Current Adoption and Industry Support

As of mid-2025, llms.txt is still far from a widely adopted or officially recognized standard. No major AI services (OpenAI, Anthropic, Google/Bard) have publicly announced support for crawling or honoring llms.txt files.

Anecdotally, site owners who have implemented it report that they do not see common AI user-agents fetching llms.txt in their server logs. This suggests that today's prominent chatbots and LLM-based search engines are largely ignoring the file. Mueller's comparison to the obsolete meta keywords tag was a gentle way of saying it has little to no effect on your search rankings or AI visibility at the moment. That said, there is a small but growing movement experimenting with llms.txt especially in tech circles:

  • Documentation sites & dev tools: Many developer-focused websites (APIs, libraries) have added llms.txt to help AI assistants answer technical questions. For example, LangChain, a popular AI framework, published an llms.txt and an extended llms-full.txt for its docs in March 2025. Tools like Cursor IDE allow importing these manifests to assist coding with AI.
  • Content management & SEO tools: Notably, Yoast SEO, a major SEO plugin for WordPress, introduced an automatic llms.txt generator in June 2025.Website owners can opt-in, and Yoast will compile a weekly-updated llms.txt highlighting recently updated posts, important pages (based on the sitemap and site structure), and any custom description the owner adds. This one-click solution indicates that SEO tooling companies see potential value in llms.txt for the future of content discovery.
  • Enterprise interest: A handful of AI-forward enterprises and SaaS companies have started hosting both llm.txt and llms.txt at their domain root as a proactive measure. These early adopters view it as an investment in future visibility: if AI agents begin looking for these files, they'll be ahead of the curve.

SEO news outlets and blogs have certainly taken notice. Search Engine Land dubbed llms.txt a "proposed standard for AI website content crawling" and explained how it works, while Search Engine Journal reported on Mueller's skepticism, essentially cautioning that "none of the AI services... even check for it" at present. The SEO community is split: some experts feel it's a clever way to increase the chances of being cited by AI answers, while others call it "mostly hype" until the big players support it.

Should You Implement llms.txt Now?

Given the current state, is it worth creating an llms.txt file for your website? The answer depends on your goals and resources. Here some pros and cons to consider:

Potential Benefits:

  • Future-proofing for AI Search : If LLM-based search engines (like Bing Chat, Google's AI overviews, or others) start actively using llms.txt, having one in place could give your site a head start. It's akin to preparing your content for a new discovery channel early.
  • Improved AI Responses: A well-crafted llms.txt might help AI agents retrieve more accurate information about your site, possibly leading to better summaries or more frequent citations of your content in AI-generated answers. For example, by pointing ChatGPT to your "Pricing" or "FAQ" pages directly, it may answer user questions about your business with fewer errors.
  • Internal Clarity and Cleanup: The process of building an llms.txt can be a useful content audit. It forces you to identify your most important pages and ensure there are clean, text-based versions. Some companies treat it as creating a "quick reference guide" to their own content, which can also benefit human team members or aid in content maintenance.

Drawbacks / Limitations:

  • No Guaranteed Impact (Yet): As of 2025, there's no evidence llms.txt provides any SEO boost or AI traffic on its own. You might implement it and see zero immediate change as some have already reported. If Google's stance is that it's like meta keywords, it could remain ignored indefinitely.
  • Maintenance Overhead: You'll need to keep the file updated as your content changes. Outdated links or missing new pages could mislead future AI crawlers. This adds a layer to your content workflow (though tools and plugins are emerging to automate it).
  • No Enforcement: Unlike robots.txt, llms.txt is purely advisory. AI bots may ignore it completely or only partially follow it. There's also the risk of divergence such as if someone manually curates llms.txt to paint a rosier picture than the actual site (a form of cloaking), it could breed mistrust or be seen as spammy if discovered..

Google's View: Google's Search Advocate John Mueller commented in April 2025, "AFAIK none of the AI services have said they're using LLMs.TXT... To me, it's comparable to the keywords meta tag – this is what a site-owner claims their site is about... (Is the site really like that? well, you can check it. At that point, why not just check the site directly?)".

This encapsulates the skepticism: if an AI has to verify your llms.txt info against your actual pages to trust it, the file might be superfluous. On the other hand, supporters argue that llms.txt serves a different purpose than old meta tags. Rather than telling the AI "what we are about" in hopes of ranking, it feeds the AI actual content slices that can be directly used in responses. Think of it less as an SEO meta signal and more as a fast lane to your best answers. As one proponent put it, "LLMs.txt curates your site's best AI-digestible content for inference", like giving the AI a quick menu of your site.

Recommendation

If you're an early adopter or have a content-heavy site, there's little harm in setting up an llms.txt. It is a low-cost experiment that could pay off if AI platforms start respecting it. But, remember that the fundamentals of content quality and technical SEO still outweigh any new protocol. In fact, Google has implied that if your pages themselves are excellent and easily parsable, an AI should "just check the site directly" rather than needing a summary file. Ensure your site isn't blocking legitimate AI user-agents in robots.txt (unless you intend to) and maybe include an AI-friendly FAQ page. Keep an eye on announcements from AI providers, and use llms.txt as a complement, not a replacement.

Our sources