llms.txt Explained: What It Is and Do You Need One in 2026?
What llms.txt is, how it differs from robots.txt, what Google and 2026 server-log data say about who reads it, whether your site needs one, and an example file you can adapt.
Quick answer: llms.txt is a proposed Markdown file placed at yoursite.com/llms.txt that gives AI tools a short summary of your site and links to its most useful pages. It is not an official standard. Google says it is not needed for its AI features, and 2026 server-log data shows most llms.txt files are never requested. It is cheap to add, but expect little direct benefit.
llms.txt has become one of the most discussed files in SEO since 2024, and many agencies now sell it as a quick win for AI visibility. The reality is more modest. It is a simple, sensible idea that has seen wide publishing but limited use by the AI systems it was designed for.
This guide explains what llms.txt is, who actually reads it, whether your site needs one and how to write one correctly, with an example file you can adapt.
Key takeaways
- llms.txt was proposed by Jeremy Howard in September 2024 as a Markdown index of a site’s key content for large language models.
- Only one element is required: an H1 with the site or project name. A summary blockquote and link lists are recommended.
- Google’s guidance says you do not need new machine-readable files or AI text files to appear in AI Overviews or AI Mode.
- An Ahrefs study of 137,210 domains found 97% of llms.txt files received zero requests in May 2026.
- It is most useful for documentation, developer tools and software products whose users work with AI coding assistants.
What is llms.txt?
llms.txt is a plain-text file written in Markdown. It sits at the root of a website, or at a subpath such as /docs/llms.txt, and lists the pages an AI model should read to understand the site. The proposal is published at llmstxt.org.
The reasoning given on llmstxt.org is that context windows, while larger than they were, are still too small for most websites in their entirety, and every wasted token costs time and money. A curated index with clean Markdown links helps a model skip navigation, scripts and adverts.
The same proposal suggests offering Markdown versions of pages by adding .md to the URL, so a model can fetch clean text instead of HTML.
How is llms.txt different from robots.txt and sitemap.xml?
| File | Purpose | Format | Status | Who reads it |
|---|---|---|---|---|
| robots.txt | Tells crawlers what they may or may not access | Plain-text directives | Long-established standard | Search and AI crawlers that respect it |
| sitemap.xml | Lists URLs for search engines to discover | XML | Widely supported standard | Search engines |
| llms.txt | Points AI models to the most useful content, with short descriptions | Markdown | Community proposal, not a formal standard | Some AI agents and developer tools, rarely by AI search crawlers |
llms.txt does not grant or block access. If you want to control AI crawlers, robots.txt is still the file that matters.
Who actually reads llms.txt?
What Google says
Google’s documentation on AI features states that you do not need to create new machine-readable files, AI text files or markup to appear in AI Overviews or AI Mode. In June 2025, Google’s John Mueller wrote that “no AI system currently uses llms.txt”, as reported by Search Engine Roundtable.
What server logs show
The most detailed public data comes from an Ahrefs study published in June 2026. It analysed 137,210 domains using Ahrefs Web Analytics and Bot Analytics data for May 2026. About 28% of those domains had an llms.txt file, and 97% of those files received no requests at all during the month.
Of the requests that did arrive, only 19.5% came from identified AI bots. SEO audit tools accounted for 21.7%. AI agents were the largest AI category, and Ahrefs noted that Claude Code made more requests than the major AI search retrievers, while PerplexityBot and OAI-SearchBot generated minimal traffic. Ahrefs concluded that, for AI search visibility, the cons outweigh the pros right now.
Where it is used
Adoption is strongest among documentation sites, where developers point AI coding assistants at product docs. Examples include Stripe’s llms.txt and Cloudflare’s developer documentation llms.txt, which links to a separate llms.txt for each product. Plugins have also made publishing easy: Yoast added a free llms.txt generator to its WordPress SEO plugin in 2025.
Do you need an llms.txt file?
For most D2C and B2B marketing sites, llms.txt is optional and low priority. It will not improve Google rankings, and current evidence does not show it improving citations in ChatGPT, Gemini or Perplexity.
It is worth adding when:
- You publish developer documentation, APIs or software that people use with AI coding tools.
- Your CMS or SEO plugin can generate and update it automatically at no cost.
- You want a single, controlled summary of your brand and key pages available for any agent that does look for it.
It is not worth adding if it takes time away from work that does affect AI visibility: crawler access in robots.txt, answer-first content, structured data, page speed and earning mentions on credible third-party sites.
How do you write an llms.txt file?
The format on llmstxt.org has a fixed order:
- An H1 with the name of the site or project. This is the only required element.
- A blockquote with a short summary containing the key information.
- Optional paragraphs or lists with more detail, but no further headings in this part.
- Zero or more H2 sections containing lists of Markdown links, each optionally followed by a colon and a short note.
- An optional H2 called “Optional” for secondary links that a tool can skip when it needs a shorter context.
Example llms.txt for an agency website
# Squarebase
> Squarebase is a 360-degree digital marketing agency, founded in 2017, offering SEO, performance marketing, brand building, social media management, website design and Shopify development for D2C and B2B brands.
Key facts:
- Services are delivered for clients in India and international markets.
- Contact and enquiries: https://squarebase.io/contact/
## Services
- [All services](https://squarebase.io/services/): Overview of every service Squarebase offers
- [Search engine optimisation](https://squarebase.io/search-engine-optimisation/): SEO, AEO and GEO programmes
- [Performance marketing](https://squarebase.io/performance-marketing/): Paid media on Google and Meta
- [Shopify development](https://squarebase.io/shopify-development/): Store builds, themes and migrations
- [Website design and development](https://squarebase.io/website-design-development-services/): Custom websites
## Support
- [Website maintenance](https://squarebase.io/website-maintenance-management-agency/): Ongoing updates, security and hosting support
## Optional
- [Brand building](https://squarebase.io/brand-building/): Positioning, identity and content
- [Social media management](https://squarebase.io/social-media-management/): Organic social content and communitySave the file as UTF-8 plain text, upload it to the root of your domain and check that it loads at https://yourdomain.com/llms.txt with a text/plain or text/markdown content type.
Writing tips
- Keep the summary factual. State what you do, for whom and where, without slogans.
- Link to canonical, indexable pages only. Do not include login pages, carts or filtered URLs.
- Write one short, specific note per link so a model knows when to use it.
- Keep it short. A curated list of 10 to 30 links is more useful than a full sitemap.
- Make sure every fact matches your website, Google Business Profile and other listings.
Treat it as public and keep it secure. Anything in llms.txt is visible to anyone. Do not list private URLs, staging sites or internal documents, and review automatically generated files before they go live. Ahrefs also recommends treating the file as controlled code rather than a marketing asset.
What are the common llms.txt mistakes?
- Treating it as a ranking factor. Selling or buying llms.txt as an SEO or GEO fix sets the wrong expectation. No major search engine has said it uses the file for ranking or citation.
- Using it to block AI crawlers. llms.txt has no access rules. Blocking or allowing bots still happens in robots.txt and at your CDN or firewall.
- Dumping the whole sitemap into it. The point of the file is curation. Hundreds of links without notes defeat the purpose.
- Letting it go stale. A file that links to removed pages or old prices gives AI tools wrong information about your brand.
- Breaking the format. Missing the H1, adding extra headings above the link sections or using HTML instead of Markdown makes the file harder for tools to parse.
llms.txt checklist
- Decide whether you need one, based on your audience and site type.
- Check whether your CMS or SEO plugin can generate it automatically.
- Include an H1, a factual blockquote summary and grouped link lists.
- Use an “Optional” section for secondary pages.
- Confirm it loads at /llms.txt and contains only public, canonical URLs.
- Review it each quarter or when key pages change.
- Check server logs to see whether any bots actually request it.
How Squarebase can help
Squarebase reviews llms.txt alongside the work that has a proven effect on AI visibility: robots.txt, structured data, content structure and off-site mentions. See our SEO services or ask about website maintenance and management to keep these files accurate over time.
Frequently asked questions
What is llms.txt?
llms.txt is a proposed Markdown file placed at the root of a website that summarises the site and links to its most useful pages for large language models. It was proposed by Jeremy Howard in September 2024 and is published at llmstxt.org. It is a community proposal, not an official web standard.
Does llms.txt help SEO or Google AI Overviews?
No. Google’s documentation says you do not need new machine-readable files or AI text files to appear in AI Overviews or AI Mode. Standard SEO best practices apply. llms.txt has no known effect on Google rankings or on inclusion in Google’s AI features.
Do ChatGPT or Perplexity read llms.txt?
Rarely, based on current evidence. An Ahrefs study of 137,210 domains found 97% of llms.txt files received no requests in May 2026, and PerplexityBot and OAI-SearchBot generated minimal traffic to these files. AI agents and coding tools requested them more often than AI search crawlers.
Is llms.txt the same as robots.txt?
No. robots.txt tells crawlers which parts of a site they may access and is a long-established standard. llms.txt does not allow or block anything. It only offers a curated list of useful pages in Markdown for AI tools that choose to read it.
Should a D2C or B2B brand create an llms.txt file?
It is optional. If your CMS or SEO plugin can generate it for free, adding one does little harm. Do not let it replace work with proven impact, such as allowing AI search crawlers, writing answer-first content, adding structured data and earning credible third-party mentions.
Sources
- llmstxt.org: The /llms.txt file (proposal by Jeremy Howard)
- Google Search Central: AI features and your website
- Search Engine Roundtable: Google says no AI system currently uses llms.txt (June 2025)
- Ahrefs: We analysed 137K sites, 97% of llms.txt files never get read (June 2026)
- Yoast: Yoast launches llms.txt generator in its SEO plugin (2025)
- Stripe documentation llms.txt
- Cloudflare developer documentation llms.txt
