llms.txt Explained: Does It Actually Do Anything?
Explains llms.txt as a content index, when to use it, how to publish and measure, and why crawlability and content quality come first.
GTM Engineer at AskWatch.ai
6 min read
On this page
I’d use llms.txt as a content index - not a way to get more AI mentions or citations. Publish it at /llms.txt if it helps agents find your docs and takes little work to maintain. It doesn’t control crawler access, and a file request doesn’t prove it changed an AI answer.
Here’s how I’d decide whether it’s worth your time:
- Check the use case: Developer docs and API references are better fits than small sites or stores with changing inventory.
- Keep setup simple: Link to key public pages, check formatting and access, and assign someone to keep it current.
- Measure results separately: Track file requests, brand mentions, and citations before and after publication.
One sample cited below recorded 7 GPTBot requests for llms.txt versus 3,990 for robots.txt across 83 sites over 12 weeks. That measures access, not results. My takeaway: <u>fix crawlability and content first</u>, then test whether this extra index earns its upkeep.
LLMs.txt - What It Is and Do You Need One?
Why AI Visibility Gains Are Still Unproven
llms.txt is a community proposal, not a formal web standard. Engines have no obligation to support it.
What Major AI Platforms Have Said
Google’s Search Central guidance says Google Search and its generative AI features don’t use or require new machine-readable files such as llms.txt. This includes AI Overviews and AI Mode. Publishing the file neither helps nor hurts visibility in those features.
OpenAI, Anthropic, and Perplexity have made no public statement confirming support for third-party llms.txt files. Platform support matters more than file syntax. A correctly formatted file doesn’t mean a platform will use it.
Why File Requests Do Not Prove Results
Requests show access, not impact.
A July 2026 sample of 83 sites over 12 weeks found GPTBot requested robots.txt 3,990 times and llms.txt 7 times.
A server log can show that a file was accessed, but it doesn’t prove the file was used. Verify crawler identity before interpreting traffic, then track visibility through repeated prompt tests before and after publication. An increase alone doesn’t establish causation.
That leaves a practical question: Is the file worth maintaining?
When llms.txt Is Worth the Upkeep
Treat llms.txt as a navigation aid, not a visibility signal. Address content quality and crawlability first. The file cannot fix broken pages or replace earned citations. Its upkeep is worth the effort only if it saves time finding content.
Here’s the tradeoff:
| Advantages | Limitations |
|---|---|
| Content curation: Points to key pages and explains what they cover. | Uneven adoption: Adoption varies, and models don’t have to follow the file. |
| Agent navigation: Gives agents a shorter route to relevant documentation. | No confirmed ranking benefit: Easier navigation doesn’t establish better search visibility. |
| Simple setup: Lightweight Markdown can be generated automatically. | Regular upkeep: Links and descriptions must match published pages. |
Good Fits: Documentation and Reference Pages
SaaS documentation, developer portals, and product knowledge bases are better fits. Large technical libraries can be hard to scan, and their clear, stable hierarchies make an index useful.
A useful index points to setup guides, API documentation, and product explanations, each with an accurate link and a short description.
Give one named person responsibility for maintenance. Update the index whenever linked pages move or product details change. Where possible, generate it from the documentation system rather than maintaining a separate copy by hand. Smaller sites and sites that change often have a harder time justifying that work.
Lower Priority: Small Sites and Changing Inventory
For small marketing sites, upkeep usually costs more effort than it’s worth. Ecommerce stores face a different problem: inventory changes can soon leave curated product links out of date.
Fix broken pages, stale inventory, and weak navigation first. For stores, crawlability and structured data matter more than another product index.
If creating the file takes almost no effort, it can be a low-cost addition. Otherwise, its upkeep needs a practical navigation use case - not an assumed visibility payoff.
How to Publish, Check, and Measure llms.txt
The goal isn’t to prove llms.txt works. It’s to test whether it changes anything. Treat publishing as a test: publish the file, verify access, then measure results.
Create a Short Content Index
Use AskWatch’s llms.txt Generator to create a short, factual index of 15-30 high-value URLs based on your sitemap and page summaries. Publish it at /llms.txt. Check that it returns HTTP 200, and leave out pages that need JavaScript to display their main content.
Check File Quality and Crawler Access
Before judging results, confirm that bots can fetch the file.
Run AskWatch’s free llms.txt Checker to check formatting, links, and robots.txt access. Review any remaining URLs manually. Passing these checks confirms technical quality - not engine adoption or citation gains.
Check server logs for GET /llms.txt and, if you’ve published it, GET /llms-full.txt. Verify bots through IP ranges or reverse DNS; user-agent strings can be spoofed. Separate fetch bots from training crawlers, coding agents, and SEO tools. Investigate 403 or 429 responses that could block access.
Measure Mentions and Citations Separately
Once you’ve confirmed access, track changes in mentions and citations. File requests show access; mentions and citations show impact.
Before publishing, record a baseline with AskWatch’s visibility trackers. Note the publication date and any other site changes. Repeat the same target prompts over time, and track file requests, brand mentions, and citations separately. Treat any change as directional until it repeats.
Conclusion: Use llms.txt to Organize Content
Publish llms.txt when its use as a content index is worth the upkeep. Developer docs and API references are the best fits. Auto-generation keeps the work light when your CMS or docs platform maintains the file.
Otherwise, focus on crawlable pages, clear answers, and authoritative citations. Track brand mentions and citations, not just whether the file exists.
Treat llms.txt as a way to organize content - not as proof of better AI visibility.
FAQs
Do I need both llms.txt and llms-full.txt?
No. llms.txt is a short, link-based index for AI agents. llms-full.txt is an unofficial companion that puts your pages’ full text into one document, so an agent can retrieve it in a single fetch.
Consider llms-full.txt only for developer platforms or documentation hubs where fewer fetches matter. For most marketing or services sites, llms.txt is enough - if you use one at all. AskWatch offers a free llms.txt generator and checker.
Can llms.txt expose sensitive content?
Publishing an llms.txt file can expose sensitive content or make the file a target for prompt injection. It may also draw attention to information you didn’t mean to highlight.
Stick to plain, verified facts and links you control. Never include internal data or instructions aimed at AI models. Use AskWatch’s free llms.txt generator and llms.txt checker to create your file and check its structure.
Can I automate updates with AskWatch’s free llms.txt tools?
Not directly. AskWatch’s free llms.txt generator creates a static file from your sitemap and page descriptions. It doesn’t automatically sync with your site.
When your content or site structure changes, regenerate the file manually or add generation to your site’s data pipeline to keep it accurate. After updates, use AskWatch’s free llms.txt checker to validate the file’s structure.
