llms.txt for AI agents

What this check does

GETs https://yourdomain.com/llms.txt and confirms it returns a 200 with a parseable file. The format is markdown with a specific structure (H1 site title, blockquote summary, sections of links).

Why it matters

llms.txt is the AI-era counterpart to robots.txt and sitemap.xml. Where those tell crawlers what they’re allowed to access, llms.txt tells LLM agents what’s worth reading — a curated index of your most important pages.

The convention is still new (proposed by Answer.AI in late 2024) and not yet universally honored. But it costs nothing to publish, and several agentic systems (Cursor, Continue, custom GPTs) already lean on it as a high-trust entry point. Sites that publish a good llms.txt get cited in LLM answers at higher rates than sites that don’t.

How to fix it

Create /public/llms.txt (or wherever your server hosts static root files) with this structure:

# MetricSpot

> Free SEO and AI-readability audits with white-label PDF reports for agencies. 91 checks across technical, on-page, performance, AI, accessibility, and privacy.

## Docs

- [HTTPS on your site](https://metricspot.com/docs/https-on-your-site/): Why HTTPS matters and how to enable it.
- [Largest Contentful Paint](https://metricspot.com/docs/largest-contentful-paint/): The first Core Web Vital, threshold and fixes.

## Pricing

- [Plans and pricing](https://metricspot.com/pricing/): Free tier, Premium, Pro.

## API

- [Public audit API](https://app.metricspot.com/api/public/pricing): Programmatic access.

Rules:

• H1 = your site / product name.
• Blockquote immediately after = one-sentence elevator pitch.
• H2 sections group related links.
• Each link is [Title](URL): one-sentence summary. — the summary is what the LLM reads when deciding whether to fetch.

Submit your file to the llmstxt.org directory once published.

Frequently asked questions

Do any LLMs actually read this today?

A growing list. Anthropic’s Claude (via the docs MCP), Cursor’s codebase agent, and several custom GPTs have built-in llms.txt awareness. ChatGPT search and Perplexity don’t yet officially honor it but the file is part of their crawled corpus.

How is this different from sitemap.xml?

Sitemap is machine-readable XML — every URL, equal weight. llms.txt is curated markdown with summaries — only your best URLs, ranked. Sitemap is for completeness; llms.txt is for editorial judgment.

What about /llms-full.txt?

A complementary convention: the same structure but with full page content inlined, so an agent can fetch one file and get all your docs at once. Useful for documentation sites; overkill for marketing sites.