Every few years a new file appears at the root of a website and the industry treats it as a ranking lever. llms.txt is the current example. It is a useful convention, but most of the advice around it is either mystical or wrong.

This guide covers what the file is, what it can realistically do, how to structure one, and how to ship it in an Astro site without creating another stale artifact nobody maintains.

What llms.txt actually is

llms.txt is a proposed convention for giving language models a concise, curated map of a website. The idea was introduced in 2024: instead of asking a model to parse an entire site, provide a markdown file at /llms.txt that explains what the site is and links to the pages that matter.

A good file looks like documentation. It has a title, a one-paragraph summary, and a short list of sections with links and descriptions. It is not a sitemap dump, and it is not a sales page.

There is also an optional llms-full.txt variant, which contains the full text of the pages rather than links. We rarely recommend it for marketing sites: it duplicates content, goes stale immediately, and creates a second source of truth that nobody owns.

What it does not do

Let us be direct, because this is where most of the confusion lives.

  • It is not a ranking factor. No engine has confirmed that llms.txt affects citation selection.
  • It does not replace robots.txt. Crawler permissions are still controlled by robots directives and server responses.
  • It does not replace sitemap.xml. Sitemaps are for discovery and freshness; llms.txt is for context.
  • It is not guaranteed to be fetched. Adoption is uneven. Treat it as a low-cost clarity layer, not a switch.
  • It cannot fix a site that is not retrievable. If pages require JavaScript to render, block crawlers, or return errors, the file will not rescue them.

The honest value proposition is this: it reduces ambiguity for any system that chooses to read it, and it forces your team to agree on which pages are canonical. That second benefit is worth the effort even if no model ever fetches the file.

Anatomy of a useful file

The format is plain markdown. Five elements do most of the work:

  1. An H1 with the entity name. Match the name you use everywhere else.
  2. A blockquote summary. One short paragraph: what the business does, who it serves, and where it operates. This is the passage a model is most likely to quote.
  3. H2 sections grouped by intent — Services, Proof, Company, Resources. Not by internal department.
  4. Links with descriptions. Every link should explain what the page answers. A bare URL is nearly useless.
  5. An Optional section. The convention reserves this for secondary links that are useful but not essential.

Keep the file under roughly 100 links. If it takes more than two screens to scan, it is a sitemap, not a summary. Use absolute URLs or root-relative paths consistently; do not mix them.

Rule: every line should help a reader answer "what is this page for?" If it only helps you rank, it belongs in a content brief, not in llms.txt.

A production-ready template

This is the structure we use. Replace the entity facts, keep the shape.

# Metaplaxio

> Independent agency working across web development, search visibility,
> and performance marketing.

Metaplaxio is based in India. Projects start at ₹24,900; monthly search
and ads retainers start at ₹14,900 per month.

## Services
- [Web Development](https://metaplaxio.com/services/web-engineering/): Landing pages, business websites, stores, and custom applications.
- [SEO, GEO & AEO](https://metaplaxio.com/services/geo-aeo/): Technical SEO, content structure, entity clarity, and AI-search readiness.
- [Performance Marketing](https://metaplaxio.com/services/performance-marketing/): Google and Meta campaigns with clear reporting.

## Work
- [Selected work](https://metaplaxio.com/case-studies/): Six real projects with clear status and live links.

## Company
- [About](https://metaplaxio.com/about/): Team and operating principles.
- [Process](https://metaplaxio.com/process/): Discovery, written proposal, and delivery.
- [Pricing](https://metaplaxio.com/pricing/): Starting prices and package scope.

## Optional
- [Insights](https://metaplaxio.com/insights/): Long-form field notes on GEO, speed, and attribution.
- [Contact](https://metaplaxio.com/contact/): Send a project enquiry.
- [Security](https://metaplaxio.com/legal/security/): Hosting, access control, and incident response.

A few details worth copying:

  • The summary names the category, audience, geography, and starting price. Those are the facts models most often get wrong.
  • Proof links point to case studies with the outcome in the description, not just the client name.
  • The Optional section holds insights and contact pages — useful, but not core to understanding the business.
  • There is no keyword stuffing, no emoji, and no marketing superlatives. The file reads like a reference document because that is what it is.

Implementation in Astro

The simplest approach is a static file in public/llms.txt. It ships as-is, has no build dependency, and is trivial to review in a pull request.

If you want the file to stay in sync with your content data, generate it as a route. In an Astro project with the Cloudflare adapter, mark it prerendered so it is emitted at build time:

// src/pages/llms.txt.ts
import type { APIRoute } from 'astro';

// export const prerender = true; // emit at build time

export const GET: APIRoute = async () => {
  const body = [
    '# Metaplaxio Growth Engine',
    '',
    '> Growth engineering agency combining headless web development,',
    '> GEO/AEO, and performance marketing.',
    '',
    '## Services',
    '- [Web Engineering](https://metaplaxio.com/services/web-engineering/): ...',
    '- [GEO & AEO](https://metaplaxio.com/services/geo-aeo/): ...',
    '',
    '## Optional',
    '- [Insights](https://metaplaxio.com/insights/): ...',
  ].join('\n');

  return new Response(body, {
    headers: {
      'Content-Type': 'text/plain; charset=utf-8',
      'Cache-Control': 'public, max-age=3600',
    },
  });
};

Two operational notes. First, set an explicit Content-Type of text/plain; charset=utf-8; some CDNs default to application/octet-stream, which makes the file download instead of display. Second, do not put the file behind authentication, a bot challenge, or a redirect chain. It should return 200 directly.

Add a link to it from your footer or resources page. That helps humans find it and gives crawlers a normal discovery path. It is also fine to reference it in your sitemap.xml as a low-priority URL, though it is not required.

Verification and monitoring

Do not assume deployment equals availability. We run five checks after every release:

  1. Status and headers. curl -I https://yourdomain.com/llms.txt should return 200 with a text content type and no X-Robots-Tag: noindex.
  2. Raw content. Fetch the body and confirm it matches the source. CDN caching and edge rules can serve an old version.
  3. Crawler access. Request the file with a GPTBot or PerplexityBot user agent from a server, not a browser. Confirm no firewall rule blocks it.
  4. Link integrity. Crawl every URL in the file and check for 200s, redirects, and canonical mismatches. A file full of 301s is worse than no file.
  5. Freshness. Compare the file against your current sitemap monthly. New service pages, renamed case studies, and retired offers should be reflected.

We also log requests to the file where the hosting layer allows it. That tells you whether any agent is actually reading it, which is useful context when someone asks for ROI on the effort.

Keeping it useful

The file decays for the same reason documentation decays: nobody owns it. Assign it to the same person who owns the sitemap and the canonical fact set. Review it on a monthly cadence alongside the prompt panel described in the GEO/AEO playbook.

A practical review checklist:

  • Did we launch or retire a service? Update the Services section.
  • Did a case study change its headline metric? Update the description.
  • Did pricing change? Update the summary — stale pricing is the most damaging error in the file.
  • Did we add a new canonical page that answers a core prompt? Add it, and remove the page it replaced.
  • Are all links still 200 and canonical? Run the crawler.

When not to bother

llms.txt is not worth the maintenance if:

  • Your site is a small brochure with five pages. A clear homepage and a sitemap are enough.
  • Your content is entirely gated. A model cannot cite what it cannot read.
  • You are blocking AI crawlers in robots.txt. Publishing a map you do not want followed is contradictory.
  • You cannot commit to keeping it current. A stale file actively misleads.

For everyone else, it is a cheap, low-risk clarity layer. It will not make you visible on its own. It will make the rest of the work — entity architecture, answer-ready pages, and technical access — easier for a retrieval system to interpret.

If you want a second pair of eyes, request a proposal. We will review your crawler access, entity consistency, and llms.txt as part of a defined search scope.