feat: agent-friendly docs (llms.txt + markdown endpoints)

[marc0olo]

Summary

Implements the Agent-Friendly Documentation spec so AI agents can discover and consume our docs programmatically.

What gets deployed

cli.internetcomputer.org/
├── llms.txt                        ← discovery index (points to /0.2/*.md URLs)
├── index.html                      ← existing redirect to /0.2/
├── versions.json                   ← existing
├── 0.2/
│   ├── llms.txt                    ← version-local copy (same content)
│   ├── quickstart.md               ← clean markdown endpoint (frontmatter stripped)
│   ├── guides/
│   │   ├── installation.md
│   │   └── ...
│   ├── quickstart/
│   │   └── index.html              ← existing Starlight HTML (now with agent signaling)
│   └── ...
├── 0.1/                            ← HTML only until docs/v0.1 branch gets the plugin
│   └── ...
└── main/
    ├── llms.txt                    ← points to /main/*.md URLs
    ├── quickstart.md
    └── ...

Components

1. astro-agent-docs.mjs — Astro integration (astro:build:done hook)

   • Copies each source .md file into build output (frontmatter stripped, title prepended as # heading, UTF-8 BOM)
   • Generates llms.txt index with pages grouped by sidebar section, using absolute URLs with the versioned base path

2. rehype-agent-signaling.mjs — Rehype plugin

   • Injects a visually-hidden <blockquote class="agent-signaling"> at the top of every HTML page
   • Points agents to /llms.txt for documentation discovery
   • Uses CSS clip: rect(0,0,0,0) (not display:none) so it survives HTML-to-markdown conversion

3. Workflow changes (docs.yml)

   • publish-versioned-docs now checks if the deployed version is the latest (via versions.json)
   • If latest, also deploys llms.txt to root so agents discover it at the well-known /llms.txt path
   • Root llms.txt is built from the same source as the versioned docs — zero content drift

Why .md files live inside versioned folders

The .md files and llms.txt are generated during the same Astro build as the HTML pages. This means:

• Zero drift — markdown endpoints always match the deployed HTML content
• Per-version consistency — each version's llms.txt links to its own .md files
• Root /llms.txt is just a copy from the latest version's build output, deployed in the same workflow step

Checks this addresses

Check	Status
llms-txt-exists	✅ Fixed — /llms.txt at root
llms-txt-valid	✅ Fixed — proper format with sections
llms-txt-links-resolve	✅ Fixed — links to versioned .md files
llms-txt-links-markdown	✅ Fixed — all links are .md URLs
markdown-url-support	✅ Fixed — .md files served alongside HTML
llms-txt-directive	✅ Fixed — agent signaling on every page
content-negotiation	⬚ Not fixable — IC asset canister doesn't support Accept header routing
http-status-codes	⬚ Separate issue — asset canister soft 404 behavior
redirect-behavior	⬚ Acceptable — root redirect is bypassed by llms.txt discovery

Test plan

• Local build succeeds with PUBLIC_BASE_PATH=/0.2/
• llms.txt contains all 31 pages grouped by section with correct absolute URLs
• .md endpoints have clean content (no frontmatter, title heading, UTF-8 BOM)
• Agent-signaling <blockquote> present in HTML output
• Verify CI build passes
• After deploy, run npx afdocs check https://cli.internetcomputer.org to confirm improvements