[plan] Add robots.txt and llms.txt to docs site for AI crawler discoverability

[github-actions]

From the GEO Audit Report (discussion #30401): the docs site (github.github.com/gh-aw/) scored 0/18 for both robots.txt and llms.txt, together representing 36 potential points.

Approach

1. Add robots.txt to docs site

Create docs/public/robots.txt that explicitly allows major AI crawlers:

User-agent: *
Allow: /

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Sitemap: https://github.github.com/gh-aw/sitemap-index.xml

2. Add llms.txt to docs site

Create docs/public/llms.txt summarizing the project for AI consumption. Mirror the structure from the README's llms.txt but tailored to the docs site, including:

• Project overview
• Key concepts (workflows, engines, MCP servers)
• Links to main docs sections
• Installation and quick-start

Files to Create

• docs/public/robots.txt
• docs/public/llms.txt

Acceptance Criteria

• https://github.github.com/gh-aw/robots.txt returns 200 with AI bot Allow rules
• https://github.github.com/gh-aw/llms.txt returns 200 with project summary
• llms.txt has at minimum 500 words and covers core concepts

Generated by Plan Command for issue #discussion #30401 · ● 2.8M · ◷

• expires on May 8, 2026, 1:52 AM UTC