Requirements Library: Document failure matrix and repair guidance

[akihikokuroda]

Create comprehensive documentation linking requirements to failure modes and repair strategies.

Documentation

• Failure matrix: map each failure mode to which requirement addresses it
• Repair guidance: document all error messages in model-friendly format
• Examples: show failures, repair feedback, and regenerated code
• Integration: link to API reference pages

Tests

• Add documentation tests verifying code examples work correctly
• Test that repair messages are well-formed and actionable

Part of #1023

[akihikokuroda]

Design: Failure Matrix and Repair Guidance Documentation

Overview

Task #1124 creates comprehensive documentation for Python code generation failures, their requirements, and repair guidance. This complements the API reference by showing real-world failure scenarios and how requirements prevent them.

Documentation Standards (from CONTRIBUTING_DOCS.md)

Key constraints:

• No H1 in body (frontmatter title only)
• Diataxis classification: reference (failure matrix table) + explanation (why each requirement matters)
• Progressive disclosure: Table first, then detailed explanations
• Code blocks: All have language tags, verified against current source
• Markdown lint: Zero warnings before merge
• Target length: 300–600 lines; split if >800
• Relative links: Within docs use relative paths (no .md extensions)
• Glossary linking: First use of Mellea terms only (Requirement, ValidationResult, etc.)

Documentation Structure

Main File: docs/docs/guides/python-requirements.md

Frontmatter:

---
title: "Python code generation requirements"
description: "Failure modes, requirement validators, and repair guidance for Python and matplotlib code generation."
# diataxis: reference
---

Why "reference"? Readers use this as a lookup when requirements fail — they need the table and specific repair patterns, not a learning sequence.

Page Organization

## Overview (1–2 paragraphs)
## Quick reference table
## Detailed failure modes
### Generic Python tool requirements (H3)
#### Code not extractable
#### Syntax errors
#### Runtime exceptions
#### Output too large
#### Unauthorized imports
### Matplotlib-specific requirements (H3)
#### Headless backend not configured
#### Plot not saved to file
#### Dependencies missing

**See also:** [Glossary](../reference/glossary) | [Requirements API](../../api/mellea/stdlib/requirements)

Page Length Target

Approximately 400–500 lines: matrix (50 lines) + 7 failure modes × 50–60 lines each.

Failure Matrix (Quick Reference)

## Quick reference

| Failure | Requirement | Detection | Feedback |
|---------|-------------|-----------|----------|
| Code not in ```python blocks | `PythonCodeExtraction` | Regex search | "Wrap code in ```python ... ```" |
| Invalid Python syntax | `PythonSyntaxValid` | AST parse | "Line N: {error}. Try: {fix}" |
| Code raises exception | `PythonExecutesWithoutError` | Execute in sandbox | "{ExceptionType}: {msg}. Try: {fix}" |
| Output exceeds limit | `OutputSizeLimit` | Measure stdout | "Output {actual} chars; limit {limit}" |
| Import not allowed | `ImportRestrictions` | AST + runtime | "Import {module} not allowed" |
| No headless backend | `MatplotlibHeadlessBackend` | AST for matplotlib.use() | "Add matplotlib.use('Agg')" |
| Plot not saved | `PlotFileSaved` | AST for savefig() | "Add plt.savefig('{path}')" |
| Missing dependencies | `PlotDependenciesAvailable` | Try imports | "Install: pip install {module}" |

Detailed Failure Modes

Template for Each Failure (50–60 lines)

### Code not extractable

**When it happens:** Model output has no recognizable Python code blocks.

**Root cause:** Model generated text without wrapping code in ```python ... ``` fencing.

**Requirement:** [`PythonCodeExtraction`](../../api/mellea/stdlib/requirements) — validates that code blocks are present and extractable from model output.

**Detection method:** Regex search for ````python\n...\n````, with fallback to generic ``` ``` if code looks like Python.

**Repair feedback:**

```text
No Python code blocks found. Wrap your code in ```python ... ``` blocks.

Example:

Model output (fails):

Here's a function that sums numbers:

def sum_numbers(x, y):
    return x + y

print(sum_numbers(3, 4))

Feedback to model:

No Python code blocks found. Wrap your code in ```python ... ``` blocks.

Regenerated output (passes):

def sum_numbers(x, y):
    return x + y

print(sum_numbers(3, 4))

#### Advanced callout placement

Use `> **Advanced:**` for technical details:

```markdown
> **Advanced:** [Glossary: `ValidationResult`](../reference/glossary#validationresult) contains both the pass/fail boolean (`as_bool()`) and detailed reason. The feedback string is extracted from `ValidationResult.reason`.

Linking to Glossary

First-use glossary links only (per CONTRIBUTING_DOCS.md):

First mention of each term should link to glossary:

• Requirement on first mention
• ValidationResult on first mention
• Sampling strategy on first mention

Subsequent mentions use plain text — no link repetition.

Cross-linking Strategy

Link FROM:

• Sampling strategies page → "See also: Failure modes"
• Requirements API docs → "For common failures and repair patterns, see Python requirements guide"

Link TO:

• Requirements API reference
• Glossary
• Examples: [docs/examples/requirements/](https://github.com/generative-computing/mellea/blob/main/docs/examples/requirements/)

Markdown Lint Configuration

Config is in docs/docs/guide/.markdownlint.json. Common checks:

• No H1 (starts with H2) ✓
• Link validity (relative paths resolve) ✓
• Code blocks have language tags ✓
• Lists use consistent formatting ✓

Code Examples in Documentation

All code blocks must:

1. Have language tag: ```python, ```text
2. Be minimal but complete (no unexplained variables)
3. Show expected output as comment or text block
4. Include all necessary imports

Non-deterministic note:

When showing model outputs:

> **Note:** LLM output is non-deterministic. Your exact error messages and repair suggestions may vary.

File Organization

docs/docs/
├── guides/
│   └── python-requirements.md (NEW)
├── reference/
│   └── glossary.md (add missing terms if needed)
└── api/
    └── mellea/stdlib/requirements/ (exists; link to it)

Terms to Add to Glossary (if missing)

Check docs/docs/reference/glossary.md and add if absent:

• Requirement / req() / check() — already there? ✓
• ValidationResult — check if exists
• PythonExecutionReq — specific class, may need entry or cross-reference
• OutputSizeLimit — specific requirement, may document in API only

Testing Before Merge

• Run markdownlint docs/docs/guides/python-requirements.md
• Verify all relative links resolve (no 404s)
• Verify external links work (API ref, glossary)
• Spell-check, US English throughout
• Frontmatter present (title, description, diataxis)
• No H1 in body (only H2+)
• All code blocks have language tags
• Glossary terms linked on first use only
• No TODO placeholders or [FIXME] markers

PR Checklist Items Specific to #1124

• Frontmatter with # diataxis: reference
• All code blocks verified against current Mellea source
• Relative links use no .md extensions (e.g., ../reference/glossary not ../reference/glossary.md)
• US English throughout (e.g., "behavior", not "behaviour")
• No real API keys or credentials
• Glossary terms linked on first use (see CONTRIBUTING_DOCS.md "Glossary and terminology")
• **See also:** footer with cross-links
• Page length 300–600 lines (check with wc -l)
• Previewed locally with mintlify dev

Integration with Examples

Link to example directory, not specific files (files may change):

> **Full examples:** See [`docs/examples/requirements/`](https://github.com/generative-computing/mellea/blob/main/docs/examples/requirements/) for end-to-end demonstrations of sampling presets and requirement validation.

What NOT to Do (per CONTRIBUTING_DOCS.md)

• ❌ No H1 in body (title via frontmatter)
• ❌ No [text.md] links (use [text])
• ❌ No "simply", "just", "easy", "obviously"
• ❌ No visible TODO placeholders
• ❌ No padding like "In this section we will..."
• ❌ Don't link to glossary on every mention (first use only)
• ❌ Don't leave code examples that are incomplete or won't run

Success Criteria

• ✅ markdownlint passes with zero warnings
• ✅ Frontmatter correct (title, description, diataxis: reference)
• ✅ All 7 failure modes documented with examples
• ✅ Links are relative, resolve correctly
• ✅ Glossary terms linked on first use
• ✅ Code blocks verified against current source
• ✅ US English throughout
• ✅ No H1 in body, only H2+
• ✅ 300–600 lines (or two pages if split)