Skip to content

Add validation error feedback for LLM retry attempts (#102)#221

Merged
neoneye merged 5 commits intomainfrom
feature/102-error-feedback-retries
Mar 11, 2026
Merged

Add validation error feedback for LLM retry attempts (#102)#221
neoneye merged 5 commits intomainfrom
feature/102-error-feedback-retries

Conversation

@neoneye
Copy link
Member

@neoneye neoneye commented Mar 9, 2026

Summary

How it works

  1. LLM returns invalid JSON that fails Pydantic validation
  2. _extract_validation_feedback() walks the exception chain and formats the error
  3. The executor stores feedback in validation_feedback and retries the same model
  4. The caller's execute_function can check executor.validation_feedback and include it in the prompt

Changes

  • worker_plan/worker_plan_internal/llm_util/llm_executor.py:
    • Add _extract_validation_feedback() — walks exception chains for Pydantic errors
    • Add max_validation_retries parameter to LLMExecutor
    • Add validation_feedback property
    • Retry loop for validation errors in run()

Test plan

  • Verify Python syntax with ast.parse()
  • Existing test_llm_executor.py tests pass (CI)
  • Integration test: structured LLM call with intentionally bad schema → verify retry with feedback

🤖 Generated with Claude Code

@neoneye neoneye deleted the branch main March 10, 2026 00:48
@neoneye neoneye closed this Mar 10, 2026
@neoneye neoneye reopened this Mar 10, 2026
@neoneye neoneye changed the base branch from feature/plan-resume-tool to main March 10, 2026 01:43
When Pydantic validation fails, extract structured error details and
make them available via LLMExecutor.validation_feedback so the caller's
execute_function can inject the feedback into the next prompt attempt.

- Add _extract_validation_feedback() to walk exception chains and
  format Pydantic ValidationError details
- Add max_validation_retries parameter to LLMExecutor
- Add validation_feedback property for callers to read
- Retry same model on validation errors with feedback available

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@neoneye neoneye force-pushed the feature/102-error-feedback-retries branch from b032785 to 1575b4d Compare March 11, 2026 14:38
neoneye and others added 4 commits March 11, 2026 15:52
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…on (#221)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@neoneye neoneye merged commit 5770599 into main Mar 11, 2026
3 checks passed
@neoneye neoneye deleted the feature/102-error-feedback-retries branch March 11, 2026 15:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant