Skip to content

Instrument requirement validators to track validation attempts and failures #473

@psschwei

Description

@psschwei

Description:
Instrument requirement validators to track validation attempts and failures.

Detailed Requirements:

  1. Add spans for validation:
    • validator.check - Individual validation check
  2. Add attributes:
    • requirement_type - Type of requirement (python, markdown, llmaj, etc.)
    • requirement_name - Specific requirement identifier
    • status - pass/fail
    • failure_reason - Reason for failure (on fail)
  3. Add span events for:
    • Validation start with input summary
    • Validation result with details
  4. Instrument all requirement types:
    • PythonRequirement
    • MarkdownRequirement
    • ToolCallRequirement
    • ALoraRequirement
    • LLMaJRequirement

Files to Modify:

  • mellea/stdlib/requirements/base.py - Base instrumentation
  • mellea/stdlib/requirements/python.py - Python requirement
  • mellea/stdlib/requirements/markdown.py - Markdown requirement
  • mellea/stdlib/requirements/tool_call.py - Tool call requirement
  • mellea/stdlib/requirements/alora.py - ALoRA requirement
  • mellea/stdlib/requirements/llmaj.py - LLMaJ requirement

Acceptance Criteria:

  • Each validation check has a span
  • Pass/fail status captured
  • Failure reasons recorded
  • All requirement types instrumented

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions