Skip to content

Instrument the sampling loop to track each iteration attempt, providing visibility into retry behavior #472

@psschwei

Description

@psschwei

Description:
Instrument the sampling loop to track each iteration attempt, providing visibility into retry behavior.

Detailed Requirements:

  1. Add spans for sampling operations:
    • sampler.sample - Overall sampling operation
    • sampler.iterate - Single sampling iteration
    • sampler.repair - Repair attempt
    • sampler.select - Selection from candidates
  2. Add attributes:
    • strategy - Sampling strategy name
    • iteration - Current iteration number
    • max_iterations - Maximum allowed iterations
    • status - success/failure/repair
  3. Track iteration outcomes:
    • Add span events for validation pass/fail
    • Record reason for rejection

Files to Modify:

  • mellea/stdlib/sampling/base.py - Instrument BaseSamplingStrategy
  • mellea/stdlib/sampling/rejection.py - Instrument RejectionSamplingStrategy
  • mellea/stdlib/sampling/sofai.py - Instrument SoFAISamplingStrategy
  • mellea/stdlib/sampling/majority_voting.py - Instrument MajorityVotingStrategy

Span Hierarchy Example:

sampler.sample
├── sampler.iterate (attempt 1)
│   ├── backend.request
│   └── validator.check (FAIL)
├── sampler.iterate (attempt 2)
│   ├── backend.request
│   └── validator.check (PASS)
└── [success]

Acceptance Criteria:

  • Each sampling iteration has its own span
  • Iteration count and outcomes tracked
  • Strategy-specific behavior captured
  • Spans correctly nested

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions