Evaluation Infrastructure for AI Agents
-
Updated
Feb 25, 2026 - TypeScript
#
ai-evals
Here are 18 public repositories matching this topic...
A hands-on interactive AI-evals course for product folks, who want to develop product sense based on real life applications.
bootcamp red-teaming rag prompt-engineering llmops ai-product-management llm-evaluation claude-code ai-pm ai-evals
-
Updated
Apr 24, 2026
🔍 Benchmark jailbreak resilience in LLMs with JailBench for clear insights and improved model defenses against jailbreak attempts.
python flask analytics openai alignment model-evaluation ai-safety security-testing red-teaming model-robustness anthropic litellm content-safety llm-jailbreaks tool-calling llm-benchmark ai-evals textual-tui
-
Updated
Apr 25, 2026 - Python
Benchmark LLM jailbreak resilience across providers with standardized tests, adversarial mode, rich analytics, and a clean Web UI.
python flask analytics openai alignment model-evaluation ai-safety security-testing red-teaming model-robustness prompt-injection anthropic litellm content-safety llm-jailbreaks tool-calling llm-benchmark adversarial-testing ai-evals textual-tui
-
Updated
Aug 12, 2025 - Python
Unofficial TypeScript starter for deterministic local contract testing around Foundry-oriented workflows with Themis.
react typescript schema-validation themis contract-testing osdk developer-tooling agentic-workflows ai-evals foundry-workflows
-
Updated
Mar 28, 2026 - TypeScript
Free, local Langfuse OSS setup with Ollama for LLM evaluation, scoring, and datasets.
docker open-source self-hosted free no-cost local-llm ollama langfuse llm-evaluation prompt-evaluation offline-ai llm-as-judge llm-observability ai-evals
-
Updated
Apr 13, 2026 - TypeScript
End-to-end AI evals orchestration platform for comparing LLM outputs across providers with transcription, structured logging, human review, and Supabase-backed decision tracking.
gemini openai multi-model transcription human-in-the-loop model-comparison supabase anthropic llm-evaluation ai-evals evaluation-pipeline
-
Updated
Mar 10, 2026 - TypeScript
Experimentation framework for LLM systems using simulated users, conversational behavioral metrics, and causal inference to evaluate prompt strategies, temperature, and model scaling.
-
Updated
Mar 8, 2026 - Python
Lightweight eval framework for LLMs & AI apps combining deterministic scoring, LLM-as-judge, and regression testing.
-
Updated
Apr 8, 2026 - Python
Hands-on Agentic AI learning project — ReAct agents, memory systems, evals, and multi-agent architecture. Built as a structured AI PM curriculum.
python product-management react-pattern ai-agent llm research-automation anthropic llm-as-judge agentic-ai ai-evals
-
Updated
Apr 22, 2026 - Python
Multi-agent system orchestrating an AI-driven software team using the Claude Agents SDK. Agents take on defined roles and collaborate autonomously on software tasks.
jira ai orchestration multi-agent confluence atlassian llm generative-ai anthropic llm-agents agentic-ai mcp-server ai-evals claude-agent-sdk
-
Updated
Feb 4, 2026 - Python
An AI-powered tool for data extration designed for the Job Intelligence Engine project
-
Updated
Apr 9, 2026 - Python
Collection of frameworks and tools for AI evalations, including tool-use, agentic AI, MCP, and multimodal
-
Updated
Apr 25, 2026 - Python
EvalLoop is a self-improving agent that iterates on its own outputs using evals + automatic policy patches. It runs a task, scores the result against a rubric, updates its rules, and re-runs until it hits a target score with a UI showing attempts, score trends, violations, and policy diffs.
-
Updated
Mar 17, 2026 - TypeScript
Portfolio project showing product strategy, evals, roadmap, and GTM for a GenAI coding assistant
developer-tools product-management portfolio-project product-strategy llm genai ai-coding-assistant ai-evals
-
Updated
Apr 2, 2026
CLI release gate for structured AI changes.
-
Updated
Mar 12, 2026 - TypeScript
Build deterministic local contract tests for Foundry workflows with TypeScript, schema validation, telemetry, and proof artifact export
react typescript schema-validation themis contract-testing osdk developer-tooling agentic-workflows ai-evals foundry-workflows
-
Updated
Apr 25, 2026 - TypeScript
Improve this page
Add a description, image, and links to the ai-evals topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the ai-evals topic, visit your repo's landing page and select "manage topics."