From the team behind Gatsby, Mastra is a framework for building AI-powered applications and agents with a modern TypeScript stack.

AI Observability & Evaluation

Python SDK for AI agent monitoring, LLM cost tracking, benchmarking, and more. Integrates with most LLMs and agent frameworks including CrewAI, Agno, OpenAI Agents SDK, Langchain, Autogen, AG2, and CamelAI

Build, Evaluate, and Optimize AI Systems. Includes evals, RAG, agents, fine-tuning, synthetic data generation, dataset management, MCP, and more.

AI observability platform for production LLM and agent systems.

Evaluation and Tracking for LLM Experiments and AI Agents

Laminar - open-source observability platform purpose-built for AI agents. YC S24.

Development platform to debug, chat, inspect, and evaluate MCP servers, MCP apps, and ChatGPT apps.

Harbor is a framework for running agent evaluations and creating and using RL environments.

Evaluate your LLM-powered apps with TypeScript

OpenSource Production ready Customer service with built in Evals and monitoring

🥤 RAGLite is a Python toolkit for Retrieval-Augmented Generation (RAG) with DuckDB or PostgreSQL

Cloud-synced dashboards for OpenCode and Claude Code. Track sessions, search with semantic lookup, export eval datasets.

AI system design guide for engineers building production AI systems and evals.

Test Generation for Prompts

[NeurIPS 2024] Official code for HourVideo: 1-Hour Video Language Understanding

Vivaria is METR's tool for running evaluations and conducting agent elicitation research.

A Node.js package and GitHub Action for evaluating MCP (Model Context Protocol) tool implementations using LLM-based scoring. This helps ensure your MCP server's tools are working correctly and performing well.

AgentEval is the comprehensive .NET toolkit for AI agent evaluation—tool usage validation, RAG quality metrics, stochastic evaluation, and model comparison—built first for Microsoft Agent Framework (MAF) and Microsoft.Extensions.AI. What RAGAS, PromptFoo and DeepEval do for Python, AgentEval does for .NET

A benchmark suite for evaluating how coding models solve real React Native tasks.