- AgentOps.ai — Observability, evals, and replay for debugging LLM agents across frameworks like AutoGen and CrewAI.
Why it fits: Seed-stage, narrowly focused on agent lifecycle rather than generic LLM apps; traction via framework integrations and agent-specific debugging features.
- Parea AI — Platform for LLM app evaluation, monitoring, and prompt experimentation with auto-generated evals.
Why it fits: YC W23, seed-stage; early team explicitly targeting eval workflows for agent and chain builders, with a developer-first API.
- Athina AI — Collaborative evaluation and monitoring platform for LLM apps, including side-by-side comparisons and custom metrics.
Why it fits: Seed-stage ($3M raised), less hyped than LangSmith but used by teams building agents; strong focus on eval collaboration and regression testing.
- Langfuse — Open-source LLM engineering platform with tracing, evals, prompt management, and datasets.
Why it fits: Seed/early-stage, strong OSS adoption (GitHub stars) and self-serve usage; evals are a core module, making it a direct fit for testing agents.
- Promptfoo — Open-source CLI and library for LLM testing, red-teaming, and evaluation.
Why it fits: Seed-stage company (Luminos) behind a popular OSS project; directly targets eval and adversarial testing, including agent outputs.
- Langtail — LLM app testing and observability platform with no-code test creation and CI/CD integration.
Why it fits: YC S23, seed-stage; focused on testing prompts and agent chains, with emphasis on regression tests in deployment pipelines.
- Okareo — Evaluation and testing platform for LLM apps, including agent-specific evals and synthetic data generation.
Why it fits: YC W23, seed-stage; early team building evaluation tooling for agents, with support for custom checks and scenario testing.
- Confident AI (DeepEval) — Open-source LLM evaluation framework (DeepEval) plus a cloud platform for CI/CD evals.
Why it fits: Seed-stage, OSS traction via DeepEval; positions itself as “unit testing for LLMs,” including agent and tool-call assertions.
- Giskard — Open-source testing and evaluation library for ML and LLMs, with vulnerability scanning and red-teaming.
Why it fits: Seed-stage, European startup; focuses on quality and security testing for AI systems, including LLM agents and RAG pipelines.
- Lunary — Open-source LLM observability and evals platform with prompt management and user feedback tracking.
Why it fits: YC W23, seed-stage; lesser-known than Helicone or Langfuse but includes eval and testing features aimed at agent builders.
White-space gaps I noticed while sourcing:
- Multi-agent interaction and environment simulation — Most tools evaluate a single agent’s output, but few simulate multi-agent negotiation, emergent behavior, or tool-use side effects in a sandboxed environment. A seed-stage company building “agent regression suites” with simulated other agents and APIs would be highly differentiated.
- Automatic test generation from production traces — There is little tooling that ingests real user sessions or agent runs and automatically converts them into eval cases, especially for non-text actions like API calls, database writes, or permission changes. This would close the loop between observability and continuous testing for agents.