Developer tools for testing and evaluating LLM agents, seed stage

1. AgentOps.ai — Observability, evals, and replay for debugging LLM agents across frameworks like AutoGen and CrewAI.

Why it fits: Seed-stage, narrowly focused on agent lifecycle rather than generic LLM apps; traction via framework integrations and agent-specific debugging features.

1. Parea AI — Platform for LLM app evaluation, monitoring, and prompt experimentation with auto-generated evals.

Why it fits: YC W23, seed-stage; early team explicitly targeting eval workflows for agent and chain builders, with a developer-first API.

1. Athina AI — Collaborative evaluation and monitoring platform for LLM apps, including side-by-side comparisons and custom metrics.

Why it fits: Seed-stage ($3M raised), less hyped than LangSmith but used by teams building agents; strong focus on eval collaboration and regression testing.

1. Langfuse — Open-source LLM engineering platform with tracing, evals, prompt management, and datasets.

Why it fits: Seed/early-stage, strong OSS adoption (GitHub stars) and self-serve usage; evals are a core module, making it a direct fit for testing agents.

1. Promptfoo — Open-source CLI and library for LLM testing, red-teaming, and evaluation.

Why it fits: Seed-stage company (Luminos) behind a popular OSS project; directly targets eval and adversarial testing, including agent outputs.

1. Langtail — LLM app testing and observability platform with no-code test creation and CI/CD integration.

Why it fits: YC S23, seed-stage; focused on testing prompts and agent chains, with emphasis on regression tests in deployment pipelines.

1. Okareo — Evaluation and testing platform for LLM apps, including agent-specific evals and synthetic data generation.

Why it fits: YC W23, seed-stage; early team building evaluation tooling for agents, with support for custom checks and scenario testing.

1. Confident AI (DeepEval) — Open-source LLM evaluation framework (DeepEval) plus a cloud platform for CI/CD evals.

Why it fits: Seed-stage, OSS traction via DeepEval; positions itself as “unit testing for LLMs,” including agent and tool-call assertions.

1. Giskard — Open-source testing and evaluation library for ML and LLMs, with vulnerability scanning and red-teaming.

Why it fits: Seed-stage, European startup; focuses on quality and security testing for AI systems, including LLM agents and RAG pipelines.

1. Lunary — Open-source LLM observability and evals platform with prompt management and user feedback tracking.

Why it fits: YC W23, seed-stage; lesser-known than Helicone or Langfuse but includes eval and testing features aimed at agent builders.

White-space gaps I noticed while sourcing:

1. Multi-agent interaction and environment simulation — Most tools evaluate a single agent’s output, but few simulate multi-agent negotiation, emergent behavior, or tool-use side effects in a sandboxed environment. A seed-stage company building “agent regression suites” with simulated other agents and APIs would be highly differentiated.
2. Automatic test generation from production traces — There is little tooling that ingests real user sessions or agent runs and automatically converts them into eval cases, especially for non-text actions like API calls, database writes, or permission changes. This would close the loop between observability and continuous testing for agents.