supereval Roadmap
Tracks what is currently supported, what is planned, and what is out of scope.
Document Sources (for supereval generate)
| Source |
Status |
Notes |
Local files (.txt, .md) |
Supported |
|
Local files (.pdf) |
Supported |
Requires pip install 'supereval[pdf]' |
| Local directory (recursive) |
Supported |
All supported file types under a path |
| Amazon S3 |
Supported |
s3://bucket/prefix — S3Source in sources.py; requires boto3 + s3:ListObjectsV2/GetObject |
| URLs |
Supported |
HTTP/HTTPS fetch + HTML stripping; auto-detected from --from |
| Amazon Kendra |
Coming Soon |
Pull documents from a Kendra index |
| Bedrock Knowledge Base |
Coming Soon |
Pull source documents from a KB |
Dataset Types
| Type |
Registry |
Generation |
Notes |
qa |
Supported |
Supported |
Query + ground truth |
classification |
Supported |
Supported |
Label-aware generation; requires --labels set on dataset |
instruction |
Supported |
Supported |
Rubric-based generation |
rag |
Supported |
Supported |
Extends qa with retrieved_contexts; Faithfulness + Answer Correctness scoring via Bedrock judge; static contexts (Option A) |
rag (live retrieval) |
Planned |
— |
Option B: contexts fetched live from your retriever at eval time; tests the full RAG pipeline end-to-end |
multiturn |
Coming Soon |
Coming Soon |
Conversation datasets |
agent |
Supported |
Supported |
Trajectory + tool-call evaluation; trace import (--from traces.jsonl) and doc-based generation (--from docs/) both supported |
Generator Backends (for supereval generate)
| Backend |
Status |
Notes |
| Amazon Bedrock |
Supported |
Default. Requires boto3 + Bedrock model access |
| Anthropic API |
Supported |
--backend anthropic; requires pip install 'supereval[anthropic]' + ANTHROPIC_API_KEY |
| OpenAI |
Supported |
--backend openai; requires pip install 'supereval[openai]' + OPENAI_API_KEY |
| Azure OpenAI |
Supported |
--backend azure-openai; requires pip install 'supereval[openai]'; set AZURE_OPENAI_ENDPOINT or pass --azure-endpoint |
CI Systems (for supereval run + .github/workflows/)
| System |
Status |
Notes |
| GitHub Actions |
Supported |
Template at .github/workflows/eval.yml |
| AWS CodeBuild / CodePipeline |
Supported |
codebuild/buildspec.yml; set SUPEREVAL_MODE=pr|main |
| GitLab CI |
Supported |
Template at gitlab/eval.yml |
| CircleCI |
Supported |
Template at circleci/config.yml |
Result Storage (for supereval run)
| Storage |
Status |
Notes |
baseline.json committed to git |
Supported |
Simple, reproducible, reviewable in PRs |
Per-run results JSON (--output) |
Supported |
Written locally; not persisted automatically |
| DuckDB local store |
Supported |
supereval history list/show/stats; SUPEREVAL_DB_PATH to override location |
| S3 results store |
Planned |
Team-shared history; needed for dashboards |
| Langfuse integration |
Coming Soon |
Post-deployment monitoring alongside offline evals; trace supereval runs to Langfuse for production comparison |
Cost & Latency Tracking
| Feature |
Status |
Notes |
| Per-run latency in results JSON |
Supported (via Promptfoo) |
|
| Per-run token cost in results JSON |
Supported (via Promptfoo) |
|
| Persistent cost/latency history |
Supported |
DuckDB store; supereval history commands |
| Cost regression alerts |
Supported |
max_cost_usd threshold in dataset.json |
| Latency p95 tracking |
Supported |
max_p95_latency_ms threshold; p50/p95 stored per run |
Agent Eval (supereval agent)
Core (v1)
| Feature |
Status |
Notes |
agent dataset type with tool catalog |
Supported |
dataset.json holds tool specs (name, description, JSON Schema) |
| Per-case mock tool environments |
Supported |
Canned responses + errors; keeps test cases self-contained and reproducible |
AgentRunner protocol |
Supported |
Framework-agnostic; customers implement run(task, tool_executor) -> Trajectory |
MockToolExecutor |
Supported |
Returns canned responses from test case definition |
| Full trajectory model |
Supported |
Steps typed as thought / tool_call / observation / answer |
| Answer scoring |
Supported |
exact / contains / regex / LLM judge |
| Tool-call scoring |
Supported |
must_call, must_not_call, must_call_with checks |
| Efficiency scoring |
Supported |
Normalized step count vs. max_steps (informational by default) |
| Reasoning scoring via LLM judge |
Supported |
Bedrock judge evaluates thought chain for coherence and hallucinations |
| Configurable score thresholds |
Supported |
Per-dataset min_answer_score, min_tool_score, min_reasoning_score |
| Baseline & regression detection |
Supported |
Same pattern as LLM evals; --compare-baseline / --update-baseline |
supereval agent run --runner module:Class |
Supported |
Python import path to customer's AgentRunner implementation |
| Cost & latency tracking |
Supported |
Unified supereval.db with LLM runs; run_type column distinguishes them |
supereval agent history subcommand |
Supported |
list / show / stats; reuses store.py |
| Reference AgentRunner examples |
Supported |
examples/: custom ReAct/Bedrock, Bedrock Agents, LangChain |
Judge improvements
Informed by comparison with AWS Bedrock's built-in LLM-as-judge metric suite (12 metrics across quality, safety, and style dimensions).
| Feature |
Status |
Notes |
Split judge_reasoning into Faithfulness + Logical Coherence |
Supported |
Two focused Bedrock calls; Faithfulness = did agent hallucinate from tool results?; Logical Coherence = did each step follow from the previous? |
| Adopt per-metric ordinal scoring scales |
Supported |
3-point for Correctness, 5-point for Completeness / Faithfulness / Coherence; native labels ("partially correct", "most faithful") surfaced alongside normalized 0.0–1.0 float |
| Make ground truth optional in answer judging |
Supported |
expected_answer is now optional; judge degrades gracefully to rubric-only scoring when omitted |
Add Helpfulness / Relevance dimension to judge_answer |
Supported |
5-point scale; third Bedrock call alongside Correctness + Completeness; normalized score averaged into composite |
Out of scope for v1 (prioritize later)
| Feature |
Notes |
| Multi-turn conversation state |
Each test case is a single task invocation; stateful conversation datasets are v2 |
| Parallel tool calls |
Tool calls are sequential for now; parallel execution adds replay complexity |
| Synthetic generation for agent cases |
Supported (supereval agent generate --from docs/) — LLM generates task scenarios from documentation; tools={} block left empty, fill via trace import |
| Real framework adapters in the package |
Reference impls live in examples/ only; not installed with supereval |
| Trajectory similarity scoring |
Comparing step sequences across runs (edit distance, embedding similarity) |
| Bedrock Agents native integration |
Polling GetAgentExecution, parsing Bedrock Agents trace format |
| Agent dataset generation from existing traces |
Supported (supereval agent generate --from traces.jsonl) — imports real execution traces; infers tool mocks from step results |
| AWS Step Functions / multi-agent orchestration |
Evaluating pipelines of agents, not single agents |
Other Features
| Feature |
Status |
Notes |
supereval providers |
Supported |
Lists Bedrock model IDs, Anthropic API model IDs, and Promptfoo provider IDs |
| Multi-dataset CI runs |
Supported |
supereval run-all; exits non-zero if any dataset fails |
| Dataset versioning (tags) |
Supported |
supereval dataset version tag/list/show/restore; semantic version snapshots stored under datasets/<name>/versions/ |
| Interactive case review |
Supported |
supereval generate --interactive and supereval agent generate --interactive; keep/edit/skip/quit per case |
| MCP server |
Supported |
supereval mcp; 17 tools for Claude Code, Kiro, and any MCP-compatible AI tool; pip install 'supereval[mcp]' |