Agentic Evaluation SDK
An evaluation SDK that turns LLM interactions into structured cases, scored with independent metrics so agent behavior is testable.
Case summary
Model interactions are difficult to release safely when evaluation is a one-off report instead of a repeatable gate.
LLM interactions become structured evaluation cases, captured, scored with independent metrics, and mapped to a deployment risk tier that can gate a release.
Chain of evidence
The first figure follows an interaction from capture to a risk tier.
An interaction is captured, scored by independent metrics, aggregated, and mapped to a risk tier that can gate a release.
Runtime evaluation, without a rewrite
The second shows the runtime path: an existing Bedrock client wrapped and evaluated in the background, feeding the same classification.
An existing Bedrock client is wrapped, not replaced. Calls are traced, redacted, and scored in the background.
Evaluation zones
Any interaction, offline or live, normalizes into the same evaluation-case shape.
Heuristic metrics and LLM-as-judge share one scoring contract, with an external framework as an optional layer.
A wrapped Bedrock client evaluates calls in the background without touching invocation code.
Metric scores and data sensitivity together decide the risk tier that can gate a release.
Case narrative
I treat evaluation as a chain of evidence: capture an interaction as a structured evaluation case, score it with heuristic metrics or LLM-as-judge, aggregate into a single result, then map scores and data sensitivity to a risk tier that can gate a release. Metrics cover RAG grounding, agent trajectories, and safety. Existing model calls can be wrapped, traced, redacted, and evaluated in the background, with no rewrite of the invocation code. Tightrope is the contract-testing cousin of the same verify-before-trust idea. Nobody should both write and merge is the policy cousin of eval gating.
Scoring criteria
Structured rubrics and fixtures replace vibe checks on agent demos.
Eval cases, rubrics, and scorers for Agentic Eval are Python.
Eval fixtures and scorers are a shared path teams can reuse without reinventing rubrics.
Captured trajectories become eval cases; scoring stays independent of the agent under test.
Independent scoring keeps agent demos from self-grading their own trajectories.
Eval corpora and scorers assume adversarial or mistaken agent output.
Operators run eval cases and rubrics from the CLI against captured trajectories.
Agentic Evaluation SDK runs foundation models through Bedrock with explicit model and eval hooks.
Agentic Evaluation SDK grounds answers in retrieved context and evaluates whether retrieval helped.
CopilotJudge drives LLM-as-judge calls through Copilot SDK sessions.
OpenAIJudge is one of the ensemble judges for agent trajectory evaluation.