Introduction

EZVals is an eval framework for LLM apps and agents, with SDKs for Python and TypeScript. It’s designed to be driven by a coding agent: your agent writes the evals, runs them from the CLI, reads the results and iterates, while you review the same runs in a local web UI. An eval is an ordinary function: give it an input, call your agent, check the output. Everything else (cases, scores, repeated trials, run history) builds on that.

from ezvals import eval, EvalContext

@eval(input="I want a refund", dataset="support")
async def test_refund(ctx: EvalContext):
    ctx.output = await support_agent(ctx.input)
    assert "refund" in ctx.output.lower(), "Should acknowledge the refund"

import assert from "node:assert";
import { evaluate } from "ezvals";

evaluate("test_refund", { input: "I want a refund", dataset: "support" }, async (ctx) => {
  ctx.output = await supportAgent(String(ctx.input));
  assert(String(ctx.output).toLowerCase().includes("refund"), "Should acknowledge the refund");
});

Why EZVals

Agent-first. A CLI with JSON output, SQL over every saved run, and a bundled skill that teaches your coding agent how to design evals and graders.
Built for nondeterminism. Repeated trials report pass@k and pass^k, and regrading re-scores stored outputs when you change a grader, without paying for new agent runs.
Local. Eval code, run files and the web UI all live in your repo and on your machine. No account, no hosted service.
One tool, two languages. Python and TypeScript evals can sit in the same directory and run as one run.

Quickstart

Install, write a first eval, run it and open the UI.

How EZVals works

Evals, cases, scores, runs and sessions in one page.

Using with coding agents

Install the skill and let your agent do the work.

Get started

Writing evals

Reviewing results

Recipes

Why EZVals

Quickstart

How EZVals works

Using with coding agents

​Why EZVals

Quickstart

How EZVals works

Using with coding agents

Why EZVals