Comparing models

Before you switch models or ship a new prompt, run your existing evals against both versions and compare the results eval by eval. Run configs do this without changing any eval code.

1. Read the model from the config

Your eval, or its target, reads the settings from ctx.config instead of hard-coding them:

from ezvals import eval, EvalContext

async def run_agent(ctx: EvalContext):
    ctx.output = await support_agent(
        ctx.input,
        model=ctx.config.get("model", "gpt-4.1"),
        temperature=ctx.config.get("temperature", 0),
    )

@eval(input="I want a refund", target=run_agent, evaluators=[policy_judge])
async def test_refund(ctx: EvalContext):
    assert "refund" in ctx.output.lower()

import assert from "node:assert";
import { evaluate, type EvalContext } from "ezvals";

async function runAgent(ctx: EvalContext) {
  ctx.output = await supportAgent(String(ctx.input), {
    model: String(ctx.config.model ?? "gpt-4.1"),
    temperature: Number(ctx.config.temperature ?? 0),
  });
}

evaluate("test_refund", { input: "I want a refund", target: runAgent, evaluators: [policyJudge] }, (ctx) => {
  assert(String(ctx.output).toLowerCase().includes("refund"));
});

2. Define a config per model

ezvals.json

{
  "configs": {
    "gpt-4.1": { "model": "gpt-4.1", "temperature": 0 },
    "sonnet": { "model": "claude-sonnet-4-5", "temperature": 0 }
  }
}

A config can hold anything your code reads: a model, a prompt version, a retrieval setting.

3. Run each config in one session

ezvals run evals/ --session model-upgrade --config gpt-4.1
ezvals run evals/ --session model-upgrade --config sonnet

Each run is named after its config. In the web UI you can instead choose the config before starting each run.

4. Compare

ezvals serve evals/ --session model-upgrade --compare-runs gpt-4.1,sonnet

The comparison view shows the pass rate, each score and latency per run, and each eval’s outputs side by side. Look at the evals whose outcome changed, not only at the totals. To do the same from the terminal, query the session.

Models are nondeterministic, so a difference of one or two evals may be noise. Add --trials 3 to both runs and compare pass^k to see which model is reliably better.

Alternative: models as cases

To get every model in a single run, put the model in each case instead. A case’s input replaces the eval’s input rather than merging with it, so put the whole input, prompt included, in every case, and have the target read the model from ctx.input:

@eval(
    target=run_agent,
    cases=[
        {"id": model, "input": {"prompt": "I want a refund", "model": model}}
        for model in ["gpt-4.1", "claude-sonnet-4-5"]
    ],
)
async def test_refund(ctx: EvalContext):
    assert "refund" in ctx.output.lower()

evaluate("test_refund", {
  target: runAgent,
  cases: ["gpt-4.1", "claude-sonnet-4-5"].map((model) => ({
    id: model,
    input: { prompt: "I want a refund", model },
  })),
}, (ctx) => {
  assert(String(ctx.output).toLowerCase().includes("refund"));
});

This is handy for a quick check. Run configs scale better: every eval runs against every model without editing each eval, and the comparison view lines up the runs for you.

Get started

Writing evals

Reviewing results

Recipes

Comparing models

1. Read the model from the config

2. Define a config per model

3. Run each config in one session

4. Compare

Alternative: models as cases