Evaluation Patterns

This guide covers proven patterns for structuring your evaluations.

Pattern 1: Simple Assertion

The most straightforward pattern—use assertions like pytest:

@eval(input="What is 2 + 2?", dataset="demo")
async def test_simple(ctx: EvalContext):
    ctx.output = await my_agent(ctx.input)
    assert ctx.output == "4", f"Expected 4, got {ctx.output}"

Use when: Most evaluations. This is the default approach.

Pattern 2: Pre-populated Decorator

Configure inputs in the decorator for cleaner code:

@eval(input="What is the capital of France?", reference="Paris", dataset="geography")
async def test_capital(ctx: EvalContext):
    ctx.output = await my_agent(ctx.input)
    assert ctx.output.lower() == ctx.reference.lower()

Use when: Clean separation of test data from logic.

Pattern 3: Target Hook

Separate agent invocation from scoring:

async def call_agent(ctx: EvalContext):
    """Reusable agent caller."""
    ctx.output = await my_agent(ctx.input)

@eval(input="What's the weather like?", target=call_agent, dataset="qa")
async def test_weather(ctx: EvalContext):
    # ctx.output already populated by target
    assert "weather" in ctx.output.lower(), "Should mention weather"

@eval(input="Tell me a joke", target=call_agent, dataset="creative")
async def test_humor(ctx: EvalContext):
    # Same agent, different assertions
    assert len(ctx.output) > 20, "Response too short"

Use when: Multiple tests with same agent, different assertions.

Pattern 4: Multiple Results

Return multiple results from one function:

from ezvals import eval, EvalResult

@eval(dataset="batch")
def test_batch_prompts():
    prompts = [
        ("Hello", "greeting"),
        ("Goodbye", "farewell"),
        ("Help me", "support"),
    ]

    results = []
    for prompt, expected_intent in prompts:
        output = my_agent(prompt)
        detected = detect_intent(output)

        results.append(EvalResult(
            input=prompt,
            output=output,
            reference=expected_intent,
            scores=[{
                "key": "intent",
                "passed": detected == expected_intent,
                "notes": f"Detected: {detected}"
            }]
        ))

    return results

Use when: Dynamic test case generation, external data sources.

Pattern 5: Cases with Context

Combine cases with context for clean data-driven tests:

@eval(
    dataset="sentiment",
    default_score_key="accuracy",
    cases=[
        {"input": "I love this product!", "reference": "positive"},
        {"input": "This is terrible", "reference": "negative"},
        {"input": "It's okay I guess", "reference": "neutral"},
    ],
)
async def test_sentiment(ctx: EvalContext):
    # ctx.input and ctx.reference are auto-populated (special names)
    ctx.output = await analyze_sentiment(ctx.input)
    assert ctx.output == ctx.reference

Or with custom input fields:

@eval(
    dataset="sentiment",
    default_score_key="accuracy",
    cases=[
        {"input": {"text": "I love this product!", "expected": "positive"}},
        {"input": {"text": "This is terrible", "expected": "negative"}},
    ],
)
async def test_sentiment(ctx: EvalContext):
    ctx.output = await analyze_sentiment(ctx.input["text"])
    assert ctx.output == ctx.input["expected"]

Use when: Testing many input/output pairs.

Pattern 6: File-Level Defaults

Share configuration across multiple evaluations:

# evals/customer_service.py

ezvals_defaults = {
    "labels": ["production"],
    "metadata": {"department": "support"}
}

@eval(input="I want a refund")
async def test_refund(ctx: EvalContext):
    ctx.output = await support_agent(ctx.input)
    assert "refund" in ctx.output.lower(), "Should acknowledge refund"

@eval(input="Your product broke")
async def test_complaint(ctx: EvalContext):
    ctx.output = await support_agent(ctx.input)
    assert "sorry" in ctx.output.lower() or "apologize" in ctx.output.lower(), \
        "Should express empathy"

@eval(input="Edge case query", labels=["experimental"])  # Override labels
async def test_edge_case(ctx: EvalContext):
    ...

Use when: Many evaluations share configuration.

Pattern 7: Multi-Metric Evaluation

Use store(scores=...) when you need multiple named metrics or numeric scores:

@eval(input="Explain quantum computing", dataset="qa")
async def test_comprehensive(ctx: EvalContext):
    ctx.output = await my_agent(ctx.input)

    # Must-pass requirements as assertions
    assert ctx.output is not None, "Got no output"
    assert len(ctx.output) > 50, "Response too short"

    # Numeric/named metrics with store()
    keywords = ["quantum", "superposition", "qubit"]
    matches = sum(1 for kw in keywords if kw in ctx.output.lower())
    ctx.store(scores=[
        {"value": matches / len(keywords), "key": "coverage", "notes": f"{matches}/{len(keywords)} keywords"},
        {"passed": len(ctx.output) < 500, "key": "brevity", "notes": f"Length: {len(ctx.output)}"}
    ])

Use when: You need numeric scores or multiple independent metrics.

Pattern 8: Comparison Testing

Compare multiple approaches:

@eval(
    input="Explain recursion simply",
    dataset="model_comparison",
    cases=[
        {"input": {"model": "gpt-4"}},
        {"input": {"model": "gpt-3.5"}},
        {"input": {"model": "claude-3"}},
    ],
)
async def test_model_comparison(ctx: EvalContext):
    ctx.metadata["model"] = ctx.input["model"]  # Track which model was used
    ctx.output = await call_model(ctx.input["model"], ctx.input)

    assert len(ctx.output) > 50, "Response too short"
    assert "recursion" in ctx.output.lower(), "Should mention recursion"

Use when: Benchmarking models, comparing approaches.

Pattern 9: Error Handling

For robustness testing, you may want to catch specific errors:

@eval(input="Complex query", dataset="robustness", timeout=10.0)
async def test_with_error_handling(ctx: EvalContext):
    try:
        ctx.output = await my_agent(ctx.input)
    except TimeoutError:
        ctx.output = "TIMEOUT"
        assert False, "Agent timed out"
    except Exception as e:
        ctx.output = f"ERROR: {e}"
        assert False, f"Agent raised exception: {e}"

    # If we get here, agent succeeded
    assert len(ctx.output) > 0, "Empty response"

Use when: Testing reliability, edge cases.

Anti-Pattern: Forgetting to Set Output

Always set ctx.output before assertions so results are recorded for debugging.

# BAD: No output recorded
@eval(dataset="demo")
async def test_bad(ctx: EvalContext):
    ctx.input = "test"
    result = await my_agent(ctx.input)
    assert result  # Output not recorded for debugging!

# GOOD: Always set output before assertions
@eval(dataset="demo")
async def test_good(ctx: EvalContext):
    ctx.input = "test"
    ctx.output = await my_agent(ctx.input)
    assert ctx.output  # Now output is preserved

Getting Started

Examples

Core Concepts

Guides

API Reference

Resources

Evaluation Patterns

Pattern 1: Simple Assertion

Pattern 2: Pre-populated Decorator

Pattern 3: Target Hook

Pattern 4: Multiple Results

Pattern 5: Cases with Context

Pattern 6: File-Level Defaults

Pattern 7: Multi-Metric Evaluation

Pattern 8: Comparison Testing

Pattern 9: Error Handling

Anti-Pattern: Forgetting to Set Output

​Pattern 1: Simple Assertion

​Pattern 2: Pre-populated Decorator

​Pattern 3: Target Hook

​Pattern 4: Multiple Results

​Pattern 5: Cases with Context

​Pattern 6: File-Level Defaults

​Pattern 7: Multi-Metric Evaluation

​Pattern 8: Comparison Testing

​Pattern 9: Error Handling

​Anti-Pattern: Forgetting to Set Output

Pattern 1: Simple Assertion

Pattern 2: Pre-populated Decorator

Pattern 3: Target Hook

Pattern 4: Multiple Results

Pattern 5: Cases with Context

Pattern 6: File-Level Defaults

Pattern 7: Multi-Metric Evaluation

Pattern 8: Comparison Testing

Pattern 9: Error Handling

Anti-Pattern: Forgetting to Set Output