> ## Documentation Index
> Fetch the complete documentation index at: https://ezvals.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Scoring

> Score evaluations using assertions, just like pytest

EZVals uses assertions as the primary way to score evaluations. If you've written pytest tests, you already know how to score in EZVals.

## Assertions

Use Python's `assert` statement to validate your outputs:

```python theme={null}
@eval(input="What is the capital of France?", dataset="qa")
async def test_answer_quality(ctx: EvalContext):
    ctx.output = await my_agent(ctx.input)
    assert "paris" in ctx.output.lower(), "Should mention Paris"
```

When assertions pass, your eval passes. When they fail, the assertion message becomes the failure reason in your results.

### Multiple Assertions

Chain assertions to check multiple conditions:

```python theme={null}
@eval(input="I want a refund", dataset="customer_service")
async def test_refund_response(ctx: EvalContext):
    ctx.output = await support_agent(ctx.input)

    assert len(ctx.output) > 20, "Response too short"
    assert "refund" in ctx.output.lower(), "Should acknowledge refund request"
    assert "sorry" in ctx.output.lower() or "apologize" in ctx.output.lower(), \
        "Should express empathy"
```

### Comparing to Reference

Use `ctx.reference` for expected output comparisons:

```python theme={null}
@eval(input="2 + 2", reference="4", dataset="math")
async def test_arithmetic(ctx: EvalContext):
    ctx.output = await calculator(ctx.input)
    assert ctx.output == ctx.reference
```

### With Cases

Assertions work naturally with case-based tests:

```python theme={null}
@eval(
    dataset="sentiment",
    cases=[
        {"input": "I love this!", "reference": "positive"},
        {"input": "This is terrible", "reference": "negative"},
        {"input": "It's okay", "reference": "neutral"},
    ],
)
async def test_sentiment(ctx: EvalContext):
    ctx.output = await classify_sentiment(ctx.input)
    assert ctx.output == ctx.reference
```

## How Assertions Become Scores

When an assertion fails:

1. The eval doesn't crash - it's caught gracefully
2. A failing score is created with the assertion message as notes
3. Your `ctx.input` and `ctx.output` are preserved for debugging

```python theme={null}
# If this assertion fails:
assert ctx.output == "Paris", "Wrong capital"

# It creates this score:
{
    "key": "pass",
    "passed": False,
    "notes": "Wrong capital"
}
```

When all assertions pass, a passing score is automatically added.

***

## store() for Explicit Scoring

Use `store(scores=...)` when you need more control than assertions provide:

* **Numeric scores** (confidence values, similarity scores)
* **Multiple named metrics** per evaluation
* **Non-binary scoring** (partial credit)

### Numeric Scores

```python theme={null}
@eval(input="Classify this text", dataset="classification")
async def test_with_confidence(ctx: EvalContext):
    result = await classifier(ctx.input)
    ctx.store(output=result["label"], scores={"value": result["confidence"], "notes": "Model confidence"})
```

### Multiple Named Metrics

```python theme={null}
@eval(input="Explain quantum computing", dataset="qa")
async def test_comprehensive(ctx: EvalContext):
    ctx.output = await my_agent(ctx.input)

    ctx.store(scores=[
        {"passed": "quantum" in ctx.output.lower(), "key": "relevance", "notes": "Mentions quantum"},
        {"passed": len(ctx.output) < 500, "key": "brevity", "notes": f"Length: {len(ctx.output)}"},
        {"value": calculate_similarity(ctx.output, reference), "key": "similarity", "notes": "Semantic similarity"}
    ])
```

### Combining Assertions and store()

You can use both in the same eval:

```python theme={null}
@eval(input="Generate a summary", dataset="demo")
async def test_mixed(ctx: EvalContext):
    ctx.output = await summarizer(ctx.input)

    # Must-pass requirements as assertions
    assert ctx.output is not None, "Got no output"
    assert len(ctx.output) > 10, "Output too short"

    # Numeric quality metric
    ctx.store(scores={"value": quality_score(ctx.output), "key": "quality", "notes": "Quality score"})
```

***

## Evaluators (Post-Processing)

Evaluators are functions that run after the eval completes and add additional scores:

```python theme={null}
def check_format(result):
    """Check if output is valid JSON."""
    try:
        json.loads(result.output)
        return {"key": "format", "passed": True}
    except:
        return {"key": "format", "passed": False, "notes": "Invalid JSON"}

@eval(input="Get user data", dataset="api", evaluators=[check_format])
async def test_json_response(ctx: EvalContext):
    ctx.output = await api_call(ctx.input)
    assert "user" in ctx.output, "Should contain user data"
```

Evaluators are useful for reusable checks across many evals.

***

## Score Structure Reference

Each score has:

```python theme={null}
{
    "key": "metric_name",    # Identifier (default: "pass")
    "value": 0.95,           # Optional: numeric score (0-1 range)
    "passed": True,          # Optional: boolean pass/fail
    "notes": "Explanation"   # Optional: human-readable notes
}
```

<Note>
  Every score must have at least one of `value` or `passed`.
</Note>

## When to Use What

| Scenario                               | Use                                |
| -------------------------------------- | ---------------------------------- |
| Simple pass/fail checks                | `assert`                           |
| Multiple conditions that must all pass | Multiple `assert` statements       |
| Numeric confidence/similarity scores   | `ctx.store(scores=0.85)`           |
| Multiple independent metrics           | `ctx.store(scores=[{...}, {...}])` |
| Reusable checks across evals           | Evaluators                         |