> ## Documentation Index
> Fetch the complete documentation index at: https://ezvals.com/llms.txt
> Use this file to discover all available pages before exploring further.

# CLI Reference

> Command-line interface for running evaluations

## Commands

EZVals has three main commands:

* `ezvals serve` - Start the web UI to browse and run evaluations interactively
* `ezvals run` - Run evaluations headlessly (for CI/CD pipelines)
* `ezvals export` - Export a run to various formats (JSON, CSV, Markdown)

## Programmatic Invocation

You can call the SDK equivalent of `ezvals run` from Python:

```python theme={null}
from ezvals import run

result = run(path="examples", concurrency=4, session="adhoc")
print(result["saved_path"])
```

## ezvals serve

Start the web UI to discover and run evaluations interactively.

```bash theme={null}
ezvals serve PATH [OPTIONS]
```

Where `PATH` can be:

* A directory: `ezvals serve evals/`
* A file: `ezvals serve evals/customer_service.py`
* A specific function: `ezvals serve evals.py::test_refund`
* A run JSON file: `ezvals serve .ezvals/sessions/default/run_123.json`

The UI opens automatically in your browser by default. Use `--no-open` to disable browser launch. Evaluations are discovered and displayed but not run until you click the Run button.

### Loading Previous Runs

You can load a previous run by passing the JSON file path directly:

```bash theme={null}
ezvals serve .ezvals/sessions/default/sleek-wolf_1705312200.json
```

This opens the UI with that run's results loaded. If the original eval source file still exists, you can rerun evaluations. If the source file was moved or deleted, the UI works in view-only mode.

### Options

<ParamField path="-d, --dataset" type="string" default="all">
  Filter evaluations by dataset.
</ParamField>

```bash theme={null}
ezvals serve evals/ --dataset customer_service
```

<ParamField path="-l, --label" type="string" default="all">
  Filter evaluations by label. Can be specified multiple times.
</ParamField>

```bash theme={null}
ezvals serve evals/ --label production -l critical
```

<ParamField path="--results-dir" type="string" default=".ezvals/sessions">
  Directory for JSON results storage.
</ParamField>

```bash theme={null}
ezvals serve evals/ --results-dir ./my-results
```

<ParamField path="--port" type="integer" default="8000">
  Port for web UI server.
</ParamField>

```bash theme={null}
ezvals serve evals/ --port 3000
```

<ParamField path="--session" type="string">
  Name for this evaluation session. Groups related runs together.
</ParamField>

```bash theme={null}
ezvals serve evals/ --session model-comparison
```

<ParamField path="--run-name" type="string">
  Open an existing run in the current session by run name. If not found, this becomes the pending run name for the next run.
</ParamField>

```bash theme={null}
ezvals serve evals/ --session model-comparison --run-name baseline
```

<ParamField path="--compare-runs" type="string">
  Comma-separated run names (2-4) to pre-load comparison mode at startup. These are resolved to `compare_run_id` query params in the opened URL.
</ParamField>

```bash theme={null}
ezvals serve evals/ --session model-comparison --compare-runs baseline,improved
```

<ParamField path="--search" type="string">
  Initial search text applied at startup.
</ParamField>

```bash theme={null}
ezvals serve evals/ --search auth
```

<ParamField path="--has-error/--no-has-error" type="boolean">
  Initial error filter for startup state.
</ParamField>

<ParamField path="--has-url/--no-has-url" type="boolean">
  Initial trace URL filter for startup state.
</ParamField>

<ParamField path="--has-messages/--no-has-messages" type="boolean">
  Initial trace messages filter for startup state.
</ParamField>

<ParamField path="--annotation" type="choice" default="any">
  Initial annotation filter (`any`, `yes`, or `no`).
</ParamField>

```bash theme={null}
ezvals serve evals/ --annotation yes --has-error
```

<ParamField path="--run" type="flag">
  Automatically run all evaluations on startup. Same as clicking the Run button immediately.
</ParamField>

```bash theme={null}
ezvals serve evals/ --run
```

<ParamField path="--open/--no-open" type="boolean" default="open">
  Control whether `ezvals serve` automatically opens your browser.
</ParamField>

```bash theme={null}
ezvals serve evals/ --no-open
```

## ezvals run

Run evaluations headlessly. Outputs minimal text by default (optimized for LLM agents). Use `--visual` for rich table output.

```bash theme={null}
ezvals run PATH [OPTIONS]
```

Where `PATH` can be:

* A directory: `ezvals run evals/`
* A file: `ezvals run evals/customer_service.py`
* A specific function: `ezvals run evals.py::test_refund`
* A case variant: `ezvals run evals.py::test_math[2-3-5]`

### Filtering Options

<ParamField path="-d, --dataset" type="string" default="all">
  Filter evaluations by dataset. Can be specified multiple times.
</ParamField>

```bash theme={null}
ezvals run evals/ --dataset customer_service
ezvals run evals/ -d customer_service -d technical_support
```

<ParamField path="-l, --label" type="string" default="all">
  Filter evaluations by label. Can be specified multiple times.
</ParamField>

```bash theme={null}
ezvals run evals/ --label production
ezvals run evals/ -l production -l critical
```

<ParamField path="--limit" type="integer">
  Limit the number of evaluations to run.
</ParamField>

```bash theme={null}
ezvals run evals/ --limit 10
```

### Execution Options

<ParamField path="-c, --concurrency" type="integer" default="1">
  Number of concurrent evaluations. `1` means sequential execution.
</ParamField>

```bash theme={null}
# Run 4 evaluations in parallel
ezvals run evals/ --concurrency 4
ezvals run evals/ -c 4
```

<ParamField path="--timeout" type="float">
  Global timeout in seconds for all evaluations.
</ParamField>

```bash theme={null}
ezvals run evals/ --timeout 30.0
```

### Output Options

<ParamField path="-v, --verbose" type="flag">
  Show stdout from eval functions (print statements, logs).
</ParamField>

```bash theme={null}
ezvals run evals/ --verbose
ezvals run evals/ -v
```

<ParamField path="--visual" type="flag">
  Show rich progress dots, results table, and summary. Without this flag, output is minimal.
</ParamField>

```bash theme={null}
ezvals run evals/ --visual
```

<ParamField path="-o, --output" type="string">
  Override the default results path. When specified, results are saved only to this path (not to `.ezvals/runs/`).
</ParamField>

```bash theme={null}
ezvals run evals/ --output results.json
ezvals run evals/ -o results.json
```

<ParamField path="--no-save" type="flag">
  Skip saving results to file. Outputs JSON to stdout instead.
</ParamField>

```bash theme={null}
# Get results as JSON without writing to disk
ezvals run evals/ --no-save | jq '.passed'
```

### Session Options

<ParamField path="--session" type="string">
  Name for this evaluation session. Groups related runs together.
</ParamField>

```bash theme={null}
ezvals run evals/ --session model-comparison
```

<ParamField path="--run-name" type="string">
  Name for this specific run. Used as file prefix.
</ParamField>

```bash theme={null}
ezvals run evals/ --session model-comparison --run-name gpt4-baseline
```

## ezvals export

Export a run file to various formats. Useful for sharing results, generating reports, or integrating with other tools.

```bash theme={null}
ezvals export RUN_PATH [OPTIONS]
```

Where `RUN_PATH` is the path to a run JSON file (e.g., `.ezvals/sessions/default/run_123.json`).

### Options

<ParamField path="-f, --format" type="choice" default="json">
  Export format: `json`, `csv`, or `md`.
</ParamField>

```bash theme={null}
ezvals export run.json -f md
ezvals export run.json -f csv
```

<ParamField path="-o, --output" type="string">
  Output file path. Defaults to `{run_name}.{format}`.
</ParamField>

```bash theme={null}
ezvals export run.json -f md -o report.md
```

### Export Formats

| Format | Description                                      |
| ------ | ------------------------------------------------ |
| `json` | Copy the raw JSON file                           |
| `csv`  | Flat CSV with all results                        |
| `md`   | Markdown with ASCII bar charts and results table |

### Examples

```bash theme={null}
# Export to Markdown
ezvals export .ezvals/sessions/default/run_123.json -f md

# Export to CSV
ezvals export run.json -f csv -o results.csv
```

## Examples

### Start the Web UI

```bash theme={null}
# Discover evals in a directory and open the UI
ezvals serve evals/

# Start UI on a custom port
ezvals serve evals/ --port 8080

# Filter what's shown in the UI
ezvals serve evals/ --dataset qa --label production

# Load a previous run to view or continue
ezvals serve .ezvals/sessions/default/sleek-wolf_1705312200.json
```

### Run All Evaluations

```bash theme={null}
ezvals run evals/
```

### Run Specific File

```bash theme={null}
ezvals run evals/customer_service.py
```

### Run Specific Function

```bash theme={null}
ezvals run evals/customer_service.py::test_refund
```

### Run Parametrized Variant

```bash theme={null}
ezvals run evals/math.py::test_addition[2-3-5]
```

### Filter by Dataset and Label

```bash theme={null}
ezvals run evals/ --dataset qa --label production
```

### Run with Concurrency and Timeout

```bash theme={null}
ezvals run evals/ -c 8 --timeout 60.0
```

### Export Results

```bash theme={null}
# Results auto-save to .ezvals/runs/ by default
ezvals run evals/

# Override output path
ezvals run evals/ -o results.json
```

### Verbose Debug Run

```bash theme={null}
# Show eval stdout and rich output
ezvals run evals/ -v --visual --limit 5
```

### Production CI Pipeline

```bash theme={null}
# Minimal output for LLM agents/CI
ezvals run evals/ -c 16 --timeout 120
```

### Session Tracking

```bash theme={null}
# Group runs under a session
ezvals run evals/ --session model-comparison --run-name baseline

# Continue the session with another run
ezvals run evals/ --session model-comparison --run-name improved
```

## Configuration File

EZVals supports a `ezvals.json` config file for persisting default CLI options. The file is auto-generated in your project root on first run.

### Default Config

```json theme={null}
{
  "concurrency": 1,
  "results_dir": ".ezvals/runs"
}
```

### Supported Options

| Option        | Type    | Description                      | Used by |
| ------------- | ------- | -------------------------------- | ------- |
| `concurrency` | integer | Number of concurrent evaluations | `run`   |
| `timeout`     | float   | Global timeout in seconds        | `run`   |
| `verbose`     | boolean | Show stdout from eval functions  | `run`   |
| `results_dir` | string  | Directory for results storage    | `serve` |
| `port`        | integer | Web UI server port               | `serve` |

### Precedence

CLI flags always override config values:

```bash theme={null}
# Config has concurrency: 1, but this uses 4
ezvals run evals/ -c 4
```

### Editing via UI

Click the settings icon in the web UI header to view and edit config values. Changes are saved to `ezvals.json`.

## Exit Codes

| Code     | Meaning                                             |
| -------- | --------------------------------------------------- |
| 0        | Evaluations completed (regardless of pass/fail)     |
| Non-zero | Error during execution (bad path, exceptions, etc.) |

<Note>
  The CLI does not currently set non-zero exit codes for failed evaluations—only for execution errors. Check the JSON output or summary for pass/fail status.
</Note>

## Environment Variables

| Variable             | Description                |
| -------------------- | -------------------------- |
| `EZVALS_CONCURRENCY` | Default concurrency level  |
| `EZVALS_TIMEOUT`     | Default timeout in seconds |

## Output Format

### Minimal Output (Default)

By default, `ezvals run` outputs minimal text optimized for LLM agents and CI pipelines:

```
Running...
Results saved to .ezvals/runs/swift-falcon_2024-01-15T10-30-00Z.json
```

### Visual Output (--visual)

Use `--visual` for rich progress dots, results table, and summary:

```
Running...
customer_service.py ..F

┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃                     customer_service                           ┃
┣━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━┫
┃ Name                ┃ Status   ┃ Score    ┃ Latency           ┃
┡━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━┩
│ test_refund         │ ✓ passed │ 1.0      │ 0.23s             │
│ test_complaint      │ ✗ failed │ 0.0      │ 0.45s             │
└─────────────────────┴──────────┴──────────┴───────────────────┘

Summary: 1/2 passed (50.0%)
```

### JSON File Output

Results are always saved as JSON to `.ezvals/runs/` (or custom path via `-o`):

```json theme={null}
{
  "run_id": "2024-01-15T10-30-00Z",
  "total": 2,
  "passed": 1,
  "failed": 1,
  "results": [...]
}
```