Skip to main content

​
0.2.0 - 2026-09-27

  • Changed: ezvals is now a single Go host (CLI, HTTP API, run storage and the web UI) that runs your evals in language workers. The Python SDK has no dependencies, and a TypeScript SDK (.eval.ts files) produces the same results, checked by a shared conformance suite.
  • Changed: Runs are stored as append-only event logs, so a run interrupted with Ctrl+C keeps its finished results. Legacy .json runs are read in place and never converted or deleted.
  • Added: Repeated trials (pass@k and pass^k), regrading of stored outputs for evals with a target, and SQL queries over runs with ezvals query.
  • Removed: OpenTelemetry span capture. trace_url and message transcripts still link and render.
  • Changed: The web UI is rebuilt as a workbench. A runs sidebar lists the session’s runs with their pass rates and replaces the run dropdown. The open run sits in its own panel with Compare, Regrade and Run, and Regrade and Compare only show when they can be used.
  • Added: A review panel beside the results table. Click a row to read the result without leaving the run, step with ↑/↓, press Enter for the full page and Esc to close. The URL keeps the open result.
  • Changed: The summary leads with the pass rate, a trend of the session’s recent runs and an outcome bar. Per-key stats appear only when a run has more than one score key.
  • Changed: The results table shows an outcome icon per row, centres its cells, keeps column headers pinned when scrolling, and shows errors as a one-line summary.
  • Changed: The result page steps between results in place, without reloading, and shows why a result errored in a strip under the header.
  • Changed: A documented design system (Geist type, semantic colours for light and dark, shared controls), shown in Storybook under Design System. Every screen meets WCAG 2.1 AA, and the story tests enforce it.
  • Changed: SDK and CLI output leads with failures, trims tracebacks to your own code, and suggests the closest option for a mistyped one.
  • Fixed: Saving settings keeps your project’s own keys and their order in ezvals.json.
  • Fixed: Switching runs no longer re-imports your eval code (about 1s to about 50ms on a heavy project).
  • Added: npm install ezvals for TypeScript projects. The host binary comes as a per-platform optional dependency (ezvals-darwin-arm64 and so on), so an install downloads only your machine’s binary (about 10 MB).

​
0.1.10 - 2026-02-24

  • Changed: Run picker dropdown now uses a listbox-style UI with inline rename, copy-to-clipboard feedback, keyboard navigation, and selection indicators replacing the old FloatingMenu.
  • Changed: Replaced all alert() calls with inline toast notifications that auto-dismiss after 4 seconds.
  • Fixed: Hardcoded dark-mode zinc colors in filter panel, more menu, columns menu, and comparison chips now use theme-aware CSS variables, fixing broken light-mode rendering.
  • Changed: Table rows are now clickable to open the detail view (excluding checkboxes and annotation indicators).
  • Changed: Column resize handles now show a blue highlight on hover for discoverability.
  • Changed: “New run” and “Compare” action buttons in the stats panel now have visible text labels.
  • Changed: Columns and Export controls promoted from the overflow menu to the main header toolbar; overflow menu now only contains “Reload Server”.
  • Added: Filter panel three-state toggle legend (Include/Exclude/Any) with color indicators and title attributes.
  • Changed: Detail page header height reduced from 120px to 80px to maximize content area.
  • Changed: Footer removed to reclaim vertical space.
  • Added: Empty state message (“No results yet”) when the results table has no rows.
  • Added: Dropdown menus now auto-flip above the anchor when they would overflow the viewport bottom.
  • Tests: Updated E2E tests for toast notifications, header toolbar layout, and clickable row blur behavior.
  • Fixed: Comparison-mode output columns now render per-run output content correctly for URL-driven compare sessions, even when the active run is not one of the selected compared runs.
  • Tests: Added E2E regression coverage for URL-only compare_run_id startup to verify compared-run output text appears and unrelated active-run rows are excluded.
  • Changed: results_dir in ezvals.json is now treated as a base directory, with runs always stored under {results_dir}/.ezvals/sessions.
  • Changed: Default config now sets "results_dir": ".", preserving default storage at ./.ezvals/sessions while removing full sessions-path config complexity.
  • Tests: Added CLI unit coverage to verify run/save and rename behavior when results_dir is configured as a base directory.
  • Changed: Detail view now restores pane layout from session state so resized horizontal and vertical splits carry across result-to-result navigation.
  • Fixed: Reference pane height no longer snaps back during detail navigation after a user resize.
  • Tests: Added E2E regression coverage for detail pane persistence across eval navigation, including a high reference-pane resize case.
  • Changed: Single-run average latency is now shown in the Time table header tooltip as (Avg: <value>s) instead of occupying space in the stats panel/chart.
  • Tests: Updated E2E coverage to assert average latency appears in the Time header tooltip and remains positive after a run completes.

​
0.1.9 - 2026-02-23

  • Added: Skill sandbox Q&A eval suite — 25 cases testing whether the EZVals skill correctly teaches agents CLI usage, session management, export, and configuration.
  • Added: Shared eval infrastructure (_common.py) with robust JSON extraction and retry logic for LLM judge calls via Claude Code headless mode.
  • Fixed: Skill docs had incorrect results storage paths (.ezvals/runs/ → .ezvals/sessions/), missing PNG export format, unimplemented keyboard shortcuts (r, e, f), and wrong jq field name in CLI docs.
  • Changed: Skill docs now emphasize that ezvals serve can run evals (not view-only), with prominent callouts in Quick Start and Two Modes sections.
  • Changed: Skill grader docs now recommend LLM judges over code-based checks for evaluating natural language responses, with guidance on holistic pass/fail for subjective evals.
  • Added: Detail view now supports URL-driven comparison mode via repeated compare_run_id query params, enabling direct shareable links like /runs/{run_id}/results/{index}?compare_run_id=...&compare_run_id=....
  • Changed: Comparison-table row links now preserve compare_run_id params when opening detail pages, so compare context survives new tabs, refresh, and browser navigation.
  • Fixed: Detail compare URLs with a single compare_run_id now auto-include the current detail run as base context.
  • Tests: Added E2E coverage for comparison row deep-link query params and detail compare hydration from both repeated and single compare_run_id query forms.
  • Changed: Comparison-mode detail view now uses a dedicated layout that prioritizes run output cards on top, with input/reference context moved to a supporting lower pane.
  • Fixed: Comparison detail no longer reuses single-run controls; duplicate eval title rows, rerun action, and the single-run sidebar are now hidden in comparison mode.
  • Added: Comparison detail run cards now include per-run score/latency/annotation context and an Open detail action that jumps directly to that run’s single-result view.
  • Tests: Expanded comparison-detail E2E coverage for layout behavior, hidden controls, per-run badges/annotations, deep-link navigation, and draggable split handles.
  • Added: Comparison-mode table cells now support the same hover preview popover behavior as single-run mode for input, reference, output, error, and scores content.
  • Tests: Added E2E coverage for comparison-table hover previews across input, reference, output, error, and scores.
  • Fixed: Detail view now keeps the Trace Data section collapsed by default, so verbose trace payloads do not push higher-signal sidebar content out of view.
  • Tests: Updated full-scenario E2E coverage to assert Trace Data starts collapsed and expands only when toggled open.
  • Added: Run configs — named configuration profiles in ezvals.json (configs key) selectable via --config flag on both run and serve, with the config dict available as ctx.config in eval code. Auto-defaults run_name to the config name for easy side-by-side comparison.
  • Added: Settings modal now includes a Run Config dropdown when configs are defined in ezvals.json.
  • Tests: Added unit and integration tests for resolve_run_config(), ctx.config, --config CLI flag, and config auto-naming.

​
0.1.8 - 2026-02-20

  • Changed: Removed --visual from ezvals run; headless runs now use the minimal output flow plus JSON results output (--output or --no-save).
  • Changed: Simplified ezvals run internals by deleting rich progress/table reporting code that was only used by --visual.
  • Tests: Updated CLI unit coverage to validate run behavior via JSON output files and removed obsolete visual-progress tests.
  • Changed: ezvals run now uses selector-first path syntax for targeted runs (for example, ezvals run evals.py::test_a,test_b) instead of a dedicated extra flag.
  • Added: Case-target selectors now support @case_id shorthand (for example, ezvals run evals.py::test_math@low,test_math@high) as an intuitive alias for bracketed case variants.
  • Changed: Evals skill docs now guide agents to use path selectors (::...) and @case_id for rerunning specific failures without label workarounds.
  • Tests: Added unit and CLI coverage for comma-separated selector lists and function@case_id selection behavior.
  • Changed: Dashboard stats/status panel is now expanded-only; compact mode and the collapse/expand toggle have been removed.
  • Fixed: Run-switch dropdown anchoring now consistently uses the expanded stats bar run selector.
  • Tests: Updated E2E coverage to validate expanded-only stats behavior and removed compact-mode toggle expectations.
  • Added: “Testing Agent Skills” use-case guide in the evals skill covering plan-only evals, headless CLI agents as targets, and CLI agent as LLM judge for evaluating skills, CLAUDE.md/AGENTS.md configs, and MCP servers.
  • Added: Settings now includes a Notifications toggle that controls run-complete alerts from the config menu.
  • Changed: Completion alerts now use a softer two-note chime and include the EZVals logo as the notification icon.
  • Changed: Notification preference is persisted as completion_notifications in ezvals.json and defaults to disabled.
  • Fixed: Notifications toggle state now saves reliably from the settings modal and persists as completion_notifications in config on submit.
  • Fixed: Browser completion notifications now trigger when an active run transitions from pending/running to done with notifications enabled.
  • Tests: Added E2E coverage for saving notification settings and triggering a completion notification when a run finishes.
  • Changed: Dashboard run controls are now a single stable Run/Stop action (no run-mode switching between rerun/new-run).
  • Added: Stats panel now includes a dedicated new-run icon beside run controls that creates a fresh auto-named run with no completed results.
  • Fixed: Creating a new run with an empty selection now preserves discovered evaluations as not_started rows instead of saving an empty 0 tests run.
  • Changed: Stats-panel run actions are now icon-first controls with tooltips, including a compact compare action in the same row.
  • Fixed: Stats-panel left-column sizing and action-row layout now stay constrained so controls no longer push/overlap chart grid content.
  • Tests: Updated E2E run-controls/stats coverage and added integration coverage for empty-selection new-run behavior.
  • Added: Cell hover preview popover on dashboard content columns (input, output, reference, error, scores) replaces row-click-to-expand with a scrollable, interactive preview on hover.
  • Fixed: Detail view output now shows a subtle animated loading state while a result is pending or running, instead of showing stale output content.
  • Tests: Added E2E coverage to verify the detail output loading state is visible during running status and stale output text remains hidden.
  • Fixed: Long score key names no longer cause the score filter select control to overflow outside the filters panel.
  • Changed: Score keys are now validated with a maximum length of 32 characters.
  • Tests: Added unit and E2E regression coverage for score key max-length validation and long-key filter layout containment.
  • Added: Column settings now include per-column search scope controls, so search can be limited to specific table columns.
  • Changed: Column visibility and search scope are now configured independently in the same Columns submenu using separate Show and Search toggles.
  • Tests: Added E2E coverage to verify search-column scoping and that hiding a column does not disable searching that column unless Search is toggled off.
  • Added: Result PATCH edits for scores, annotation, and annotations now append result.correction_history entries with field, before, after, and UTC timestamp.
  • Changed: Evals skill guidance now tells agents to inspect result.correction_history whenever users mention notes, corrections, or manual edits.
  • Added: Evals skill now includes a Phoenix Arize -> EZVals migration quick map that translates common Arize eval patterns into EZVals concepts and workflows.
  • Tests: Added integration coverage for correction history persistence, including guard behavior that skips duplicate entries when values are unchanged.
  • Changed: Dashboard Columns and Download actions now live in the overflow (three-dot) menu instead of dedicated header buttons.
  • Changed: Overflow submenu behavior now uses hover-open side flyouts with directional carets and improved spacing/layering for clearer navigation.
  • Tests: Updated E2E UI interaction and full-scenario coverage to open column/export controls from the overflow menu.
  • Changed: Detail view annotation editing now uses a single explicit control (the pencil icon), and the empty state shows No annotation instead of a secondary add button.
  • Fixed: The annotation edit pencil in detail view is always visible instead of appearing only on hover.
  • Added: Detail sidebar dataset values now link back to the dashboard with the current run selected and the dataset filter pre-applied.
  • Changed: In detail view, the reference pane is stacked under input in the left column, while output remains in the right column.
  • Tests: Added and updated E2E coverage for dataset-link filtering, always-visible annotation edit control, and left-column reference pane layout geometry.
  • Fixed: Not-started evaluation titles in the dashboard are now clickable links to detail pages while retaining their gray not-started styling.
  • Fixed: Detail routes now load synthesized not-started results for the active run when no run file has been saved yet, preventing pre-run “Run not found” errors.
  • Tests: Added E2E coverage for clicking a not-started title and navigating to its detail page.
  • Tests: Added integration coverage for pre-run detail page/API behavior when results are sourced from discovered functions.
  • Changed: ezvals run now supports --rename <run_id> <new_name> to rename saved runs by ID from scripts and terminal workflows.
  • Changed: Evals skill docs now document run-id rename mode under ezvals run --rename.
  • Tests: Added CLI unit coverage for ezvals run --rename help text, successful rename, and missing-run failure.
  • Fixed: Running rows in the dashboard now show a subtle spinner indicator instead of a prominent “running” text chip.
  • Tests: Added E2E coverage to verify running status uses the subtle indicator and no running text chip is rendered.
  • Fixed: Publish workflow now assembles ezvals/skills/evals/ezvals-docs/ before building the wheel, so ezvals skills add --claude installs bundled API reference docs from PyPI packages.
  • Fixed: Skill sync workflow now sources docs/setup.mdx as quickstart.mdx, matching current docs layout while preserving skill doc naming.
  • Tests: Publish workflow now verifies both wheel payload and installed-wheel behavior by running ezvals skills add --claude and asserting required ezvals-docs files.
  • Changed: EZVals logo in the web UI header now links to ezvals.com.

​
0.1.7 - 2026-02-17

  • Added: SDK run(...) API for programmatic eval execution with ezvals run option parity; CLI ezvals run now calls this shared core path.
  • Changed: SDK run(...) now defaults to built-in runtime settings without creating ezvals.json; CLI continues to honor config by explicitly opting into config loading.
  • Changed: Runner internals now share one execution path for sequential and concurrent result shaping, reducing duplicated callback/loader/result mapping code.
  • Changed: Serve startup for eval-path and JSON-path flows now uses a shared server boot helper (port/url/browser/server lifecycle) to reduce duplication.
  • Tests: Stabilized the full-rerun E2E control test by asserting new-run artifacts in storage, removing flaky UI timing dependence under pytest -n auto.
  • Added: Detail sidebar score cards now support inline editing with per-score pencil controls, Save/Cancel actions, and persistence through the existing result PATCH API.
  • Changed: Score editing in detail view is type-locked; boolean scores remain boolean and value scores remain value-only, and keyboard save shortcuts were removed in favor of explicit Save clicks.
  • Changed: Detail edit pencils for scores and annotation now appear on hover for a cleaner default sidebar presentation.
  • Tests: Added E2E coverage for detail score editing, including type-specific controls and persisted updates after page reload.
  • Fixed: ezvals skills add now installs the full bundled skill directory, including nested use-cases/ and ezvals-docs/ folders.
  • Tests: Added CLI unit assertions that skills add installs nested skill content for both canonical and .agents/ fallback installs.
  • Changed: ezvals skills add now uses explicit target flags (--agents, --claude, --codex, --cursor, --windsurf, --kiro, --roo) instead of --agents <name>.
  • Changed: ezvals skills add now fails fast when no target flag is provided.
  • Changed: When --agents is selected, .agents is always treated as canonical source (including combined targets like --agents --claude).
  • Tests: Added CLI unit coverage for required target flags and .agents canonical-priority behavior.
  • Added: Agent skill now prompts users to file GitHub issues when they mention EZVals friction, bugs, or feature ideas.
  • Changed: Docs navigation reorders examples to show RAG Agent before Granular Evals.

​
0.1.6 - 2026-02-13

  • Fixed: Dataset and label chips now truncate long text with ellipsis and expose the full value via hover tooltip in results rows, comparison rows, and filter pills.
  • Fixed: Detail sidebar dataset/label chips now truncate long values so long names do not stretch metadata layout.
  • Changed: Example eval labels now include intentionally long three-word values so chip truncation behavior is easier to verify in the UI.
  • Tests: Added E2E coverage for long dataset/label chip truncation and tooltip behavior.
  • Fixed: Dashboard table column resizing now keeps the dragged edge aligned with cursor movement by stabilizing visible header widths at drag start.
  • Fixed: Resizing a column no longer unintentionally toggles table sorting on mouse release.
  • Tests: Added E2E coverage for column-resize drag behavior (edge tracking, width change, and no-sort-on-drag regression guard).
  • Added: Dashboard now includes a slim overflow menu action to reload the EZVals server from the UI when state gets stuck or eval files change.
  • Changed: In-app reload now performs a full ezvals serve process restart with the same serve arguments and preserves the active bound port.
  • Tests: Added unit and E2E coverage for restart re-exec flow, restart API signaling, and overflow-menu restart interaction.
  • Added: Detail view now auto-detects chat-style message arrays in input, output, and reference, rendering them in a chat-like “Pretty” mode with a “Raw” JSON toggle.
  • Changed: Message normalization/rendering now lives in a reusable UI component (DataViewer) used across detail panels and the trace messages drawer.
  • Changed: examples/demo_trace_data.py::test_direct_trace_data now includes message-array examples for input, output, and reference.
  • Tests: Added E2E coverage to verify pretty/raw message rendering and toggling across input/output/reference and the messages pane.
  • Changed: Default score-key naming now uses pass/Pass in runtime fallbacks, examples, tests, and docs where the previous default used correctness/Correctness.
  • Tests: Updated integration, unit, and E2E expectations to assert the pass default score key label.
  • Changed: PNG export now uses a compact options button with a cleaner, more ergonomic configuration panel.
  • Added: PNG export supports editable title, score color tuning, comparison run rename/reorder controls, and footer toggles for test count and average latency.
  • Fixed: PNG preview updates no longer flicker while typing configuration inputs.
  • Fixed: Comparison-mode PNG exports no longer overlap footer metrics with the run key legend.
  • Tests: Added E2E coverage for configurable PNG export options, comparison run reordering in export, and preview regeneration behavior.
  • Changed: Detail view metadata now renders as a readable key-value list instead of raw JSON, with human-friendly key labels.
  • Fixed: URL metadata values in detail view are now clickable links.
  • Tests: Added E2E coverage for metadata label formatting and clickable metadata URLs in detail view.
  • Added: Detail sidebar now shows a subtle “Tools” row with unique tool names detected from trace_data.messages tool calls.
  • Changed: Detail message parsing now supports common message-array tool-call schemas (tool_calls, tool_use, tool_call, and function_call) so tool usage is extracted more reliably.
  • Tests: Added E2E coverage for mixed message-array tool-call schemas to verify tool name extraction and deduplication in the detail sidebar.
  • Added: Run controls now include Pause/Resume in addition to Stop. Pause lets in-flight evaluations finish while holding queued work; Resume continues pending evaluations from the same run.
  • Added: ezvals serve now auto-builds frontend assets when ui/src is newer than ezvals/static, reducing stale-UI surprises during local development.
  • Changed: Running-state controls are now an icon-only connected cluster (pause/play + stop) with theme-matched styling.
  • Fixed: Quick pause/resume interactions no longer spawn overlapping runner loops; resume now unpauses the active run thread instead of starting a duplicate execution path.
  • Fixed: Stopped runs now remain immediate-cancel behavior, while paused runs continue polling until in-flight rows finish so statuses update without manual refresh.
  • Tests: Added integration coverage for pause/resume API behavior and E2E coverage for pause/resume UI and run control state transitions.
  • Added: ezvals serve now supports --open/--no-open to control whether the browser opens automatically on startup.
  • Changed: CLI and web UI docs now describe browser auto-open as default behavior and document --no-open for terminal-only startup flows.
  • Tests: Added CLI unit coverage to verify --no-open is exposed in help and propagated for both eval-path and JSON-path serve flows.
  • Changed: Run metadata on EvalContext (ctx.run_id, ctx.session_name, ctx.run_name, ctx.eval_path) is now read-only after context initialization.
  • Tests: Added unit coverage to verify run metadata fields on EvalContext cannot be reassigned.
  • Added: Dashboard now keeps shareable URL query state in sync for search, filters, compare selection, and table sort (sort=<col>,<dir>,<type>).
  • Changed: Canonical comparison links now use repeated compare_run_id params without run_id in compare mode.
  • Changed: Evals skill URL-sharing guidance now shows compare-mode links using compare_run_id-only format.
  • Fixed: Launch URLs shaped like run_id=<id>&compare_run_id=<id> now reliably hydrate two-run comparison mode.
  • Tests: Added E2E coverage for single-compare query hydration, live filter/search URL sync, and sort query hydration/sync.
  • Added: Evals skill running.md now documents the per-result EvalResult output schema (results[*].result) with concrete JSON examples.
  • Added: Evals skill includes parsing recipes for common agent tasks like finding failing evals, filtering by error text, and extracting/aggregating score values.
  • Added: Up/down controls on comparison chips so runs can be reordered directly in compare mode without exiting.
  • Changed: Comparison chip order now immediately drives comparison chart bar order and comparison table column order.
  • Tests: Added E2E coverage for reordering comparison runs from chip controls.
  • Fixed: Detail view now uses viewport-aware defaults for input/output split, sidebar width, and reference panel height so first-time users do not start with broken proportions.
  • Tests: Added E2E coverage to verify fresh-session detail layout keeps sidebar width and reference panel height within sane viewport-relative bounds.
  • Fixed: Run dropdown now appears when a serve-started unsaved active run has another existing run in the same session, so headless ezvals run results are immediately switchable.
  • Fixed: Compare action now appears when there is at least one other run available in the current session, even if the active run is not yet saved.
  • Tests: Added E2E coverage for the unsaved-active + existing-session-run dropdown scenario to prevent regression.
  • Fixed: Expanded table rows no longer force all cells to top-align, preventing the visual jump when clicking rows with short content.
  • Tests: Added E2E coverage to verify row clicks keep cell vertical alignment stable for rows with non-overflowing content.
  • Fixed: ezvals serve now reuses the requested port more reliably on rapid stop/start cycles by probing availability with socket reuse enabled.
  • Tests: Added unit coverage to prevent regressions in serve port probing during quick restarts.

​
0.1.5 - 2026-02-07

  • Added: PNG export option in the export dropdown. Opens a preview modal with Save and Copy buttons. Renders stats bars, metrics, and branding on a 1200x630 canvas at 2x resolution for crisp output.
  • Added: PNG export supports comparison mode with grouped bars, outline-styled run name chips, and per-run colors.
  • Added: CI guard workflow to prevent accidentally committing generated UI artifacts.
  • Added: Wheel verification step in publish workflow to ensure UI assets are bundled.
  • Added: Agent skill guidance for sharing focused results views via direct URLs when the user already has the UI running.
  • Changed: Frontend build artifacts (ezvals/static/) are no longer tracked in git. The publish workflow builds the UI and bundles assets into the wheel at release time.
  • Changed: Actionable error message when serving from a git checkout with missing UI assets.
  • Changed: Stats bars and comparison bars now render with gradient fills, color-matched glow shadows, and a glassy highlight effect.
  • Changed: Serve startup state now uses explicit readable URL parameters and no longer uses the encoded startup payload.
  • Fixed: Comparison mode stats bars and comparison run counts now compute from the currently visible filtered rows instead of full-run aggregates.
  • Fixed: Comparison chips showed run IDs instead of run names when comparison runs were loaded from URL parameters.
  • Tests: Added E2E coverage for comparison mode to verify stats bar percentages update after applying filters.
  • Tests: Added test for missing-UI-assets error response.
  • Tests: Added coverage for startup URL hydration and explicit verification that legacy startup payload query input is ignored.

​
0.1.4 - 2026-02-05

  • Changed: Migrated UI source from JSX to non-strict TypeScript (TSX).
  • Fixed: Tailwind CSS content config to scan .ts/.tsx files after TypeScript migration.
  • Added: Sticky table header so column names stay visible while scrolling.

​
0.1.3 - 2026-01-25

  • Fixed: PyPI publish workflow failing due to duplicate files in wheel (removed redundant force-include).
  • Tests: E2E tests now use dynamic port allocation for parallel execution with pytest-xdist.

​
0.1.2 - 2026-01-24

  • Changed: Replaced @parametrize with cases= on @eval using list-of-dict cases.
  • Changed: Case dicts can override input, reference, metadata, dataset, labels, default_score_key, timeout, target, and evaluators; id controls variant naming.
  • Changed: Web UI rebuilt with React and Vite for improved maintainability. No user-facing behavior changes.
  • Removed: @parametrize decorator and tuple-based case syntax.
  • Fixed: Run button incorrectly showed “Stop” instead of “Run” on fresh page load when no evaluations were running.
  • Fixed: Inline run name editing now works when multiple runs exist (dropdown visible).
  • Fixed: Race condition in discovery module when clearing cached modules (sys.modules changed during iteration).
  • Tests: Updated E2E tests for React UI migration—fixed selectors, element IDs, and interaction patterns.
  • Tests: Updated E2E tests to expect dropdown visible when rows are selected (matching current UI behavior).

​
0.1.1 - 2026-01-03

  • Changed: Stats bar error count now always displays (shows 0 when no errors) for consistent layout.
  • Changed: Reorganized stats bar metrics into rows for better visual hierarchy.
  • Changed: Updated documentation UI screenshots with new logo.

​
0.1.0 - 2026-01-02

  • Added: Markdown export with stats visualization and filtered data support.
  • Added: Export dropdown menu in header with JSON, CSV, and Markdown formats.
  • Added: ezvals export CLI command for exporting runs to JSON, CSV, or Markdown.
  • Added: Markdown export includes ASCII progress bars with color emoji indicators.
  • Added: Filtered Markdown export respects current filters and column visibility.
  • Added: Comparison mode for viewing 2-4 runs side-by-side. Click ”+ Compare” in stats bar to select runs, view grouped bar charts with color-coded metrics, and compare outputs across runs in a unified table with per-run columns.
  • Added: GET /api/runs/{run_id}/data endpoint for fetching run data without changing the active run.
  • Added: Run dropdown selector to switch between past runs in the same session. Dropdown appears when 2+ runs exist.
  • Added: Activate run endpoint POST /api/runs/{run_id}/activate to switch the active run being viewed.
  • Added: Run metadata on EvalContext for observability tagging. Access ctx.run_id, ctx.session_name, ctx.run_name, ctx.eval_path (run-level) and ctx.function_name, ctx.dataset, ctx.labels (per-eval) inside eval functions for LangSmith/observability integration.
  • Changed: Export moved from Settings modal to dedicated dropdown in header.
  • Changed: Stats bar chart values now show stacked format—percentage prominent on top, ratio smaller below (e.g., “87%” over “54/62”).
  • Changed: Run selection persisted across page reloads within session.
  • Fixed: Hot reload now works when editing target modules. Previously, changes to imported modules weren’t picked up on Rerun because Python’s sys.modules cache wasn’t cleared.
  • Fixed: Rerun after renaming a run now works correctly. Previously, app.state.run_name wasn’t synced after rename, causing results to write to a duplicate file.
  • Fixed: Progress bar now uses CSS variable --progress-bar-bg for proper light/dark mode theming.
  • Fixed: Edit run name button now works when dropdown is shown (hides dropdown, shows input).
  • Fixed: Run dropdown now hides when rows are selected for rerun.
  • Tests: Added E2E tests for comparison mode functionality.
  • Tests: Added hot reload test for module cache clearing.
  • Tests: Added tests for run metadata injection into EvalContext.

​
0.0.2a17 - 2025-12-16

  • Added: Per-case dataset and labels support via @parametrize and input_loader. Dataset overrides function-level; labels merge (no duplicates).
  • Added: input_loader parameter on @eval decorator for dynamic data loading from external sources (databases, APIs like LangSmith). Loader is called lazily at eval time, each example becomes a separate eval run.
  • Added: Load previous runs with ezvals serve path/to/run.json. Opens UI with that run loaded; rerun enabled if source eval file exists, view-only mode if not.
  • Added: HTTP API reference documentation for building custom UIs and integrations.
  • Added: Inline run name editing—click pencil icon in stats bar to rename runs. Enter or checkmark saves, Escape or blur cancels.
  • Added: Click-to-copy on session/run names with “Copied!” tooltip feedback.
  • Changed: Stats panel now dynamically updates when filters/search are active. Shows “filtered/total” format for test count, with latency and score chips calculated from visible rows only.
  • Changed: Removed separate filtered summary bar—main stats panel now serves as single source of truth for all statistics.
  • Changed: Scores column is now sortable in the results table.
  • Fixed: Sticky header no longer overlaps content when scrolling.
  • Fixed: Progress bar uses subtle animation instead of shimmer effect.
  • Fixed: Latency display now shows correct values for filtered results.
  • Fixed: Detail page correctly displays reference values and trace URLs.
  • Tests: Added unit tests for input_loader functionality.
  • Tests: Updated E2E filter test to verify new dynamic stats behavior.
  • Tests: Added integration tests for loading previous run JSON files.
  • Tests: Added E2E tests for run rename feature and UI fixes.

​
0.0.2a9 - 2025-12-06

  • Changed: Project renamed from Twevals to EZVals. Package is now ezvals, CLI command is ezvals, config file is ezvals.json, results directory is .ezvals/.

​
0.0.2a16 - 2025-12-06

  • Changed: Results storage restructured to hierarchical sessions with overwrite-by-name semantics. New directory structure: .ezvals/sessions/{session_name}/{run_name}_{timestamp}.json.
  • Changed: serve command auto-generates session names (each serve = new session). run command defaults to “default” session.
  • Changed: Config results_dir default updated from .ezvals/runs to .ezvals/sessions.
  • Added: GitHub-style split button for Run controls with Rerun/New Run dropdown options.
  • Added: Run mode persistence via localStorage - dropdown selection is sticky across sessions.
  • Added: Descriptive subtext in dropdown menu (“Overwrite current run results” / “Create a fresh run in this session”).
  • Added: API endpoints for new run: POST /api/runs/new (creates without overwrite), DELETE /api/runs/{run_id}, DELETE /api/sessions/{session_name}.
  • Fixed: Run button state no longer sporadically resets after clicking Run due to race condition in _hasRunBefore tracking.
  • Fixed: Run name now displays immediately on server start (previously only appeared after first run file was saved).
  • Tests: Added E2E tests for split button behavior and run controls.
  • Tests: Updated unit tests for new storage structure and session management.

​
0.0.2a15 - 2025-12-05

  • Changed: add_output() and add_score() replaced with unified store() method. Use ctx.store(output=..., scores=..., messages=..., trace_url=...) for all context updates. Same score key overwrites, different key appends.
  • Changed: run_data renamed to trace_data with structured TraceData schema. First-class messages and trace_url properties, plus arbitrary extra fields via dict-style access.
  • Added: --run flag for ezvals serve to auto-run all evaluations on startup.
  • Added: Light mode support with theme toggle in UI header.
  • Changed: UI completely refactored to client-side rendering with new dashboard design featuring bar charts and score breakdown.
  • Added: Three-state filtering for datasets and labels (include/exclude/any) in the web UI.
  • Added: Filter persistence across navigation using sessionStorage.
  • Added: Keyboard shortcuts for table view (r=refresh, e=export, f=filter).
  • Added: Experience specifications (.spec/) as canonical source of truth for product behavior.
  • Tests: Added E2E tests for keyboard shortcuts and stats bar.
  • Tests: Added unit tests for CLI filtering and store() method.

​
0.0.2a14 - 2025-12-01

  • Changed: Documentation rewritten for clarity—simplified examples, removed redundant code patterns, consistent use of direct ctx.output assignment.
  • Changed: README simplified to match new documentation patterns—assertions as primary scoring, direct property assignment, removed set_params() references.
  • Changed: Removed set_params() method from EvalContext (use direct property assignment instead).
  • Added: UI screenshot in documentation assets.

​
0.0.2a13 - 2025-11-30

  • Added: --no-save flag for ezvals run to skip saving results to file (outputs JSON to stdout instead).
  • Added: Dataset and label pill filters in the web UI sidebar for quick filtering.
  • Added: Shift-click range selection for result checkboxes in the web UI.
  • Added: Trace URL link button in result detail page when run_data.trace_url is present.
  • Added: Footer with GitHub and documentation links in the web UI.
  • Changed: Error messages now include full stack traces for better debugging.
  • Changed: Errors print immediately to console (in red) during evaluation runs in both serve and run commands.
  • Changed: Server logging reduced to warnings only (no access logs) for cleaner output.
  • Changed: Web UI header is now sticky for better navigation.

​
0.0.2a12 - 2025-11-30

  • Changed: CLI restructured into explicit subcommands - ezvals serve <path> starts the UI, ezvals run <path> runs headless. The --serve flag is removed.
  • Changed: ezvals serve no longer auto-runs evaluations. Discovered evals are displayed in the UI with “not_started” status; users click Run to start execution.
  • Changed: Removed --limit, --dev, --host, --list, and --quiet flags from ezvals serve to simplify the serve command.
  • Changed: ezvals run now outputs minimal text by default (optimized for LLM agents) - just “Running…” and “Results saved to…”. Use --visual for rich output.
  • Added: --visual flag for ezvals run to show progress dots, results table, and summary (the previous default behavior).
  • Added: --verbose/-v flag for ezvals run now shows stdout from eval functions instead of controlling output verbosity.
  • Changed: Results are always auto-saved to file. Priority: --output flag > config results_dir > default .ezvals/runs/.
  • Changed: Removed --csv and --json flags from ezvals run - results always save as JSON.
  • Changed: Concurrency minimum is now 1 (sequential execution). Values < 1 throw an error.
  • Added: Support for running selected evaluations from the initial “not_started” state via checkbox selection in the UI.
  • Added: ezvals.json config file for persisting CLI defaults. Auto-generated on first run with concurrency and results_dir.
  • Added: Settings UI in the web interface to edit runtime options (concurrency, results_dir, timeout). Changes take effect on next run without restarting the server.
  • Removed: rerun_config from run JSON files (YAGNI - feature for viewing historical runs didn’t exist).
  • Tests: Updated CLI tests to use new run subcommand syntax.

​
0.0.2a11 - 2025-11-29

  • Added: Full-page detail view for individual evaluation results with dedicated URL (/runs/{run_id}/results/{index}), enabling new tab support and multi-monitor workflows.
  • Added: Keyboard navigation on detail page (Arrow keys to navigate between results, Escape to return to table).
  • Added: Expandable table rows - click any row to expand and view truncated content inline; click function name to navigate to full detail page.
  • Added: Scores wrap and show all values when table row is expanded (previously limited to 2 with “+N” overflow).
  • Added: Markdown rendering and syntax highlighting for Output content on detail page.
  • Added: Collapsible Run Data section on detail page.
  • Changed: Scroll position is now preserved when returning from detail page to table view.
  • Changed: Removed expandable detail rows from table in favor of row expand/collapse pattern and dedicated detail page.
  • Fixed: Server log output no longer experiences staircase effect when using --serve due to improved terminal handling.
  • Added: Server can now be stopped by pressing Esc key when running with --serve.
  • Changed: EvalContext detection now uses type annotation (: EvalContext) instead of parameter name matching. Any parameter name works as long as it’s typed correctly.
  • Added: Session management for grouping related eval runs together with --session and --run-name CLI flags.
  • Added: Auto-generated friendly names (adjective-noun format like “swift-falcon”) when session/run names not provided.
  • Added: File naming with run-name prefix: {run_name}_{timestamp}.json.
  • Added: Session/run metadata (session_name, run_name, run_id) in run JSON files.
  • Added: UI display of current session and run name in the stats bar.
  • Added: API endpoints for session management: GET /api/sessions, GET /api/sessions/{name}/runs, PATCH /api/runs/{run_id}.
  • Added: Run controls - stop button to cancel running/pending evaluations mid-run.
  • Added: Selective rerun - select individual results via checkboxes and rerun only those.
  • Added: UI selection checkboxes with select-all functionality and indeterminate state.
  • Added: POST /api/runs/stop endpoint to cancel running evaluations.
  • Changed: POST /api/runs/rerun now accepts optional indices parameter for selective reruns.
  • Tests: Added E2E tests for run controls (selection, start, stop, selective rerun).
  • Changed: Codebase cleanup reducing ~510 lines through consolidating duplicate patterns, simplifying async/sync execution, and removing redundant code.

​
0.0.2a10 - 2025-11-28

  • Added: run_evals() function for programmatic execution of multiple evals with support for functions, paths, concurrency, and all CLI options.
  • Added: Direct call support for parametrized evals - calling a parametrized function now runs all variants and returns List[EvalResult].
  • Changed: EvalFunction.__call__ now detects __param_sets__ attribute and automatically runs all parametrized variants.
  • Added: UI redesign with dark mode support, amber/zinc color scheme, improved typography, and responsive layout.
  • Added: Background evaluation execution - UI loads immediately while evals run in background with live status updates.
  • Added: Auto-open browser when starting ezvals --serve for faster workflow.
  • Added: Rerun configuration stored in run JSON for reproducible reruns from UI.
  • Changed: Simplified server/CLI code by inlining background execution logic.

​
0.0.2a9 - 2025-11-23

  • Added: timeout parameter to @eval decorator for setting per-evaluation timeout limits in seconds.
  • Added: --timeout CLI flag for ezvals run command to set a global timeout that overrides individual test timeouts.
  • Added: Timeout enforcement for both sync and async functions using concurrent.futures.ThreadPoolExecutor and asyncio.wait_for.
  • Added: Timeout support for target hooks with proper error handling and latency tracking on timeout.
  • Tests: Added comprehensive timeout tests covering async/sync functions and target hooks.

​
0.0.2a8 - 2025-11-23

  • Added: --list flag to ezvals run command to list evaluations without running them, preserving all filtering options.
  • Changed: Removed standalone list command in favor of run --list.
  • Tests: Added regression test for concurrency output capturing.

​
0.0.2a7 - 2025-11-23

  • Added: File-level defaults via ezvals_defaults dictionary - set global properties (dataset, labels, metadata, etc.) at the top of test files that all tests inherit, similar to pytest’s pytestmark pattern.
  • Added: Support for all decorator parameters in file defaults including evaluators, target, input, reference, default_score_key, metadata, and metadata_from_params.
  • Added: Deep merge for metadata - when both file defaults and decorator specify metadata, they are merged with decorator values taking precedence on conflicts.
  • Added: Deep copy of mutable values (lists, dicts) in file defaults to prevent shared mutation between tests.
  • Added: Validation and warnings for unknown keys in ezvals_defaults dictionary.
  • Changed: default_score_key parameter default changed from “pass” to None to enable file-level defaults, with “pass” still applied as final fallback via EvalContext.
  • Tests: Added 17 comprehensive tests for file defaults including inheritance, overrides, deep merge, mutable value copying, and default_score_key priority chain.

​
0.0.2a6 - 2025-11-23

  • Added: target parameter to @eval decorator allowing pre-hook functions that run before the evaluation function, enabling separation of agent/LLM invocation from evaluation logic.
  • Added: Target hooks receive EvalContext and can populate output, metadata, and custom attributes before the eval function executes.
  • Added: Target hooks support both sync and async functions, with automatic latency tracking.
  • Added: Parametrize integration with targets - parametrized values are automatically available to target hooks via ctx.input and ctx.metadata.
  • Added: Target return value handling - targets can return dicts (treated as output payload) or EvalResult objects for flexible result injection.
  • Changed: Parametrize now defaults ctx.input to parametrized values when no explicit input is provided, making param data accessible to targets.
  • Tests: Added comprehensive unit tests for target functionality including output injection, error handling, async support, and parametrize integration.

​
0.0.2a5 - 2025-11-22

  • Added: --json flag for ezvals run command to output results as compact JSON to stdout, omitting null values for machine-readable output.
  • Added: Pytest-style progress reporting during evaluation execution with colored output (green for pass, red for fail/error).
  • Added: Progress display shows one line per file with filename prefix followed by status characters (., F, E), matching pytest’s output format.
  • Added: Detailed failure reporting after progress output showing dataset::function_name, error messages, and input/output for failed evaluations.
  • Added: Progress hooks (on_start, on_complete) to EvalRunner for extensible progress reporting.
  • Changed: Replaced spinner-only progress indicator with real-time pytest-style character output.
  • Tests: Added comprehensive tests for progress reporting hooks and CLI progress output validation.
  • Tests: Added tests for --json flag output format validation and null value omission.

​
0.0.2a4 - 2025-11-22

  • Added: Function name filtering using file.py::function_name syntax, similar to pytest. Run specific evaluation functions or parametrized variants (e.g., ezvals run tests.py::my_eval or tests.py::my_eval[param1]).
  • Tests: Added comprehensive tests for function filtering including exact matches, parametrized variants, and combined filters with dataset/labels.

​
0.0.2a3 - 2025-11-22

  • Fixed: add_output() now correctly handles dicts without EvalResult fields (like {'full_name': 'Kim Diaz'}) by storing them as-is in the output field, instead of incorrectly treating them as structured EvalResult dicts.
  • Fixed: Eval results table now preserves source file order instead of alphabetically sorting functions by name (resolves #2).
  • Changed: Replaced ParametrizedEvalFunction class with function attributes (__param_sets__, __param_ids__) for simpler architecture.
  • Changed: generate_eval_functions() is now a standalone function instead of a class method.
  • Changed: Removed special-case handling for parametrized functions in @eval decorator, unifying code paths.
  • Tests: Added comprehensive tests for add_output() behavior with arbitrary dict structures to prevent regression.
  • Tests: Added test to verify source file order preservation in discovery.

​
0.0.2a2 - 2025-11-22

  • Fixed: Failed assertions now create failing scores instead of error states, properly treating them as validation failures rather than execution errors.
  • Fixed: Output field is now preserved in results table when assertions fail, instead of being overwritten with error messages.
  • Changed: Table formatter only displays error in output column when output is empty, preventing loss of actual output data.
  • Tests: Added comprehensive tests for assertion handling including edge cases for assertions without messages, non-assertion errors, and multiple assertions.

​
0.0.2a1 - 2025-11-22

  • Fixed: Module discovery now properly handles relative imports by temporarily adding parent directory to sys.path with cleanup.

​
0.0.2a0 - 2025-11-22

  • Added: EvalContext (accessible as ctx, context, or carrier parameter) provides a mutable builder pattern for constructing eval results incrementally with methods like add_output(), add_score(), and set_params().
  • Added: Context manager support for EvalContext allowing with statement usage in eval functions.
  • Added: Automatic context injection - functions accepting a ctx/context/carrier parameter receive an EvalContext instance automatically.
  • Added: Auto-return feature - eval functions using context don’t need explicit return statements; context is auto-built at function end.
  • Added: Smart add_output() method that extracts EvalResult fields (output, latency, run_data, metadata) from dict responses or sets output directly.
  • Added: Flexible add_score() supporting boolean pass/fail, numeric values, custom keys, and full control via kwargs.
  • Added: set_params() helper for parametrized tests to set both input and metadata from params.
  • Added: Filters in the web UI for dataset, labels, and status with multi-select support and persistence.
  • Added: Reference column in results table to display expected/ground truth outputs.
  • Changed: Tests that complete without calling add_score() now automatically pass with a default “pass” score, similar to pytest behavior.
  • Changed: EvalContext defaults to “pass” as the default score key (previously required explicit key).
  • Changed: Migrated from Poetry to uv for faster dependency management and installation.
  • Changed: EvalContext.build() auto-adds {"key": "pass", "passed": True} when no scores are provided and no error occurred.
  • Changed: EvalRunner ensures all results without scores and without errors receive a default passing score.
  • Tests: Added comprehensive unit tests for EvalContext methods, context manager pattern, and decorator integration.
  • Tests: Added integration tests for context usage patterns, parametrize auto-mapping, and assertion preservation.
  • Tests: Added e2e test for advanced UI filters functionality.

​
0.0.1a0 - 2025-09-04

  • Added: Results Web UI with expandable rows, multi-column sorting, column toggles and resizable columns, copy-to-clipboard, and summary chips for score ratios/averages.
  • Added: Inline editing in the UI for dataset, labels, metadata (JSON), scores (key/value/passed/notes), and a free-form annotation. Edits are persisted to JSON.
  • Added: Actions menu in the UI with Refresh, Rerun full suite, Export JSON, and Export CSV.
  • Added: Server endpoints: PATCH /api/runs/{run_id}/results/{index} for updates; POST /api/runs/rerun; GET /api/runs/{run_id}/export/{json|csv}.
  • Added: ezvals serve --dev hot-reload mode for rapid UI/eval iteration.
  • Added: CLI CSV export via ezvals run ... --csv results.csv (in addition to JSON -o).
  • Added: ResultsStore for robust, atomic JSON writes under .ezvals/runs/ with latest.json convenience copy.
  • Added: EvalResult.run_data for run-specific structured data, displayed in the UI details panel.
  • Changed: CLI run prints a results table by default and a concise summary below it.
  • Tests: Added integration tests for server (JSON flow, export endpoints, rerun) and e2e UI tests (Playwright), plus unit tests for storage behavior.