When every eval in a file uses the same labels, target or evaluators, set them once at the top of the file instead of on each eval.
The defaults can include any eval option except cases and input_loader: input, reference, dataset, labels, metadata, default_score_key (defaultScoreKey), timeout, trials, target and evaluators. An unknown key is an error when the file loads.
Precedence
From lowest to highest: the file name (for dataset), then file defaults, then the eval’s own options, then each case.
- An eval’s own value replaces the default. For example,
test_complaint above has only the label edge, not production.
metadata is the exception: it merges, and the eval wins on a key clash.
evaluators set on an eval replace the default evaluators. To keep both, list both: evaluators=[default_check, specific_check].
One file per area (support.py, billing.py, search.eval.ts) gives you a dataset per area for free, because dataset defaults to the file name.