Skip to main content

evals.remote.evaluators

Reusable evaluator execution for real (non-synthetic) eval runs.

Implements the common Datalayer evaluators (equals_expected, equals, contains, pass_rate_threshold) so examples, the CLI, and integrations can grade real agent outputs instead of fabricating scores. Evaluator names mirror the evaluator catalog (see agent_runtimes/specs/evals); unknown names degrade gracefully to a skipped record so callers never crash on an unsupported evaluator.

run_case_evaluators​

def run_case_evaluators(
        *, output: Any, expected: Any,
        evaluators: list[dict[str, Any]] | None) -> dict[str, Any]

Grade a single real output against its per-case evaluators.

Returns {"passed": bool, "score": float, "evaluators": [...]}. A case passes when every applicable evaluator passes; the score is the mean of the evaluator scores (and defaults to 1.0 when no evaluator applies). Unsupported evaluators are recorded as skipped and ignored.

evaluate_run​

def evaluate_run(cases: list[dict[str, Any]],
                 outputs: list[Any],
                 *,
                 evalset_evaluators: list[dict[str, Any]] | None = None,
                 report_evaluators: list[dict[str, Any]] | None = None,
                 statuses: list[str] | None = None) -> dict[str, Any]

Grade real per-case outputs and return run metrics.

outputs is aligned with cases (each entry a str or a mapping with a text key). Evalset-level evaluators run for every case; report-level evaluators run once over the resulting case outcomes. The returned metrics mirror the synthetic shape (case_results and evaluator_results) so the UI and report render real and synthetic runs identically.

evaluate_evalset​

def evaluate_evalset(evalset_spec: dict[str, Any],
                     outputs: list[Any],
                     *,
                     statuses: list[str] | None = None) -> dict[str, Any]

Grade real outputs against a declarative evalset spec.

Convenience wrapper that pulls cases, evalset_evaluators and report_evaluators out of an evalset spec dict (as produced by :func:agent_runtimes.evals.remote.load_evalset_spec) and delegates to :func:evaluate_run. This is the single entry point examples and the CLI use so evaluator execution lives in the evals API rather than the caller.

run_and_evaluate_evalset​

def run_and_evaluate_evalset(
        evalset_spec: dict[str, Any],
        run_case: CaseRunner,
        *,
        statuses: list[str] | None = None) -> dict[str, Any]

Execute every case through a runner callback, then grade the outputs.

This bakes the per-case execution loop into the evals API so consumers (examples, GitHub Actions, and other integrations) never re-implement the "run each case, then evaluate" orchestration. run_case is called once per case as run_case(case, index) and may return either:

  • a plain output (str or a mapping with a text key), or
  • a mapping {"output": <output>, "status": <status>} where status is an optional per-case run status (e.g. "failed") that forces the case to fail regardless of evaluator outcome.

Per-case and report-level evaluators from the spec then run for real over the collected outputs via :func:4, returning the same metrics shape as synthetic runs (case_resultsandevaluator_results``) so reports and the UI render real and simulated runs identically.