Skip to main content

evals.report

Real evaluation and reporting logic for Datalayer evals.

This module hosts the evals report engine and the helper functions used by the CLI commands, the Python evals API, and the examples. The CLI command layer in agent_runtimes.commands.evals imports from here so it can stay a thin Typer wrapper around this logic.

iter_report_runs​

def iter_report_runs(
        report: dict[str, Any]) -> list[tuple[str, dict[str, Any]]]

Return (experiment_name, run) tuples for every run in a report.

Operates on the structured report produced by :func:build_eval_report, so consumers (the GitHub Action, dashboards, alerts) don't re-walk the nested experiments -> runs shape themselves.

collect_report_failures​

def collect_report_failures(report: dict[str, Any]) -> dict[str, Any]

Aggregate failure information across every run in a report.

Returns a dict with the failed-run count, failed-status run count, a breakdown by failure type, and structured failure records (experiment, run id, stage, type, message, trimmed detail excerpt, execution URL). This is the shared aggregation the GitHub Action renders into its step summary.

average_latest_pass_rate​

def average_latest_pass_rate(report: dict[str, Any]) -> float | None

Return the mean of each experiment's latest_pass_rate (or None).

build_eval_report​

def build_eval_report(client: AgentClient,
                      evalset_id: str,
                      *,
                      billing_entity_uid: Optional[str] = None,
                      account_uid: Optional[str] = None,
                      run_limit: int = 50) -> dict[str, Any]

Return the structured eval report for an evalset.

render_eval_report_markdown​

def render_eval_report_markdown(report: dict[str, Any],
                                *,
                                run_limit: int = 50,
                                colorize: bool = False) -> str

Render a structured eval report as markdown.

write_eval_report_csv​

def write_eval_report_csv(report: dict[str, Any],
                          output_path: str | Path) -> Path

Write a structured eval report to CSV and return its path.

write_eval_reports​

def write_eval_reports(client: AgentClient,
                       evalset_id: str,
                       *,
                       billing_entity_uid: Optional[str] = None,
                       account_uid: Optional[str] = None,
                       run_limit: int = 50,
                       output_dir: str | Path = ".",
                       basename: str = "report",
                       timestamped: bool = True,
                       export_csv: bool = True) -> dict[str, Any]

Build and persist markdown (and optionally CSV) eval reports.