Skip to main content

evals.local.runner

Local, in-process eval runner.

:class:EvalRunner orchestrates evaluation suites against an in-process agent callable. It does not launch any runtime and does not create platform evalset/experiment/run records — for that remote workflow use :func:agent_runtimes.evals.remote.execute_evalset_spec.

EvalReport Objects​

@dataclass
class EvalReport()

Public-facing eval report returned by EvalRunner.run().

EvalRunner Objects​

class EvalRunner()

Executes evaluation suites against registered agents.

Parameters​

agent_id : str Identifier of the agent to evaluate. agent_fn : Callable[[dict], Awaitable[str]] | None Async function that takes {"prompt": ...} and returns agent output. If not provided, a no-op stub is used. output_dir : str Directory for storing eval reports.

run​

async def run(eval_spec: list[dict[str, Any]],
              agent_system_prompt: str | None = None,
              tool_schemas: list[dict[str, Any]] | None = None) -> EvalReport

Run the full evaluation suite.

Parameters​

eval_spec : list[dict] The evals section from the agentspec. agent_system_prompt : str | None System prompt for synthetic case generation. tool_schemas : list[dict] | None Tool JSON schemas for grounding.