Skip to main content

evals.remote.runner

Reusable evalset execution runner.

This module hosts the end-to-end "execute an evalset spec against one or more agentspecs" workflow so that examples, the GitHub Action, and any other integration can launch real eval runs without re-implementing the orchestration (create evalset -> launch cloud runtime(s) -> run each case through the agent -> grade outputs -> persist runs -> teardown runtimes).

execute_evalset_spec​

def execute_evalset_spec(
        client: AgentClient,
        *,
        spec: dict[str, Any],
        agentspec_ids: list[str],
        run_evalset: bool = True,
        create_report: bool = False,
        run_limit: int = 1,
        run_environment: str = "sdk",
        environment_name: str = DEFAULT_ENVIRONMENT_NAME,
        billing_entity_uid: Optional[str] = None,
        account_uid: Optional[str] = None,
        credits_limit: float = 100.0,
        evalset_name: Optional[str] = None,
        backend_run_environment: str = "sdk",
        launch_source: str = "datalayer-core",
        agent_name: str = DEFAULT_AGENT_NAME,
        execution_target: str = "cloud",
        local_agent_base_url: str = DEFAULT_LOCAL_AGENT_BASE_URL,
        auto_start_local_agent_runtime: bool = False,
        local_agent_log_level: str = "info",
        request_timeout_seconds: int = DEFAULT_REQUEST_TIMEOUT_SECONDS,
        log: Optional[Callable[[str], None]] = print) -> dict[str, Any]

Execute an evalset spec against one or more agentspecs and persist runs.

Creates an evalset from spec, runs every case through each agentspec run_limit times against either a cloud runtime (one per agentspec) or a local agent-runtimes server, grades the outputs with the evals API, and stores one run record per execution. Execution resources (cloud runtimes or the local agent registration/server) are always torn down before returning, including on error.

Parameters​

client : AgentClient An authenticated client. spec : dict[str, Any] Evalset spec (as loaded by :func:load_evalset_spec). agentspec_ids : list[str] Agentspec ids to execute. One experiment is created per id (plus one cloud runtime per id when execution_target='cloud'). run_evalset : bool Whether to execute the evalset after creating it. Defaults to True. When False, this function only creates the evalset and returns. create_report : bool Whether to generate markdown/CSV reports via :func:3 before returning. Defaults to False. run_limit : int Number of runs to create per experiment (minimum 1). run_environment : str Run-environment label stored on run summaries (for example sdk). environment_name : str Runtime environment to launch cloud agents in (cloud only). billing_entity_uid : Optional[str] Optional billing entity UID context. account_uid : Optional[str] Optional account UID context. credits_limit : float Target credits budget used to size each cloud runtime reservation. evalset_name : Optional[str] Optional explicit evalset name. Defaults to a timestamped name derived from the spec name. backend_run_environment : str run_environmentvalue persisted on the created evalset. launch_source : str launch_sourcerecorded on experiments and runs. agent_name : str Agent route/name used when contacting the runtime. execution_target : str cloud(default) launches one cloud runtime per agentspec;local executes against a localagent-runtimesserver. local_agent_base_url : str Base URL of the localagent-runtimesserver (local only). Ignored whenauto_start_local_agent_runtimestarts a new server. auto_start_local_agent_runtime : bool Whenexecution_target='local', start a local agent-runtimes server process automatically and terminate it during cleanup. The runner first attempts to reuselocal_agent_base_urland will auto-start a local runtime only if that server is unreachable. local_agent_log_level : str Log level for an auto-started localagent-runtimesserver. request_timeout_seconds : int Per-request timeout (seconds) for a single agent chat call. When an agent does not respond within this window the call is aborted, the case is recorded as failed, and execution continues with the next case. This avoids one hung request blocking all remaining cases/experiments/runs. Defaults to180(3 minutes per call). log : Optional[Callable[[str], None]] Optional logging callback (defaults toprint; pass None`` to silence progress output).

Returns​

dict[str, Any] {"evalset_id", "evalset_name", "experiment_ids", "run_ids", "view_url"} plus optional report paths when create_report=True.

Raises​

ValueError If agentspec_ids is empty, the spec has no cases, or execution_target is not cloud/local. RuntimeError If the platform returns an unexpected create response or a cloud runtime is missing its ingress/pod.