Skip to main content

BenchmarkSpec

Agent Runtimes

Agent Runtimes / types/benchmarks / BenchmarkSpec

Interface: BenchmarkSpec

Defined in: src/types/benchmarks.ts:9

Evaluation benchmark specification.

Properties​

category​

category: "Coding" | "Knowledge" | "Reasoning" | "Agentic" | "Safety"

Defined in: src/types/benchmarks.ts:19

Category: Coding, Knowledge, Reasoning, Agentic, or Safety

dataset_editability?​

optional dataset_editability?: "read-only" | "editable"

Defined in: src/types/benchmarks.ts:47

Whether cases are editable in hosted UI

dataset_source?​

optional dataset_source?: "local" | "hosted" | "hybrid"

Defined in: src/types/benchmarks.ts:31

Dataset source mode used by this benchmark

description​

description: string

Defined in: src/types/benchmarks.ts:17

Description of the benchmark

difficulty​

difficulty: "medium" | "easy" | "hard" | "expert"

Defined in: src/types/benchmarks.ts:27

Difficulty level

evaluator_shapes?​

optional evaluator_shapes?: ("pass_rate" | "numeric" | "categorical" | "error_only")[]

Defined in: src/types/benchmarks.ts:37

Shapes emitted by evaluators (pass_rate, numeric, categorical, error_only)

evaluators​

evaluators: string[]

Defined in: src/types/benchmarks.ts:41

Evaluator IDs (optionally versioned) used to score this benchmark

id​

id: string

Defined in: src/types/benchmarks.ts:11

Unique benchmark identifier

languages​

languages: string[]

Defined in: src/types/benchmarks.ts:29

Relevant languages

metric​

metric: string

Defined in: src/types/benchmarks.ts:23

Primary metric (e.g., 'pass@1', 'accuracy', 'success_rate')

name​

name: string

Defined in: src/types/benchmarks.ts:15

Display name

optional recommended_windows?: string[]

Defined in: src/types/benchmarks.ts:43

Suggested time windows for monitoring UIs

sdk_support?​

optional sdk_support?: "none" | "experimental" | "stable"

Defined in: src/types/benchmarks.ts:49

SDK maturity level for this benchmark

source​

source: string

Defined in: src/types/benchmarks.ts:25

Source URL or repository

supports_experiment_comparison?​

optional supports_experiment_comparison?: boolean

Defined in: src/types/benchmarks.ts:35

Whether this benchmark supports side-by-side run comparison

supports_live_monitoring?​

optional supports_live_monitoring?: boolean

Defined in: src/types/benchmarks.ts:33

Whether this benchmark can be tracked in live monitoring

task_count​

task_count: number

Defined in: src/types/benchmarks.ts:21

Number of tasks in the benchmark

trace_integration?​

optional trace_integration?: boolean

Defined in: src/types/benchmarks.ts:45

Whether traces include links from results to execution spans

version?​

optional version?: string

Defined in: src/types/benchmarks.ts:13

Version