EvalSpec
Agent Runtimes / types/evals / EvalSpec
Interface: EvalSpec
Defined in: src/types/evals.ts:9
Built-in evaluator specification.
Properties
category
category:
"Comparison"|"Type Validation"|"Performance"|"LLM-as-a-Judge"|"Span-Based"|"Report"
Defined in: src/types/evals.ts:19
Evaluator family
cost_tier
cost_tier:
"free"|"llm"
Defined in: src/types/evals.ts:39
Cost tier for running this evaluator
default_config?
optionaldefault_config?:Record<string,unknown>
Defined in: src/types/evals.ts:47
Suggested baseline configuration
description
description:
string
Defined in: src/types/evals.ts:17
Description of the evaluator
evaluator_type
evaluator_type:
"case"|"report"
Defined in: src/types/evals.ts:27
Case-level or report-level evaluator
id
id:
string
Defined in: src/types/evals.ts:11
Unique eval identifier
latency
latency:
"instant"|"fast"|"slow"
Defined in: src/types/evals.ts:41
Expected latency profile
name
name:
string
Defined in: src/types/evals.ts:15
Display name
output_kind
output_kind:
"boolean"|"boolean_with_reason"|"score"|"score_and_assertion"|"report_table"|"report_curve"
Defined in: src/types/evals.ts:31
Primary output shape
pydantic_class
pydantic_class:
string
Defined in: src/types/evals.ts:29
Pydantic evaluator class name
requires?
optionalrequires?:string[]
Defined in: src/types/evals.ts:43
Runtime requirements (e.g. expected_output, logfire, model)
source
source:
string
Defined in: src/types/evals.ts:45
Source documentation URL
version?
optionalversion?:string
Defined in: src/types/evals.ts:13
Version