`aaa.eval`¶

EvalSuite ¶

EvalSuite(
    dataset: Dataset,
    algo: Algorithm,
    metrics: Sequence[Metric],
    *,
    name: str | None = None,
    concurrency: int = 4,
    repeats: int = 1,
    budget: Budget | None = None,
    tenant: str | None = None,
    extract: Callable[[Any], str] = default_extract,
    make_input: Callable[[Case], dict[str, Any]]
    | None = None,
)

EvalSuite(dataset, algo, [metrics]).run(engine) -> EvalReport.

Each case is a real run (engine.run) with meta={"aaa_eval": True}; the trajectory (spans), steps, tokens and cost are read back from its log, so tool-use and budget metrics see what really happened.
repeats runs each case several times (agents are non-deterministic); the report keeps every run.
A metric that raises is recorded as a failed score, not a crashed suite; a run the harness could not start is error.
budget bounds each case, so one runaway case cannot spend the whole eval.

Dataset ¶

Dataset(cases: Iterable[Case], name: str = 'dataset')

from_list `classmethod` ¶

from_list(
    rows: Sequence[dict[str, Any]], name: str = "dataset"
) -> Dataset

Rows are {"input": ..., "expected": ...}; input may be a plain string (becomes {"task": ...}).

Case ¶

Bases: BaseModel

CaseResult `dataclass` ¶

CaseResult(
    case: Case,
    outcome: Outcome[Any] | None,
    text: str = "",
    run_id: str | None = None,
    spans: list[Span] = list(),
    steps: int = 0,
    tokens: int = 0,
    cost: float = 0.0,
    latency_s: float = 0.0,
    error: str | None = None,
    repeat: int = 0,
    scores: list[Score] = list(),
)

EvalReport `dataclass` ¶

EvalReport(name: str, results: list[CaseResult] = list())

pass_rate `property` ¶

pass_rate: float

Share of case runs where every pass/fail-like score passed (and the harness ran the case).

regressions ¶

regressions(
    baseline: dict[str, Any] | str | Path,
    *,
    tolerance: float = 0.02,
) -> list[Regression]

Metrics whose mean fell by more than tolerance against a saved baseline, plus a fall in the overall pass rate.

assert_ok ¶

assert_ok(
    *,
    min_pass_rate: float = 1.0,
    baseline: dict[str, Any] | str | Path | None = None,
    tolerance: float = 0.02,
) -> None

For CI: raise AssertionError with the failing cases if the run is below the bar or regressed.

OnlineEvaluator ¶

OnlineEvaluator(
    engine: Engine,
    metrics: Sequence[Metric],
    *,
    sample: float = 1.0,
    sinks: Sequence[Any] = (),
    max_cost: float | None = None,
    tenant: str | None = None,
    group: str = "aaa.eval",
)

Metric ¶

Bases: Protocol

LLMJudge ¶

LLMJudge(
    client: LLMClient,
    rubric: str,
    *,
    model: str | None = None,
    name: str = "judge",
    use_expected: bool = True,
)

A model grades the answer against a rubric. The judge runs OUTSIDE the evaluated run (its calls are not part of that run's trace or budget); what it cost is reported on the Score and summed in the report.

Budgeted ¶

Budgeted(
    *,
    steps: int | None = None,
    tokens: int | None = None,
    cost: float | None = None,
    latency_s: float | None = None,
    name: str = "within_budget",
)

Within a cost/step/token/latency allowance; the score falls linearly past the limit to 0 at twice the limit.

AutoevalsMetric ¶

AutoevalsMetric(
    scorer: Any,
    *,
    name: str | None = None,
    pass_threshold: float | None = None,
)

Run one autoevals scorer on the case: output is the agent's answer, expected the case's, input the task text.

openai_shim ¶

openai_shim(
    client: LLMClient, *, model: str | None = None
) -> Any

An OpenAI-SDK-shaped object backed by an AAA LLMClient (async only: use autoevals' eval_async).

DeepEvalMetric ¶

DeepEvalMetric(
    metric: Any,
    *,
    name: str | None = None,
    retrieval_context_key: str = "retrieval_context",
    context_key: str = "context",
)

Run one DeepEval metric (single-turn LLMTestCase metrics) on a case result.

DeepEvalJudge ¶

DeepEvalJudge(
    client: LLMClient,
    model: str | None = None,
    **defaults: Any,
) -> Any

A DeepEvalBaseLLM over client. Totals of what it used are on .calls, .tokens and .cost.

RagasMetric ¶

RagasMetric(metric: Any, *, name: str | None = None)

RagasJudge ¶

RagasJudge(
    client: LLMClient,
    model: str | None = None,
    **params: Any,
) -> Any

An InstructorBaseRagasLLM over client: structured replies are requested with the pydantic model's JSON Schema.

aaa_solver ¶

aaa_solver(
    setup: Callable[
        [],
        AbstractAsyncContextManager[
            tuple[Engine, Algorithm]
        ],
    ],
    *,
    make_input: Callable[
        [str], dict[str, Any]
    ] = lambda text: {"task": text},
    extract: Callable[[Any], str] = default_extract,
    budget: Budget | None = None,
) -> Any

An Inspect Solver that runs one AAA run per sample and puts its answer in state.output.

aaa.eval¶