Skip to content

aaa.llm

Protocols and handler

LLMClient

Bases: Protocol

The only thing algorithms know about LLMs. Any gateway/provider/custom backend that can implement this works; errors must be raised as aaa.llm.errors types.

StreamingLLMClient

Bases: Protocol

Optional capability: incremental output. Clients without it still work; the handler falls back to a single completion.

LLMHandler 

LLMHandler(
    client: LLMClient,
    *,
    concurrency: int = 8,
    emit_deltas: bool = True,
    context: ContextPolicy | None = None,
    guards: GuardPolicy | None = None,
)

Capability handler: aaa.llm.requested in -> aaa.llm.completed|failed out. Request data: {"request": LLMRequest, "tag": optional str echoed back}. Failures are facts on the log (the algorithm decides what to do), never exceptions. The completed event carries top-level usage/cost so the Engine's budget ledger sees them. With request.stream and a streaming client, aaa.llm.delta events are emitted as text arrives. Tool-call ids are rewritten to be unique per call (providers may omit or repeat them).

With context= (an aaa.context.ContextPolicy) each request is fitted to the model's window just before it is sent: the conversation the algorithm and the log hold is never rewritten, only the copy the provider receives. Summaries it writes are persisted as aaa.context.summary events.

With a server-state client (ServerStateClient, e.g. the OpenAI Responses API) the provider holds the conversation: a continuing request sends only the new messages plus the system prompt, referring to the provider's own handle (state_id, recorded on each reply). The full conversation is sent instead whenever the context policy had to rewrite it, the provider has lost the state, or no handle is known. Replies carry state = continued | fresh | fallback so the saving is visible in the log.

forget_state async 

forget_state(state_ids: Sequence[str]) -> int

Delete provider-held state for these handles (used by engine.erase).

Configuration and factories

LLMConfig

Bases: BaseModel

Declarative LLM selection. Secrets are referenced as ${ENV_VAR}, never stored.

build_client 

build_client(config: LLMConfig) -> LLMClient

register_provider 

register_provider(
    name: str, factory: Callable[[LLMConfig], LLMClient]
) -> None

Plug in a custom backend: register_provider('mine', lambda cfg: MyClient(cfg)).

OpenAICompatClient 

OpenAICompatClient(
    config: LLMConfig,
    *,
    http_client: AsyncClient | None = None,
)

One adapter for every gateway that speaks OpenAI chat-completions (Bifrost, Portkey, Kong, Cloudflare, Vercel, MLflow, LiteLLM proxy, vLLM, Ollama, OpenAI itself ...).

stream async 

stream(request: LLMRequest) -> AsyncIterator[StreamChunk]

Server-sent-events streaming. Not retried once started (caller can't un-see text).

OpenAIResponsesClient 

OpenAIResponsesClient(
    config: LLMConfig,
    *,
    http_client: AsyncClient | None = None,
    store: bool = True,
    truncation: str | None = None,
)

stream async 

stream(request: LLMRequest) -> AsyncIterator[StreamChunk]

Server-sent events. Not retried once started.

forget_state async 

forget_state(state_ids: Sequence[str]) -> int

Delete stored responses at the provider (DELETE /responses/{id}); already-gone ids are fine.

ServerStateClient

Bases: Protocol

Optional capability: the PROVIDER keeps the conversation (OpenAI's Responses API with store, and similar). A client with it sets server_state = True, returns LLMResponse.state_id, accepts LLMRequest.continue_from (then messages holds only what is new), and can forget_state.

The handler uses it to avoid resending the whole conversation every turn, and falls back to a full request whenever the state is missing or the context policy had to rewrite the conversation. The cost is that the provider now holds your data: erasing a run deletes its provider-side state too.

forget_state async 

forget_state(state_ids: Sequence[str]) -> int

Delete provider-held state; returns how many ids were removed.

Gateway presets

gateways

Presets for common AI gateways. All speak the OpenAI chat-completions dialect, so a preset is only a base URL + auth/routing headers. Conventions verified against vendor docs; the two version-dependent ones are noted. Every field can be overridden afterwards.

litellm_proxy 

litellm_proxy(
    model: str,
    base_url: str = "http://localhost:4000/v1",
    **kw: Any,
) -> LLMConfig

bifrost 

bifrost(
    model: str,
    base_url: str = "http://localhost:8080/v1",
    **kw: Any,
) -> LLMConfig

Model is provider-prefixed, e.g. 'openai/gpt-4o-mini'. NOTE: path (/v1 vs /openai) has varied across Bifrost versions; override base_url if your deployment differs.

portkey 

portkey(
    model: str,
    provider: str = "openai",
    base_url: str = "https://api.portkey.ai/v1",
    **kw: Any,
) -> LLMConfig

Portkey key authenticates to Portkey; the Bearer key is the upstream provider's.

cloudflare 

cloudflare(
    model: str,
    account_id: str,
    gateway_id: str = "default",
    authenticated: bool = True,
    **kw: Any,
) -> LLMConfig

Unified /compat endpoint; model is '{provider}/{model}'. Authenticated gateways need cf-aig-authorization (Cloudflare token); the Bearer key is the upstream provider's.

vercel 

vercel(
    model: str,
    base_url: str = "https://ai-gateway.vercel.sh/v1",
    **kw: Any,
) -> LLMConfig

Model is 'creator/model' (case-sensitive), e.g. 'anthropic/claude-sonnet-4.5'.

kong 

kong(model: str, route_url: str, **kw: Any) -> LLMConfig

Kong's ai-proxy (route_type llm/v1/chat) exposes the OpenAI format at your route URL (e.g. http://localhost:8000/chat), so no path suffix is appended.

mlflow 

mlflow(
    endpoint: str,
    base_url: str = "http://localhost:5000/gateway/mlflow/v1",
    **kw: Any,
) -> LLMConfig

endpoint is the gateway endpoint name (used as the model). NOTE: older MLflow 3.x docs use http://host:5000/v1; override base_url to match your server version.

Reliability wrappers

RetryingClient 

RetryingClient(
    inner: LLMClient,
    *,
    max_attempts: int = 3,
    base_delay: float = 0.5,
    max_delay: float = 20.0,
    sleep: Callable[
        [float], Awaitable[None]
    ] = asyncio.sleep,
)

Bases: _Wrapper

Bounded exponential backoff with full jitter on retryable errors only. Honors a gateway's Retry-After. Use only one retry layer: if your gateway already retries, set max_attempts=1.

CircuitBreakerClient 

CircuitBreakerClient(
    inner: LLMClient,
    *,
    failure_threshold: int = 5,
    cooldown_s: float = 30.0,
    clock: Callable[[], float] = time.monotonic,
)

Bases: _Wrapper

closed -> (N consecutive transient failures) -> open -> (cooldown) -> half-open probe. Only transient/rate-limit failures count; caller mistakes never trip it.

Messages

LLMRequest

Bases: BaseModel

LLMResponse

Bases: BaseModel

Message

Bases: BaseModel

ToolCall

Bases: BaseModel

ToolSpec

Bases: BaseModel

Usage

Bases: BaseModel

StreamChunk

Bases: BaseModel

EmbedResponse

Bases: BaseModel

collect async 

collect(chunks: AsyncIterator[StreamChunk]) -> LLMResponse

Fold a chunk stream into the same LLMResponse a non-streaming call returns.

Test doubles

FakeLLMClient 

FakeLLMClient(
    script: Iterable[LLMResponse | Exception]
    | Callable[[LLMRequest], LLMResponse],
)

Scripted responses (in order, or via a function of the request) for deterministic tests. Exceptions in the script are raised, so error paths are testable too.

stream async 

stream(request: LLMRequest) -> AsyncIterator[StreamChunk]

Replays the scripted response as word-sized chunks (and tool-call deltas).

ReplayLLMClient 

ReplayLLMClient(events: Iterable[Event])

Bases: FakeLLMClient

Serve the responses recorded in a run's aaa.llm.completed events, in order, for deterministic golden-test replays of a real run.

Errors

errors

LLMConfigError

Bases: LLMError

Bad or incomplete configuration (missing env var, unknown provider).

TransientError

Bases: LLMError

Timeouts, connection errors, 5xx: safe to retry.

ContextOverflow

Bases: LLMError

Prompt too long for the model; retrying unchanged will not help.

StateNotFound

Bases: LLMError

The provider no longer holds the conversation state we tried to continue (expired, deleted, or never stored). Not a failure of the request: the handler resends the full conversation.

CircuitOpen

Bases: LLMError

Short-circuited by the circuit breaker; not retried.

GuardrailBlocked 

GuardrailBlocked(
    stage: str = "",
    categories: list[str] | None = None,
    guard: str | None = None,
)

Bases: LLMError

A guardrail blocked the request or the reply. Never retried; algorithms end the run as OutcomeKind.BLOCKED.