Module llm 

An Agent backed by any OpenAI-compatible /chat/completions endpoint.

Why this exists

Two reasons, and the second matters more.

It proves the protocol plumbing survives a real streaming model rather than a fixture. And it is the architecture test: docs/DESIGN.md claims Agent is the LLM boundary and that no crate in this workspace depends on a model library. This agent reaches the model with reqwest, a handful of serde structs and nothing else, and implements nothing but Agent. That it compiles and streams is what turns that claim into evidence — so keep rig, async-openai and friends out of it.

Why the OpenAI wire format rather than a vendor’s own

This used to speak Gemini’s native :streamGenerateContent dialect, and being bound to one vendor cost real time: the free tier ran out, the harness fell back to a sibling model, the sibling was a 3.x model that requires thoughtSignature echoed back in tool loops, and the run died on HTTP 400: Function call is missing a thought_signature in functionCall parts. — a failure that was invisible until the fallback fired.

/chat/completions is the one shape nearly everything serves: Gemini’s compatibility endpoint, Ollama, llama.cpp, LM Studio, vLLM, Groq, Together. Pointing this agent at a different provider is now an env var, thought signatures are the compatibility layer’s problem rather than ours, and the whole vendor-schema translation this file used to carry is gone — the request takes an AG-UI Tool’s JSON Schema through unchanged.

The mapping, and the parts of it that bite

docs/QA.md records the whole mapping. The awkward corners, all handled below and all covered by the tests at the bottom of this file:

• Tool-call arguments arrive as partial JSON accumulated across frames, keyed by tool_calls[].index. A fragment can split anywhere, including mid-string and between a backslash and the character it escapes, so nothing may parse a fragment on its own.
• tool_calls[].index is not always there. The spec says it is, and OpenAI, Ollama and Groq send it — but Gemini’s compatibility endpoint omits it entirely and puts two parallel calls in one frame, distinguished only by id. Keying on index alone merges parallel calls into JSON soup, so Calls falls back to id, then to array position.
• The stream ends at a data: [DONE] sentinel, unlike the native API, which just EOFs.
• finish_reason may arrive on a frame carrying no content, which must not become an empty TEXT_MESSAGE_CONTENT.
• tool_calls[].id comes from the server. It is used as the AG-UI toolCallId as-is; one is synthesized only for a server that sends none.
• Line terminators differ between endpoints of the same vendor: Gemini’s native SSE frames end \r\n\r\n and its OpenAI-compatible ones end \n\n. Both are accepted — see take_block.

Structs

Endpoint

What the environment says to talk to.

LlmAgent

An Agent that answers with an OpenAI-compatible model, and can call one tool on the way.

MissingApiKey

LlmAgent::from_env found no API key, and the endpoint it would have talked to needs one.

Constants

API_KEY_ENV

The environment variable holding the API key.

BASE_URL_ENV

The environment variable holding the base URL.

DEFAULT_BASE_URL

Where requests go unless BASE_URL_ENV says otherwise.

DEFAULT_MODEL

The model this agent talks to by default.

FALLBACK_API_KEY_ENV

Read when API_KEY_ENV is unset, because the default endpoint is Gemini’s and a contributor who has used this repo before already has this one set.

MODEL_ENV

The environment variable holding the model id.

QWEN_API_KEY_ENV

The key that goes with QWEN_BASE_URL_ENV.

QWEN_BASE_URL_ENV

Read when neither BASE_URL_ENV nor a key for the default endpoint is set: Qwen Cloud’s OpenAI-compatible mode, for a contributor who has that subscription rather than a Gemini key. The base URL is the one DashScope documents for compatible mode, ending in /compatible-mode/v1.

QWEN_DEFAULT_MODEL

The Qwen model used when QWEN_MODEL_ENV is unset. Pinned, like the default: an alias that moves changes behaviour without a code change.

QWEN_MODEL_ENV

The model that goes with QWEN_BASE_URL_ENV, when MODEL_ENV is unset.

WEATHER_TOOL

The tool this agent owns and executes itself.

Functions

weather_tool

The AG-UI definition of the tool this agent owns.