Skip to main content

​
High-level overview

Every model call targets a Gateway. Bare /v1/... fails closed with a "select a gateway" 400. The Gateway, not the LLM ingress, is what you create and grant in Connect. A model call through the LLM gateway looks like this:
  1. An operator adds a model connection (and often a pool), puts it on a Gateway, and grants an agent access.
  2. The agent points an OpenAI or Anthropic client at https://<host>/gateways/<id>/v1 with a Dome agent token.
  3. Each call runs admission, pool resolution, Rules, request Guards, provider dispatch with injected credentials, response Guards, and audit.
The agent never sees the provider credential. The LLM gateway never exposes provider credentials to the agent.

​
Endpoints

The LLM gateway exposes provider-native shapes so existing SDKs work without modification. Use the Gateway’s returned OpenAI or Anthropic base URL as-is. Streaming and non-streaming are both supported. SSE chunks are filtered in flight. For embeddings, encoding_format (float or base64) controls the response wire shape only. Refer to Develop. Passthrough limits for custom connections are on the Models reference.

​
Pipeline

Model calls share these stages, with pool resolution and failover specific to this ingress:
  1. Ingress: Resolve the Gateway from the path, validate the bearer credential, extract agent identity, check kill switches, and translate the native request into Dome’s canonical shape.
  2. Pool resolution: Resolve the request’s model value to a pool or direct connection. Order and match_when are on the Pools reference.
  3. Authorize: Evaluate Cedar against the upstream that will be dispatched. Stale policy fails closed. Refer to Authorization.
  4. Dispatch: Call the provider through the connection’s adapter. Dome injects credentials at request time. Providers and credential pairings are on the Models reference.
  5. Filter: Apply Guards over streamed and non-streamed content before the agent sees it.
  6. Failover: On a transient upstream error, try further eligible pool members up to the failover limit. Each candidate is re-authorized first. Limits and streaming rules are on the Pools reference.
  7. Audit: Emit llm.* events with token counts, matched rule, pool, and final upstream. Event types are on the Events reference.
Shared credential models and deployment topology live on the Data plane concept. Denial and credential wire shapes live on Develop.

​
Authorization

LLM calls authorize against the same Cedar policy as tool calls, with resource type Dome::LLMModel. Policy evaluates the upstream that is actually dispatched: the primary member up front, and each failover candidate immediately before its own attempt. A forbid on a model or connection attribute holds for every upstream the call could reach, not just the primary. Use resource.name for the caller-facing alias, resource.resolved_model for the upstream that will run, and resource.pool when the request was routed through a pool. Injected attributes and evaluation notes are on the Rules reference. Pool grants and listing behavior for GET /v1/models follow the same rules. Refer to Rules concept and Authorize Access. Non-streaming. A deny on the primary is terminal. A policy-refused failover candidate is skipped (with access.denied) rather than treated as a successful route-around. Streaming. Failover applies only before the first response byte reaches the caller. After that, a later upstream failure terminates the stream. Verified act-as identity concept is forwarded into evaluation on every LLM endpoint. Configure verification on Delegated agents. Runtime errors and denial shapes are on Develop.

​
Next steps

With the LLM gateway model in mind, continue with:
  • Gateways concept for membership, grants, and URL shape
  • Tools gateway concept for the sibling MCP ingress
  • Data plane concept for the shared enforcement pipeline
  • Models concept and Pools concept for connections and routing
  • Models reference and Pools reference for providers, credentials, and routing catalogs
  • Rules reference for LLM resource attributes
  • Models and Pools to configure connections
  • Develop to point SDKs at a Gateway and handle denials