Stream live events

Stream live events shows governed traffic and live operational signals in real time. It reads the same Audit v1 trail as historical investigation, and can attribute LLM usage or export OpenTelemetry and product telemetry into your stack.

Refer to Audit events concept for how the one trail works. Contracts are on the Event contracts reference.

Overview

Streaming correlates several signals on the same governed traffic. Each signal answers a different operational question.

Streaming reads the same Audit v1 trail as Audit events. Stream for live operations. Query and export for history and compliance.

The typical workflow is:

1. Stream events for live governed traffic.
2. Optionally review token usage and cost.
3. Optionally configure OpenTelemetry or product usage telemetry for your own stack.

Token classes

Every LLM call resolves to a disjoint partition across five token classes. The window total is their sum.

Adapters normalize provider-specific usage records into this partition. Per-class pricing is applied at read time and surfaced as estimated_cost_usd.

Usage cubes

Token usage and call counts fan out across the workspace aggregate and four read-time cubes so you can attribute consumption by the dimension you care about.

Requirements

Before you begin:

• Authenticate to Dome and select a workspace
• Have traffic in the workspace when you want live events to appear

Permissions

Live event streams and token-usage reads require platform permissions. People and scoped API keys call these surfaces. Agent credentials are not the usual path.

All workspace roles can stream events and read token usage with audit.view. OpenTelemetry and product usage telemetry are configured on the api-server and gateway processes with environment variables. They do not use workspace permissions.

Stream events

Stream audit events in real time with server-sent events (SSE). The stream delivers events as they occur, without polling.

dome audit stream

dome audit stream --classes governing --results denied --agent-id <uuid>

GET /v1/audit/events/stream?event_classes=EVENT_CLASS_GOVERNING&results=RESULT_DENIED&agent_id=<uuid>

Query the most recent 20 audit events of type authorization.evaluate for agent "a1b2c3d4-...".

Each streamed event is a full AuditEventV1 envelope.

{
  "event_id": "<uuid>",
  "event_type": "tool.call",
  "event_class": "EVENT_CLASS_GOVERNING",
  "stage": "STAGE_COMPLETED",
  "result": "RESULT_ALLOWED",
  "occurred_at": "2026-05-30T10:30:00Z",
  "scope": { "workspace_id": "ws_..." },
  "actor": { "kind": "ACTOR_KIND_AGENT", "id": "agent_..." },
  "correlation": { "trace_id": "0af7651916cd43dd8448eb211c80319c" },
  "payload": {
    "@type": "type.googleapis.com/dome.audit.v1.ToolCallV1",
    "tool": "database-query",
    "backend": "my-mcp-server",
    "latency_ms": 142
  }
}

Filter the stream

Narrow the stream further by combining multiple type filters.

dome audit stream --types tool.call

GET /v1/audit/events/stream?event_types=tool.call

Query the most recent 20 audit events of type tool.call.

The stream remains open until you terminate it. It fits live monitoring during deployments, incident investigation, or development testing.

Token usage and cost

Query LLM consumption across every agent, API key, identity, and served model in the workspace. The dashboard token panel, the dome usage CLI, and the dome_usage MCP tool share one summary endpoint, so numbers line up everywhere.

dome usage

{
  "evaluations_allow": 1842,
  "evaluations_deny": 17,
  "tool_calls_total": 921,
  "llm_input_tokens": 412000,
  "llm_cache_read_tokens": 1820000,
  "llm_cache_write_tokens": 88000,
  "llm_output_tokens": 64000,
  "llm_reasoning_tokens": 12000,
  "llm_total_tokens": 2396000,
  "llm_calls": 318,
  "estimated_cost_usd": 3.4521,
  "agents_active": 12,
  "buckets": [
    { "window_start": "2026-06-05T10:00:00Z", "llm_input_tokens": 18000, "llm_output_tokens": 2400, "estimated_cost_usd": 0.092 }
  ]
}

Configure OpenTelemetry

Every Dome service exports OpenTelemetry traces, metrics, and logs over OTLP/HTTP. Point the standard OTEL_* env vars at your collector (Jaeger, Tempo, Datadog Agent, OTel Collector, Axiom, or anything that speaks OTLP) to wire Dome into your existing observability stack.

Enable

Set OTEL_EXPORTER_OTLP_ENDPOINT on the api-server and gateway processes. When the variable is unset, the SDK runs in silent no-op mode. That is useful for local development without a collector.

export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
export OTEL_SERVICE_NAME=dome-api-server      # also: dome-gateway
export OTEL_RESOURCE_ATTRIBUTES=deployment.environment=production

Health probes (/healthz, /readyz, /health, /ready) are filtered out of traces and HTTP metrics automatically.

Identity attributes

Every span and log record carries the same dome.* vocabulary so you can pivot from a trace to its logs with a single tag filter:

Logs also carry the OTel trace_id, so a single query (dome.tenant_id="...") returns the full request story across spans and logs for that tenant.

Product metrics

Eight product-level instruments emit on the global OTel meter. Counters omit tenant_id to keep cardinality bounded. Pivot per-tenant via the span attributes above.

Sampling and cost

Trace volume is head-sampled at 1% by default. Boilerplate database spans from otelpgx (transaction state, pool acquire, prepare statements) are dropped at the source. They accounted for ~99% of trace volume in early staging. Tail-sampling decisions belong at your collector.

# Local dev: full traces, full logs
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318
export OTEL_TRACES_SAMPLER_ARG=1.0

Configure product usage telemetry

Stream product-usage events to your own analytics backend to see which features each tenant exercises. Telemetry is separate from audit. It answers "which orgs are active, which features get used, where do users get stuck", not "what did this agent do".

Telemetry is opt-in and off by default. Set USAGE_TELEMETRY_ENABLED=true on the api-server and gateway processes to turn it on. Any other value, including unset, installs a no-op client and emits nothing.

Sinks

Select a sink with USAGE_TELEMETRY_SINK. Leave it unset to use the env-driven default.

A misconfigured sink (missing token, unreachable endpoint at startup, malformed headers) falls back to noop and logs the error. Startup never fails because of telemetry.

Configure

Set the telemetry envvars in the api-server and gateway environments. One client per process. The same configuration applies to both.

Local development (logger sink, visible at DEBUG):

export DOME_ENV=local
export DOME_LOG_LEVEL=debug
export USAGE_TELEMETRY_ENABLED=true

Self-hosted OTLP collector:

export USAGE_TELEMETRY_ENABLED=true
export USAGE_TELEMETRY_SINK=otlp
export USAGE_TELEMETRY_OTLP_ENDPOINT={{COLLECTOR_URL}}
export USAGE_TELEMETRY_OTLP_HEADERS=Authorization=Bearer {{COLLECTOR_TOKEN}}

Event coverage

Telemetry covers two surfaces: lifecycle events when resources are created, changed, or removed, and runtime events on every request the gateway evaluates.

Event envelope

Every event, regardless of sink, has the same normalized shape:

{
  "name": "tool_call.completed",
  "version": 1,
  "timestamp": "2026-05-15T14:22:18.041Z",
  "scope": {
    "org_id": "org_...",
    "tenant_id": "tenant_...",
    "workspace_id": "ws_..."
  },
  "entity": {
    "type": "agent",
    "id": "agent_..."
  },
  "properties": {
    "result": "success",
    "tool": "database-query",
    "backend": "my-mcp-server",
    "duration_ms": 142,
    "trace_id": "trace_..."
  }
}

Add a new field to an existing event freely. Additive changes do not bump version. Renaming or repurposing a field requires a new event name or a version bump.

PII handling

Telemetry drops sensitive properties before any sink sees them. Property keys matching email, username, password, secret, token, prompt, response, arguments, or args, and keys containing oauth_code, tool_result, user_name, raw_*, or act_as_subject, are stripped at the client. Runtime constructors never accept raw tool args, prompts, or model responses in the first place.

Putting PII under a non-matching key name does not bypass this. The package is the second line of defense.

Tuning behavior

V1 ships with fixed batching parameters: 100 events per batch, 10 s flush interval, 10,000-event in-memory queue, 5 s send timeout, 2 retries with 250 ms backoff. These are not configurable via env in V1.

• Queue full → event drops, warn log, dropped counter increments.
• Sink error → batch retries once, then drops, error log, dropped increments by batch size.
• Shutdown → queue drains, final batch flushes, sink shuts down.

Next steps

• Audit events for historical queries, exports, and the full event envelope
• Subscribe to Events to push signed events to external systems
• Set Usage Limits when quota exhaustion shows up in the stream