Dome Systems
Dashboard
DocsLearnHands-on LabsHelpCLIAPIMCPSDKs
Production

Deploy an always-on agent

Build and deploy a governed HTTP service that stays ready for requests

An always-on agent has an address. A person, webhook, or application sends it a request; the service decides what work to do and answers on that request's timeline. It must be ready for concurrent traffic, expose health separately from work, and survive instances being replaced underneath it.

This lab builds an incident-desk API. Each POST /incidents request makes one governed tool call through Dome and returns the result. You will containerize it, keep its agent key in a secret manager, deploy it as a private HTTP service, invoke it, and find the call in Audit.

Hand this to an AI agent. It builds the HTTP service, provisions its Dome identity, deploys it, invokes it, and verifies the governed call.

Open in Cursor

To do this, you will:

Build the HTTP process

Create a health endpoint and a request handler that makes one governed tool call.

Provision the agent

Give the process one identity and one allowed tool.

Build the image

Produce an immutable container and push it to a registry.

Inject the credential

Store the long-lived agent key outside the image.

Deploy and invoke

Run the private service, send a request, and inspect Audit.

Roll back and clean up

Prove the service can move between revisions without changing its Dome identity.

Why this application stays available

The incident desk answers requests initiated by another system. It cannot know when the next request will arrive, and the caller expects an HTTP response. That makes a request-serving process the right shape:

  • start once and listen on $PORT
  • handle more than one request per process
  • keep request state local and short-lived
  • keep durable state in external systems
  • expose a cheap liveness endpoint
  • expect instances to start, stop, and overlap during rollout

Cloud Run may scale the service to zero between requests. “Always-on” here describes the application contract—an address that accepts requests—not a promise that one VM runs forever.

This lab uses Google Cloud Run to make the steps executable. The same process can run on Amazon ECS Express Mode, Azure Container Apps, Fly.io, or Render. Translate the registry, secret, service, and identity commands; keep the process contract and Dome configuration.

Prerequisites

You will need:

  • Docker with Buildx
  • the Google Cloud CLI, authenticated with gcloud auth login
  • a non-production Google Cloud project where you may use Cloud Build, Artifact Registry, Secret Manager, IAM, and Cloud Run
  • the Dome CLI, signed in with permission to provision a sandbox, attach a tool, register an agent, and deploy Rules

Set the Google Cloud values used throughout:

export PROJECT_ID="<non-production-project>"
export REGION="us-central1"
export REPOSITORY="agent-labs"
export SERVICE="incident-desk"
export SERVICE_ACCOUNT="incident-desk"

gcloud config set project "$PROJECT_ID"
gcloud auth list

Confirm the project is not production before continuing.

Build the HTTP process

Create an empty project:

mkdir incident-desk
cd incident-desk

Create the dependencies:

requirements.txt
fastapi
httpx
uvicorn

Create app.py. The application exchanges its long-lived agent API key for a short-lived bearer token on each request, then calls one MCP tool through its Gateway:

app.py
import json
import os

import httpx
from fastapi import FastAPI, HTTPException

app = FastAPI()

CONTROL_PLANE = os.environ["DOME_CONTROL_PLANE"].rstrip("/")
GATEWAY_MCP_URL = os.environ["DOME_GATEWAY_MCP_URL"]
AGENT_API_KEY = os.environ["DOME_AGENT_API_KEY"]


def exchange_token() -> str:
    response = httpx.post(
        f"{CONTROL_PLANE}/dome.identity.v1.Identity/ExchangeToken",
        json={"grantType": "api_key", "apiKey": AGENT_API_KEY},
        headers={"Content-Type": "application/json"},
        timeout=10,
    )
    response.raise_for_status()
    return response.json()["accessToken"]


def parse_mcp(response: httpx.Response) -> dict:
    if "text/event-stream" not in response.headers.get("content-type", ""):
        return response.json()
    for line in response.text.splitlines():
        if line.startswith("data:"):
            message = json.loads(line[5:].strip())
            if "result" in message or "error" in message:
                return message
    raise ValueError("MCP stream ended without a response")


def call_incident_tool() -> dict:
    response = httpx.post(
        GATEWAY_MCP_URL,
        json={
            "jsonrpc": "2.0",
            "id": 1,
            "method": "tools/call",
            "params": {
                "name": "demo-ops/it/get_incidents",
                "arguments": {},
            },
        },
        headers={
            "Authorization": f"Bearer {exchange_token()}",
            "Content-Type": "application/json",
            "Accept": "application/json, text/event-stream",
        },
        timeout=20,
    )
    response.raise_for_status()
    return parse_mcp(response)


@app.get("/health")
def health() -> dict:
    return {"status": "ok"}


@app.post("/incidents")
def incidents() -> dict:
    try:
        result = call_incident_tool()
    except (httpx.HTTPError, KeyError, ValueError) as exc:
        raise HTTPException(status_code=502, detail="governed tool call failed") from exc
    if "error" in result:
        raise HTTPException(status_code=403, detail=result["error"])
    return result["result"]

/health proves only that the process can answer. It deliberately does not exchange a token or call the tool; liveness checks should not generate agent activity or restart a healthy process because one dependency is down.

Create the container:

Dockerfile
FROM python:3.13-slim

ENV PYTHONUNBUFFERED=1
WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app.py .

CMD ["sh", "-c", "uvicorn app:app --host 0.0.0.0 --port ${PORT:-8080}"]

Exclude local credentials and build output:

.dockerignore
.git
.env
.venv
__pycache__
*.pyc

Provision the agent

Create a Dome sandbox for the deployed service:

dome sandbox provision --scope=workspace --workspace-name deploy-http
dome context sync
dome context use sandbox-deploy-http
dome context current

Stop if the workspace does not start with sandbox-.

Attach the public demo backend:

dome tool add \
  --name demo-ops \
  --url https://demo-mcp.domesystems.ai/mcp \
  --protocol streamable-http \
  --auth-method none \
  --gateway Default

Register the workload identity and admit it to the Default Gateway:

dome agents register --name incident-desk-api --if-not-exists
dome gateway access grant Default incident-desk-api

Create incident-desk.cedar. Discovery and one incident read are allowed; everything else is forbidden:

incident-desk.cedar
permit(
  principal is Dome::Agent,
  action == Dome::Action::"mcp:discover",
  resource
);

permit(
  principal is Dome::Agent,
  action == Dome::Action::"mcp:call",
  resource == Dome::MCPTool::"demo-ops/it/get_incidents"
);

forbid(
  principal is Dome::Agent,
  action == Dome::Action::"mcp:call",
  resource
) unless {
  resource == Dome::MCPTool::"demo-ops/it/get_incidents"
};

Deploy and simulate it:

dome rules apply incident-desk.cedar \
  --agent incident-desk-api \
  --name incident-desk-api

dome rules simulate --agent incident-desk-api --action mcp:call \
  --resource demo-ops/it/get_incidents --resource-type mcp_tool

dome rules simulate --agent incident-desk-api --action mcp:call \
  --resource demo-ops/finance/get_salary --resource-type mcp_tool

Expect ALLOW, then DENY.

Create the long-lived API key. It is an exchange credential, not a bearer token to send directly to the Gateway:

dome agents create-key incident-desk-api --name cloud-run

Keep the one-time Token: ... value available for the Secret Manager step without saving it in source.

Read the control-plane and Gateway endpoints:

dome auth status
dome agents get incident-desk-api --gateway Default

Set:

export DOME_CONTROL_PLANE="https://api.domesystems.ai"
export DOME_GATEWAY_MCP_URL="https://<gateway-host>/gateways/<gateway-id>/mcp"

Use the server and gateway returned by your active environment rather than these example hosts when they differ.

Test the container locally

Read the API key without echoing it:

read -rsp "Dome agent API key: " DOME_AGENT_API_KEY
echo

Build and run:

docker build -t incident-desk:local .

docker run --rm -p 8080:8080 \
  -e DOME_AGENT_API_KEY \
  -e DOME_CONTROL_PLANE \
  -e DOME_GATEWAY_MCP_URL \
  incident-desk:local

In another terminal:

curl --fail http://localhost:8080/health
curl --fail -X POST http://localhost:8080/incidents

The first call returns {"status":"ok"} and creates no Dome activity. The second returns incident data and records a tool.call.

Build and push the image

Enable the APIs and create an Artifact Registry repository:

gcloud services enable \
  artifactregistry.googleapis.com \
  cloudbuild.googleapis.com \
  run.googleapis.com \
  secretmanager.googleapis.com

gcloud artifacts repositories create "$REPOSITORY" \
  --repository-format=docker \
  --location="$REGION" \
  --description="Dome agent lab images"

Use an immutable tag:

export IMAGE_TAG="$(git rev-parse --short HEAD 2>/dev/null || date +%Y%m%d%H%M%S)"
export IMAGE="$REGION-docker.pkg.dev/$PROJECT_ID/$REPOSITORY/$SERVICE:$IMAGE_TAG"

gcloud builds submit --tag "$IMAGE" .

The image contains code and dependency metadata only. DOME_AGENT_API_KEY was runtime input and never entered the build context.

Store the agent key

Create a dedicated runtime service account:

gcloud iam service-accounts create "$SERVICE_ACCOUNT" \
  --display-name="Incident desk Cloud Run service"

export SERVICE_ACCOUNT_EMAIL="$SERVICE_ACCOUNT@$PROJECT_ID.iam.gserviceaccount.com"

Write the key directly from the shell variable into Secret Manager:

printf '%s' "$DOME_AGENT_API_KEY" | \
  gcloud secrets create incident-desk-agent-key --data-file=-

unset DOME_AGENT_API_KEY

Allow only the runtime identity to read it:

gcloud secrets add-iam-policy-binding incident-desk-agent-key \
  --member="serviceAccount:$SERVICE_ACCOUNT_EMAIL" \
  --role="roles/secretmanager.secretAccessor"

Deploy and invoke

Deploy a private service from the immutable image:

gcloud run deploy "$SERVICE" \
  --image="$IMAGE" \
  --region="$REGION" \
  --service-account="$SERVICE_ACCOUNT_EMAIL" \
  --set-secrets="DOME_AGENT_API_KEY=incident-desk-agent-key:1" \
  --set-env-vars="DOME_CONTROL_PLANE=$DOME_CONTROL_PLANE,DOME_GATEWAY_MCP_URL=$DOME_GATEWAY_MCP_URL" \
  --no-allow-unauthenticated

Get its URL:

export SERVICE_URL="$(
  gcloud run services describe "$SERVICE" \
    --region="$REGION" \
    --format='value(status.url)'
)"

Invoke it with your Google identity:

export GOOGLE_ID_TOKEN="$(gcloud auth print-identity-token)"

curl --fail \
  -H "Authorization: Bearer $GOOGLE_ID_TOKEN" \
  "$SERVICE_URL/health"

curl --fail -X POST \
  -H "Authorization: Bearer $GOOGLE_ID_TOKEN" \
  "$SERVICE_URL/incidents"

If Cloud Run returns 403, your Google identity needs roles/run.invoker on this service. Do not make the service public to bypass that boundary.

Inspect the service logs without exposing the secret:

gcloud run services logs read "$SERVICE" \
  --region="$REGION" \
  --limit=20

Then inspect the governed call:

dome audit query --types tool.call --limit 10

The Cloud Run request log proves the HTTP process ran. The Dome event proves incident-desk-api called demo-ops/it/get_incidents through its one-tool boundary. You need both records to investigate the full request.

Roll out and roll back

Each Cloud Run deploy creates a revision. Make a harmless response change, build a second immutable tag, and deploy it with the same service account, secret, and Dome endpoints.

List revisions:

gcloud run revisions list \
  --service="$SERVICE" \
  --region="$REGION"

To roll all traffic back:

gcloud run services update-traffic "$SERVICE" \
  --region="$REGION" \
  --to-revisions="<previous-revision>=100"

The revision changes the code carrying the agent. It does not create another Dome agent or widen its rule. Stable identity across replaceable revisions is the deployment property this lab is proving.

Clean up

Delete the deployment first, then the secret and image repository:

gcloud run services delete "$SERVICE" --region="$REGION"
gcloud secrets delete incident-desk-agent-key
gcloud artifacts repositories delete "$REPOSITORY" --location="$REGION"
gcloud iam service-accounts delete "$SERVICE_ACCOUNT_EMAIL"

Delete the Dome sandbox:

dome workspace delete sandbox-deploy-http

Next steps

You deployed a request-serving agent whose code is replaceable, identity is stable, credential is external to the image, and tool boundary is enforced at the Gateway.

On this page

Why this application stays available
Prerequisites
Build the HTTP process
Provision the agent
Test the container locally
Build and push the image
Store the agent key
Deploy and invoke
Roll out and roll back
Clean up
Next steps
Was this page helpful?
CLIAPIMCP