Skip to main content

Operations

Running a Broods deployment day to day. First-time setup is in self-hosting, how commits reach the cluster is in CI/CD, and log and trace plumbing is in observability.

Runtime topology​

The managed service runs on one k3s cluster, deployed from the infra repo (kubernetes/charts/releases/). The AWS data plane comes from apps/core/sst.config.ts, one copy per SST stage. Service names and hosts below are the ones apps/edge/src/generate.ts routes to.

PieceReplicasWhy
core1The machine sandbox registry and the in-process worker queue live in memory
gatewayanyStateless. On SIGTERM it closes sockets with 1012 so clients reconnect elsewhere
discord-forwarder1, strategy: RecreateA second pod means a second socket per token and every message answered twice
Traefik1Rate-limit buckets count in memory per pod
cloudflare-mcpone Worker per stageCore uses it only once infra sets CLOUDFLARE_MCP_URL and its key; until then Lambda
  • Core authenticates to AWS with an access key for the per-stage core-runtime IAM user. Convex uses the convex-bootstrap user, which can only assume ConvexAwsRole.
  • Core schedules nothing. The Convex crons component owns every schedule and POSTs each firing to core's /v1/cron-runs in-cluster.
  • A green image build deploys nothing. rollout.yaml dispatches the infra workflow that rolls the pod. See CI/CD.

Core runtime limits​

KnobDefaultEffect
MAX_INPROCESS_WORKERS8Runs in flight per pod, streamed runs too. The rest queue, with their lease renewed
WORKER_TIMEOUT_BUDGET_MS600000A run still going 5 s past this loses its slot and fails
MODEL_FIRST_CHUNK_TIMEOUT_MS, MODEL_CHUNK_TIMEOUT_MS300000No text, reasoning, tool input or file from the model for this long fails the run
REQUEST_TIMEOUT_BUDGET_MS600000Work deadline of one request
SHUTDOWN_DEADLINE_MS25000Drain time on SIGTERM before live runs are failed
SANDBOX_SWEEP_INTERVAL_SECONDS3600How often core releases reserved sandboxes whose conversation never came back

Time inside a tool call does not count toward the chunk timeouts, except a tool the model provider runs itself (such as its web search).

An async run on one core pod:

What a core restart does, from shutdown() and boot in apps/core/src/server.ts:

Without the interrupt, a conversation stays locked for the 15-minute lease TTL. Convex promotes each queue atomically, so an overlapping pod never runs one twice.

Edge limits​

Traefik's broods-edge IngressRoute is generated from apps/edge/src/routes.ts. A route added to core or the config plane needs a row there, then the regenerated file in the infra repo.

RouterLimit per client address
core, config plane, download links1200 a minute (burst 200)
WebSocket upgrades120 a minute (burst 60)
channel webhooks, media links, healthnone, providers share addresses

Over the limit answers 429 with CORS headers. Traefik binds the node's ports 80 and 443 (hostPort) to see each client's own address; behind a load balancer without PROXY protocol every client shares one bucket.

The gateway has its own limit: GATEWAY_AUTH_FAILURES_PER_MINUTE (default 20) failed socket logins per address. A token core could not check, on a 5xx or timeout, gets 502 and does not count.

Service secrets​

Each secret has one job and none falls back to another. Core refuses to boot without the first four, and the gateway without TERMINAL_TICKET_SECRET.

SecretJobSet onFormat
SERVICE_AUTH_SECRETService bearer, with X-Account-Id; keys the config plane's failed-login bucketscore, Convexsingle value
STAGE_TICKET_SECRETSigns bdts_ stage tickets (15 min)Convex signs, core readssingle value
TERMINAL_TICKET_SECRETSeals sandbox terminal ticketscore seals, gateway openslist
MEDIA_TICKET_SECRETSeals /v1/media/:ticket linkscorelist
ACCOUNT_CONFIG_ENCRYPTION_SECRETWraps each account's data keycore, Convexlist
ADMIN_ACCOUNT_SECRETAdmin bearer for the account admin routescore, Convex, dashboardsingle value
ANALYTICS_ID_SECRETKeys the PostHog user and AI session ids; optionalcore, Convexsingle value

A list is comma-separated: the first entry seals, every entry opens.

  • TERMINAL_TICKET_SECRET, MEDIA_TICKET_SECRET: the steps above. A media link expires on its own after MEDIA_TICKET_TTL_SECONDS (default 30 days); dropping a value stops every link it sealed at the origin, though a CDN that fetched the bytes in time can serve its copy until the ticket would have expired, which is where the route's cache-control stops it. A copy fetched before 2026-11 keeps the old one-year max-age.
  • ACCOUNT_CONFIG_ENCRYPTION_SECRET: follow the rotation runbook. It adds a rewrapAllKeys step; dropping a value before the rewrap finishes makes every config it wrapped unreadable.
  • SERVICE_AUTH_SECRET, STAGE_TICKET_SECRET: change core and Convex together. Rotating the stage secret logs out open dashboard log streams and broods logs until they mint a new ticket.
  • ANALYTICS_ID_SECRET: never rotate it for security; it guards nothing. A new value makes every user and AI session look new in PostHog.

Service token rules​

The service token never crosses the public door. Only Convex sends it, always to core's in-cluster URL (BROODS_ACCOUNT_MANAGE_URL).

  • Core (isServiceToken in apps/core/src/shared/auth.ts) and the config plane both refuse the service token on any request that carries x-broods-via-gateway. The header name is VIA_GATEWAY_HEADER in packages/convex/model/serviceBridge.ts.
  • The in-cluster-only paths /v1/cron-runs and /v1/mcp-service/rpc are routed to core like any other path; core refuses them because a stamped request can never carry a valid service token.
  • If Convex cannot reach core, fix the network; there is no public path to open. Set BROODS_ACCOUNT_MANAGE_URL to http://core.beeblast.svc.cluster.local and allow egress from the convex namespace.

Forwarders​

Discord sends regular messages only over a Gateway socket. Every other channel posts to a webhook and needs no forwarder:

  • One release serves every config plane. Plane dev reads its admin key from CONVEX_DEPLOY_KEY_DEV. Add a plane to the array, never a release per stage.
  • Convex decrypts and returns only the token and webhook path. The forwarder never holds ACCOUNT_CONFIG_ENCRYPTION_SECRET.
  • A plane that errors keeps its last answer, so a Convex blip closes no socket. /healthz never waits on Convex; /readyz is 503 until the first plane answers.
  • A webhook that still rejects an event after three tries loses it.

A Discord socket, from apps/discord-forwarder/src/socket.ts:

  • Discord resets a token after 1000 IDENTIFYs in 24 hours. DISCORD_IDENTIFY_LIMIT (default 500) parks the socket first. The counter is in memory, so a crash loop defeats it.
  • A RESUME spends no IDENTIFY and only dials a host under .discord.gg, so the bot token cannot be sent elsewhere.
  • Events go out as GATEWAY_MESSAGE_CREATE with a credential derived from the bot token (discordForwarderToken in apps/core/src/shared/discord-wire.ts) in x-discord-gateway-token, plus thread for a message in a thread. The bot token itself never crosses the edge, and core refuses it there.

Runbooks​

SymptomFix
Core exits at boot naming a secretSet the missing service secret in the pod env
New route answers 404, or reaches the wrong planebun run --filter @broods/edge generate kubernetes, land it in infra
Crons never fire, sandbox deletes leave reservationsPoint BROODS_ACCOUNT_MANAGE_URL at core in-cluster; match SERVICE_AUTH_SECRET on both sides
Every agent config fails to decryptPut the value that wrapped the keys back in ACCOUNT_CONFIG_ENCRYPTION_SECRET
deny-all or restricted lambda sandboxes fail to launchSet MICROVM_EGRESS_NETWORK_CONNECTOR_ARN from the microvmEgressNetworkConnectorArn output
Discord agent answers /new but ignores mentionsdiscord-forwarder down or missing that plane. Check /readyz
Image built but pods run the old versionCheck the build workflow's rollout job and INFRA_DISPATCH_TOKEN
broods logs or dashboard stream stops after about 15 minutesStage ticket could not be renewed. Re-run broods login

A Discord agent that stays silent:

Drift cleanup​

See CI/CD.