Operations
Running a Broods deployment day to day. First-time setup is in self-hosting, how commits reach the cluster is in CI/CD, and log and trace plumbing is in observability.
Runtime topology
The managed service runs on one k3s cluster, deployed from the infra repo (kubernetes/charts/releases/). The AWS data plane comes from apps/core/sst.config.ts, one copy per SST stage. Service names and hosts below are the ones apps/edge/src/generate.ts routes to.
| Piece | Replicas | Why |
|---|---|---|
| core | 1 | The machine sandbox registry and the in-process worker queue live in memory |
| gateway | any | Stateless. On SIGTERM it closes sockets with 1012 so clients reconnect elsewhere |
| discord-forwarder | 1, strategy: Recreate | A second pod means a second socket per token and every message answered twice |
| Traefik | 1 | Rate-limit buckets count in memory per pod |
| cloudflare-mcp | one Worker per stage | Core uses it only once infra sets CLOUDFLARE_MCP_URL and its key; until then Lambda |
- Core authenticates to AWS with an access key for the per-stage
core-runtimeIAM user. Convex uses theconvex-bootstrapuser, which can only assumeConvexAwsRole. - Core schedules nothing. The Convex crons component owns every schedule and POSTs each firing to core's
/v1/cron-runsin-cluster. - A green image build deploys nothing.
rollout.yamldispatches the infra workflow that rolls the pod. See CI/CD.
Core runtime limits
| Knob | Default | Effect |
|---|---|---|
MAX_INPROCESS_WORKERS | 8 | Runs in flight per pod, streamed runs too. The rest queue, with their lease renewed |
WORKER_TIMEOUT_BUDGET_MS | 600000 | A run still going 5 s past this loses its slot and fails |
MODEL_FIRST_CHUNK_TIMEOUT_MS, MODEL_CHUNK_TIMEOUT_MS | 300000 | No text, reasoning, tool input or file from the model for this long fails the run |
REQUEST_TIMEOUT_BUDGET_MS | 600000 | Work deadline of one request |
SHUTDOWN_DEADLINE_MS | 25000 | Drain time on SIGTERM before live runs are failed |
SANDBOX_SWEEP_INTERVAL_SECONDS | 3600 | How often core releases reserved sandboxes whose conversation never came back |
Time inside a tool call does not count toward the chunk timeouts, except a tool the model provider runs itself (such as its web search).
An async run on one core pod:
What a core restart does, from shutdown() and boot in apps/core/src/server.ts:
Without the interrupt, a conversation stays locked for the 15-minute lease TTL. Convex promotes each queue atomically, so an overlapping pod never runs one twice.
Edge limits
Traefik's broods-edge IngressRoute is generated from apps/edge/src/routes.ts. A route added to core or the config plane needs a row there, then the regenerated file in the infra repo.
| Router | Limit per client address |
|---|---|
| core, config plane, download links | 1200 a minute (burst 200) |
| WebSocket upgrades | 120 a minute (burst 60) |
| channel webhooks, media links, health | none, providers share addresses |
Over the limit answers 429 with CORS headers. Traefik binds the node's ports 80 and 443 (hostPort) to see each client's own address; behind a load balancer without PROXY protocol every client shares one bucket.
The gateway has its own limit: GATEWAY_AUTH_FAILURES_PER_MINUTE (default 20) failed socket logins per address. A token core could not check, on a 5xx or timeout, gets 502 and does not count.
Service secrets
Each secret has one job and none falls back to another. Core refuses to boot without the first four, and the gateway without TERMINAL_TICKET_SECRET.
| Secret | Job | Set on | Format |
|---|---|---|---|
SERVICE_AUTH_SECRET | Service bearer, with X-Account-Id; keys the config plane's failed-login buckets | core, Convex | single value |
STAGE_TICKET_SECRET | Signs bdts_ stage tickets (15 min) | Convex signs, core reads | single value |
TERMINAL_TICKET_SECRET | Seals sandbox terminal tickets | core seals, gateway opens | list |
MEDIA_TICKET_SECRET | Seals /v1/media/:ticket links | core | list |
ACCOUNT_CONFIG_ENCRYPTION_SECRET | Wraps each account's data key | core, Convex | list |
ADMIN_ACCOUNT_SECRET | Admin bearer for the account admin routes | core, Convex, dashboard | single value |
ANALYTICS_ID_SECRET | Keys the PostHog user and AI session ids; optional | core, Convex | single value |
A list is comma-separated: the first entry seals, every entry opens.
TERMINAL_TICKET_SECRET,MEDIA_TICKET_SECRET: the steps above. A media link expires on its own afterMEDIA_TICKET_TTL_SECONDS(default 30 days); dropping a value stops every link it sealed at the origin, though a CDN that fetched the bytes in time can serve its copy until the ticket would have expired, which is where the route's cache-control stops it. A copy fetched before 2026-11 keeps the old one-yearmax-age.ACCOUNT_CONFIG_ENCRYPTION_SECRET: follow the rotation runbook. It adds arewrapAllKeysstep; dropping a value before the rewrap finishes makes every config it wrapped unreadable.SERVICE_AUTH_SECRET,STAGE_TICKET_SECRET: change core and Convex together. Rotating the stage secret logs out open dashboard log streams andbroods logsuntil they mint a new ticket.ANALYTICS_ID_SECRET: never rotate it for security; it guards nothing. A new value makes every user and AI session look new in PostHog.
Service token rules
The service token never crosses the public door. Only Convex sends it, always to core's in-cluster URL (BROODS_ACCOUNT_MANAGE_URL).
- Core (
isServiceTokeninapps/core/src/shared/auth.ts) and the config plane both refuse the service token on any request that carriesx-broods-via-gateway. The header name isVIA_GATEWAY_HEADERinpackages/convex/model/serviceBridge.ts. - The in-cluster-only paths
/v1/cron-runsand/v1/mcp-service/rpcare routed to core like any other path; core refuses them because a stamped request can never carry a valid service token. - If Convex cannot reach core, fix the network; there is no public path to open. Set
BROODS_ACCOUNT_MANAGE_URLtohttp://core.beeblast.svc.cluster.localand allow egress from theconvexnamespace.
Forwarders
Discord sends regular messages only over a Gateway socket. Every other channel posts to a webhook and needs no forwarder:
- One release serves every config plane. Plane
devreads its admin key fromCONVEX_DEPLOY_KEY_DEV. Add a plane to the array, never a release per stage. - Convex decrypts and returns only the token and webhook path. The forwarder never holds
ACCOUNT_CONFIG_ENCRYPTION_SECRET. - A plane that errors keeps its last answer, so a Convex blip closes no socket.
/healthznever waits on Convex;/readyzis503until the first plane answers. - A webhook that still rejects an event after three tries loses it.
A Discord socket, from apps/discord-forwarder/src/socket.ts:
- Discord resets a token after 1000 IDENTIFYs in 24 hours.
DISCORD_IDENTIFY_LIMIT(default 500) parks the socket first. The counter is in memory, so a crash loop defeats it. - A RESUME spends no IDENTIFY and only dials a host under
.discord.gg, so the bot token cannot be sent elsewhere. - Events go out as
GATEWAY_MESSAGE_CREATEwith a credential derived from the bot token (discordForwarderTokeninapps/core/src/shared/discord-wire.ts) inx-discord-gateway-token, plusthreadfor a message in a thread. The bot token itself never crosses the edge, and core refuses it there.
Runbooks
| Symptom | Fix |
|---|---|
| Core exits at boot naming a secret | Set the missing service secret in the pod env |
New route answers 404, or reaches the wrong plane | bun run --filter @broods/edge generate kubernetes, land it in infra |
| Crons never fire, sandbox deletes leave reservations | Point BROODS_ACCOUNT_MANAGE_URL at core in-cluster; match SERVICE_AUTH_SECRET on both sides |
| Every agent config fails to decrypt | Put the value that wrapped the keys back in ACCOUNT_CONFIG_ENCRYPTION_SECRET |
deny-all or restricted lambda sandboxes fail to launch | Set MICROVM_EGRESS_NETWORK_CONNECTOR_ARN from the microvmEgressNetworkConnectorArn output |
Discord agent answers /new but ignores mentions | discord-forwarder down or missing that plane. Check /readyz |
| Image built but pods run the old version | Check the build workflow's rollout job and INFRA_DISPATCH_TOKEN |
broods logs or dashboard stream stops after about 15 minutes | Stage ticket could not be renewed. Re-run broods login |
A Discord agent that stays silent:
Drift cleanup
See CI/CD.