Security
What Broods stores, how secrets are protected, and how untrusted code is contained. Who can do what in the dashboard and CLI is in the security guide. The model is simple on purpose: it keeps provider secrets out of plaintext Convex rows, but it is not a final production-grade secrets system.
Trust boundaries
Everything inside "tenant code" is treated as hostile. Anything it can reach is reachable by tenant code.
What is stored
- The account key is returned once on create or rotation. Only
secretHashis kept. - Provider credentials must be usable at runtime, so they are encrypted, not hashed.
- The workspace, skills, tool-bundles and MicroVM artifact buckets block public access.
denyUnlessProjectPrincipal()inapps/core/sst.config.tsdeniess3:*to everyone but the stage'ssandbox-s3mount,microvm-buildandmicrovm-executionroles, thecore-runtimeuser, theconvex-awsrole, the GitHub Actions deploy roles and the account root. - On production, when
RUNTIME_EGRESS_IPSis set (the cluster's public address), thecore-runtimeandconvex-bootstrapusers deny every AWS call that does not come from it, so their static keys are useless off the cluster. Unset, those keys are not IP-restricted.
Encryption at rest
Envelope encryption, one codec for both sides (packages/convex/model/envelope.ts):
- The additional data binds a blob to its tenant, table and column, and the
keyIdnames the key that opens it. - Encrypted columns (
ENVELOPE_COLUMNS):agents.encryptedConfigandencryptedSourceConfig, the same two onsandboxConfigs,environmentVariables.ciphertext,accountEnvVars.ciphertext,agentRuntimeSecrets.ciphertext,agentDeployments.apiKeyCiphertext,channelEndpoints.tokenCiphertext,connections.ciphertext,mcp.connectionCiphertext,auditSinks.encryptedSecret. - The CLI compares env values by plain SHA-256, which
listEnvBySecretHashcomputes per request from the decrypted value. The stored digest is keyed, so a table dump cannot be brute-forced. - Convex runs the cipher on Web Crypto and builds the keyring per request (
model/accountKeys.ts). Core runs it onnode:crypto(src/shared/node-aead.ts) and caches unwrapped keys per account for 5 minutes (src/shared/convex/account-keys.ts), refreshing once when a row names an unknown key. The format is the same on both sides.
Rotation runbook
Both are internal mutations, run as the deployment admin. Each pages through its rows with a self-reschedule and is idempotent. Call it with no continuation arguments and wait for the scheduled batches to drain; the returned value covers only the first batch, so isDone: false is normal. A batch that throws stops the walk and shows as a failed scheduled function.
Rotate one account's DEK
- Run
bunx convex run account/keys:rotateAccountKey '{"accountId": "<id>"}'. - It mints a new key, which every write uses from then on, and rewrites every blob of the account under it, table by table.
- When the walk finishes it retires the older keys. Re-running it before then, or after a failed batch, resumes the same rotation.
- A config an HTTP action sealed under a retired key is refused with
409, and the caller retries.
Rotate the KEK
- Set
ACCOUNT_CONFIG_ENCRYPTION_SECRET=old,newon core and on Convex. Deploy both. - Set
new,oldon both. Deploy both. - Run
bunx convex run account/keys:rewrapAllKeysand wait for the batches to drain. - Check that no
accountKeysrow still carries the oldkekId. - Set
newalone on both. Deploy both.
No blob is rewritten in a KEK rotation; only the wrapped keys change. The two-step order exists because core and Convex pick up an env change at different moments.
Redaction
Reads redact secret-like field names (token, secret, privateKey, apiKey, cookie, authorization) and every headers value as ********. One rule, isSecretName in packages/convex/model/secretNames.ts, decides secret names for config reads, MCP header refs, logs and policies. A value made only of ${NAME} refs, optionally after an auth scheme word, is shown as is. Sending ******** back in a patch keeps the stored value. Logs: see observability.
Credentials
| Prefix | Credential | Scope |
|---|---|---|
bask_ | Account key | Whole tenant |
bask_ | Scoped account key | Its own policy and optional stage pin, until expiresAt. Only the hash is stored |
bcli_ | CLI login token | An org owner or admin, re-checked against membership on every request. Also the account API, never key rotation |
bsk_ | Runtime key | One account, project, stage and endpoint. Encrypted, recoverable by its owner |
bpdk_ | Project key | One project and stage |
brole_ | Role | Never used directly. Exchanged for a session |
bsts_ | Role session | The role's policy. Default 1 hour, max 12. Only the hash is stored |
bdts_ | Stage session ticket | 15 minutes, signed by Convex with STAGE_TICKET_SECRET |
bdob_ | Logs ticket | 5 minutes at most, never past its login. Reads one stage's logs and traces, nothing else |
brt_ | Run token | Reads its own agent's runs. Signed by core, never stored |
Every credential starts with b, and the broods-credential rule in .gitleaks.toml matches each full shape (brole_ ids are not secrets). Core resolves a bearer in resolveBearerAuth() (apps/core/src/shared/auth.ts):
Every branch that names an account requires it active, except the account key on DELETE /v1/account, so an owner can retry a deletion. bcli_ and bpdk_ never reach core; the config plane checks them in packages/convex/cli/http.ts, and resolveBearerAuth in config/routes/shared.ts takes a bcli_ as account auth for /v1/... in one cli/auth.resolveAccountLogin call, audited as cli with the user's auth id, which outlives the pruned token row.
| Credential | Limits the code enforces |
|---|---|
Runtime key bsk_ | Own stage only (another stage's agentId answers 404). Needs publicAccess: true. No system or model override without allowRunOverrides (403 run_overrides_disabled). continue: true only into direct-API conversations. With a user token, only that user's namespace; without one, no user's (see below). An agent with requireUserToken refuses it without a token. Opens its stage's agent WebSocket, where every frame passes these same checks; never the observability socket. |
Stage ticket bdts_ | Every agent of its stage. continue: true re-enters channel sessions too. |
Project key bpdk_ | Syncs its own stage. Cannot replace a skill or hook another stage manages. Sets and lists env vars but never reads a value. Rotates its stage's user token signing secret. |
Role session bsts_ | Cannot mint sessions, rotate the account key or touch /v1/roles. A pinned session's resource must resolve to its stage first (authorize() in packages/convex/model/apiAuthorization.ts). A disabled role kills every live session. |
Scoped key bask_ | Every role session limit, on the key's own policy and pin. The config plane and core resolve it only after the account key misses, so the CLI plane, assume-role, connections and skill downloads, which look up the account key alone, answer 401. Revoking deletes the row. The audit actor is accountApiKey with the key's id. |
Login bcli_ | broods login binds the one-time code with S256 PKCE. broods env get reveals are recorded in environmentVariableReveals. Refused for POST /v1/account/rotate-secret. The dashboard's agent dock mints one per session for an owner or admin only (the tiers that hold the whole account), expiring with its user token and marked lentToAgent. The CLI plane never resolves a lent login, and managesCredentials in config/http.ts refuses it (403) on role writes, assume-role, connection writes, PATCH /v1/account, audit writes and key rotation. broods logout revokes its own token (DELETE /v1/account/auth/token). cli/auth.pruneExpiredTokens deletes expired rows daily. |
End-user tokens
The runtime key is embeddable, so it carries no end-user identity. A buat_ user token rides beside it in X-Broods-User-Token (or the broods.user.<token> subprotocol on the agent socket, which the gateway forwards as that header on every core call).
- Format. The same signed ticket as
bdts_(sealSignedTicketinpackages/convex/model/stageSessionTicket.ts): base64url JSON{userId, conversations?, credentialSealed?, expiresAt}and an HMAC-SHA256 over it, keyed by SHA-256 ofbroods-user-token:plus the stage'sbsig_secret. The context string keeps it apart from every other ticket key.openUserToken(packages/convex/model/userToken.ts) verifies with WebCrypto, which compares in constant time, and refuses anexpiresAtin the past or more than 60 minutes out. - Secret. One per stage on its
agentDeploymentsrow, sealed under the account DEK inuserTokenSecretCiphertext(anENVELOPE_COLUMNSentry, so key rotation rewrites it).getByApiKeyHashhands core the sealed blob with the deployment; core opens it withdecryptAccountBlobonly when a request carries a token.rotateStageUserTokenSecret(agent/deployments.ts) mints and audits it for both callers: the dashboard'srotateUserTokenSecret(keys:write), and the CLI routePOST /v1/account/projects/{project}/stages/{stage}/user-token-secretthroughcli/sync.rotateUserTokenSecretBySecretHash, where an account or project key holdskeys:writeon its stage by issue and abcli_login token is checked against its member's policies. Either shows it once; afterwards only its hint, which the manifest sync'sdeployment.userTokenSecretHintcarries so the CLI can tell whether its.env.localcopy is current. The CLI mints one only when the stage has none, and only warns when its copy is missing or stale, so a CI deploy or a second checkout never voids the secret a running backend signs with. - Check.
resolveRuntimeKeyAuth(apps/core/src/shared/auth.ts) refuses the whole credential when a token does not open, so a forged token is a 401 and the gateway's auth-failure limiter counts it. A valid one lands on thedeploymentauth asuser, and itsuserIdheads the run's principal chain, so MCP servers see it inX-Broods-Principal. - Namespace.
parseDirectPayloadputs the conversation key and event id underbsu:<userId>:(USER_CONVERSATION_PREFIX, reserved for every other caller inruntime-keys.ts). A user id cannot contain:, so the split is unique.embeddableKeyReaches(apps/core/src/harness/status-access.ts) is the one rule for the runtime key: with a user only that user's keys, without one no key in any user namespace.continue, run reads and subagent reads through a parent all go through it; steering and answers go throughparseDirectPayloadwith the namespaced key. - Files.
Session.userConversationKeyis set for absu:conversation (isUserConversationKey) and handed to its subagents.isolatedWorkspaceNamespacethen mountsusers/<hash>under whatever folder the workspace's isolation picks, andreservedAgentSandboxgives a persistent agent sandbox one reservation per user conversation unlessoptions.reservationKeypins a shared one. Without this a public agent's end users would share the workspace root. - Requirement.
agents.requireUserToken, a row field rather than config so no sync rewrites it. Both inserts (agent/agents.create,ensureAgentsRowForConfig) set it; older rows have none and answer without a token until the owner flipsagent/config.setRequireUserToken. A token is honored either way. - Socket. The scope lookup returns
conversationPrefixwhen a token rode the upgrade, and the gateway subscribes NATS on the namespaced key core publishes to. - Credential. An optional
credential(printable ASCII, at most 512) is sealed intocredentialSealed: AES-256-GCM, key HKDF-SHA-256 from thebsig_secret (infobroods-user-token-credential), base64url of a random 12-byte IV then the ciphertext. A token whose credential does not decrypt does not open. Once opened it ridesDirectInboundEvent.userCredentialand theSession's private field, never an ingress envelope, ledger row or log.mcpConnectionadds it asX-Broods-User-Credentialbeside the principal headers, forhostedrows only, outside the tool-listing cache key. Subagents inherit it in memory; a rebuilt or queued run has none. - Limits. A token is a bearer until it expires: it is not bound to a client and can be replayed within its lifetime, which is why it is short. Rotating the secret voids every token at once.
Apply tickets
The dashboard's Apply deploys approved files without handing a deploy key to anyone. agentDock.apply checks the member can write the stage, mints a bdap_ ticket (model/applyTicket.ts: {accountId, projectId, stageId, authId, expiresAt}, 5 minutes, signed with STAGE_TICKET_SECRET under its own key context so it never opens as bdts_), and asks core's in-cluster POST /v1/apply-runs (service token; refused through the gateway) to run bun install and broods deploy --prune in the APPLY_SANDBOX sandbox with the ticket as BROODS_TOKEN.
- The config plane takes a
bdap_only on its own stage'smanifest,mcp-bundle-uploadsandskill-uploadsroutes, and re-checks the member's write access at each use (resolveApplyTicketAuth). It syncs fenced like a project key, and the audit row names the member. - Core runs no user code: the files run only in that sandbox, reserved per stage and
bun.lockhash so two stages never share one. The dock agent's own sandbox never sees the ticket. - The ticket is a bearer for its 5 minutes: it reaches one stage's manifest, which the member could already deploy.
The dock agent reading logs
Logs carry end users' chats, and the runtime key cannot open them. The dock agent reads them with a login lent by the signed-in owner or admin, under rules enforced in code, not in its prompt. verification/Broods/LentObservability.lean proves them over any sequence of steps.
- A separate ticket.
POST /v1/account/observability-sessiontrades a login for abdob_ticket (model/observabilityTicket.ts) under its own key context, for one stage. Core's auth resolves it to nothing, so every route refuses it except the log socket's scope check. It cannot start a run, reach a sandbox or touch the account. - It dies with the login. Its expiry is 5 minutes or the login's, whichever comes first, and the socket closes then.
- Reading makes the conversation read-only. Text in logs may try to steer the agent, so the mint marks the login's conversation in the same mutation, before any ticket exists. From then on the config plane refuses every non-
GETfrom any lent login of that conversation (observedLoginRefusal, coderead_only_after_logs), including one re-minted later. A write already past that check when logs open can still land: the agent decided on it before it could see a log line. A lent login with no conversation cannot open logs at all.dock__propose_changesstill works, since the signed-in user reviews and applies those. - No way out for what it read. The dock agent has no web tools, and its sandbox has no network and no credential.
- Audited. A conversation's first opening of each stage writes an
observability.openedevent, and so does each mint by a person's own login.
Not covered: the agent's answer text and links. A crafted log line can still try to mislead what it tells the user, which is why it must say what it read and treat logs as data.
Hosted MCP servers
Account-uploaded MCP bundles never run in the core process. They run on the tool-runner Lambda or the Cloudflare Worker; both paths are drawn in tools and MCP.
Lambda (apps/lambda/handler.mjs):
- One shared function,
tool-runner, outside a VPC, so egress is open internet. Warm environments can serve several accounts. The separation is the child process plusreapStrays, which kills every other process of the function's user before each spawn. - The child gets a scrubbed env (
PATH,HOME,TMPDIR,NODE_ENV, its timeout) and a fresh scratch dir per batch. It is containment, not a trust boundary: same OS user, so it can read the function's environment. The execution role grants only CloudWatch Logs. - The bundle arrives by presigned URL valid 120 s, reaches the child on fd 3, is sha256-checked and imported from memory. The function holds no S3 or data-plane access.
- A child serves one
accountId:agentId:sha256only, so two agents of one account never share one. Module-level state persists across that key's calls. MCP_TENANT_ISOLATION=trueon both SST and core createsmcp-runnerwithPER_TENANTtenancy and sendsaccountId:agentIdas the Lambda tenant id. It is off because AWS rejects tenancy config for this AWS account.
Cloudflare (apps/cloudflare-mcp/src/index.ts):
- Each
tenantId:sha256gets its own Dynamic Worker isolate withenv: {}, no compatibility flags and no bindings, so tenant code reads no secret. - Egress goes through
TenantOutbound, which refuses non-http(s) schemes, the runtime's own host and the bundle store's host. There is noconnect(). - The R2 copy is keyed by content hash and re-hashed against the row on every load.
Code hooks
Inline code hooks run in a V8 isolated-vm isolate. Bun cannot load isolated-vm, so core spawns Node runners (src/harness/isolate/runner/runner.mjs) and talks NDJSON frames (src/harness/frames.ts).
- Pool of
ISOLATE_WORKER_POOL_SIZE(4) warm workers, reaped afterISOLATE_WORKER_IDLE_SECONDS(120).ISOLATE_POOL=0spawns per call, at mostISOLATE_RUNNER_CONCURRENCY(8). - 128 MB and 30 s per isolate, 1 MiB output. A runner gets only
PATHand the isolate settings from core's env. - No filesystem, no imports. Network only through
ctx.fetch(runner/pinned-fetch.mjs): resolves, refuses private and metadata ranges, connects to the validated address, caps bodies at 5 MB and requests at 30 s. - A pooled runner that never says ready (missing addon, broken runner, or silent for 5 s) is spawned 3 times in all, 100 ms then 200 ms apart, then that hook is skipped. The next fire tries again. The 30 s run limit starts once the runner is ready. A one-shot runner (
ISOLATE_POOL=0) has no ready step: its 30 s starts at spawn. - A bundle is read from S3 once per account, storage key and sha256 and reused across fire points, up to 8 MiB of encoded source, least recently used out first. Only bytes matching the sha256 are kept, and the runner checks it again.
- A hook that throws or times out is skipped, so a hook deny is best effort. Anything that must hold belongs in an enforced policy.
Sandboxes
- Runs start from a cleared env: declared
envVarsplus the image's reserved vars. No account key enters a sandbox; background jobs call back with a per-job token. - Workspace mounts get one-hour STS credentials whose session policy allows only the workspace prefix. The session carries the agent or account in its name and, on the platform role,
SourceIdentityand session tags, so CloudTrail attributes every S3 call. See sandboxes and storage. isolation: "agent"gives each agent its own namespace, sandbox and mount credentials.- File tools normalize paths and reject traversal before a provider sees the command.
- E2B, Daytona and Vercel run outside the AWS boundary: isolated mounts, minimal env vars, provider-side egress controls.
More in sandboxes.
Outbound requests
User-supplied URLs must be public https before core sends credentials to them: MCP url and OAuth tokenUrl, lifecycle webhook url, channel apiUrl overrides and storage endpoint. Private, loopback, link-local and metadata addresses are refused. MCP and webhook delivery do not follow redirects.
Audit ledger
One append-only, hash-chained ledger per account in Convex (auditEvents, written only through appendAuditEvent in packages/convex/model/auditEvents.ts). It holds config mutations from every surface plus two runtime rows: run.completed, appended by the usage write in the same mutation so the per-turn Convex budget is unchanged, and tool.denied, appended when an enforcing policy stops a tool. There is no run.started, and a run cut off at pod shutdown writes no row. Rows never carry tool input, config blobs or secrets; detailsJson is capped at 8 KB.
| Part | Rule |
|---|---|
| Chain | hash is sha256 over canonical JSON (keys sorted, no whitespace, absent optionals left out) of every other field. The head is read and patched in the same mutation, so Convex OCC serializes appends. |
| Verify | Recomputes hashes and links over a range; ok covers only checkedFrom to checkedTo. A gap is reported at its seq. Without toSeq the last row must match the head, so a deleted tail shows. |
| Export | PUT /v1/audit/sink stores one webhook, its secret sealed under the account key. The cron posts up to 10 batches of 200 rows per sink per tick with X-Broods-Signature: sha256=<hmac>, 10 s timeout each, and advances exportedSeq on 2xx. |
| Retention | auditRetentionDays, 90 by default, set through PATCH /v1/account. A row the sink has not exported is never dropped. The head is never deleted, and rows go oldest first, so the kept range always verifies. |
| Access | The account key, or a role with audit:read (ledger) and audit:write (sink). Setting retention also needs account:write. |
Agent principal and run tokens
Every run acts as one agent of one account. Core builds a Principal (apps/core/src/shared/domain/principal.ts), { kind: "agent", accountId, agentId, chain }, where the chain records who asked, oldest first:
| Run | Chain |
|---|---|
| Channel turn | [{ kind: "user", id, name?, channel }] |
| Direct API | [{ kind: "api", keyKind: "account" | "deployment" }] |
| Direct API with a user token | [{ kind: "user", id: userId }, { kind: "api", keyKind: "deployment" }] |
| Cron firing | [{ kind: "api", keyKind: "cron" }] |
| Subagent, or a session-messaging run | the parent's chain, then { kind: "agent", agentId: parent } |
| Requester not known | no chain |
Core never guesses a link: a rebuilt queued envelope, an async-tool continuation or a channel re-entry without a stored sender runs with no chain, since the hash-chained ledger could never correct a guess. The link shape is principalLinkValidator in packages/convex/model/principal.ts. The principal reaches:
X-Broods-Principal is base64url JSON of the chain without display names. The hosted path puts both headers in each requests[].mcpRequest.headers, since one batch mixes requesters. A row header of either name is dropped. Neither is part of the tool-listing cache key.
A run token lets sandbox code read its agent's runs:
- The payload is
{ accountId, agentId, exp }, signed, not encrypted: no chain, user id or display name. HKDF infobroods-run-tokenkeeps it from ever opening a stage ticket. - TTL is
WORKER_TIMEOUT_BUDGET_MSplus 5 minutes, capped at 2 hours. Minting is lazy, so a run with no exec signs none. mergeSandboxEnvdrops the fourBROODS_*names from accountenvVarsand per-call env, so nothing configured can spoof them. They ride each exec, never a persistent sandbox's create-time env, and background jobs get none.- The machine socket and the account verbs refuse a run token. Log redaction strips
brt_bearers. - Not yet: starting runs with a run token. That needs the holder's narrowing (channel
denyToolsand policies, parent policies, the cron flag), a depth cap and stage scope first.
Limits
- Upload URLs for hosted MCP bundles and workspace files: 20 open grants per account per hour. Unregistered blobs are deleted after a day. A workspace file's size is read from the stored blob and refused over 512 KB.
- Anyone with
ACCOUNT_CONFIG_ENCRYPTION_SECRETand table access can decrypt config. This guards against accidental table exposure, not compromised application code. - By design there is no Secrets Manager object per account and no KMS call per config read.