Skip to main content

Security

What Broods stores, how secrets are protected, and how untrusted code is contained. Who can do what in the dashboard and CLI is in the security guide. The model is simple on purpose: it keeps provider secrets out of plaintext Convex rows, but it is not a final production-grade secrets system.

Trust boundaries​

Everything inside "tenant code" is treated as hostile. Anything it can reach is reachable by tenant code.

What is stored​

  • The account key is returned once on create or rotation. Only secretHash is kept.
  • Provider credentials must be usable at runtime, so they are encrypted, not hashed.
  • The workspace, skills, tool-bundles and MicroVM artifact buckets block public access. denyUnlessProjectPrincipal() in apps/core/sst.config.ts denies s3:* to everyone but the stage's sandbox-s3mount, microvm-build and microvm-execution roles, the core-runtime user, the convex-aws role, the GitHub Actions deploy roles and the account root.
  • On production, when RUNTIME_EGRESS_IPS is set (the cluster's public address), the core-runtime and convex-bootstrap users deny every AWS call that does not come from it, so their static keys are useless off the cluster. Unset, those keys are not IP-restricted.

Encryption at rest​

Envelope encryption, one codec for both sides (packages/convex/model/envelope.ts):

  • The additional data binds a blob to its tenant, table and column, and the keyId names the key that opens it.
  • Encrypted columns (ENVELOPE_COLUMNS): agents.encryptedConfig and encryptedSourceConfig, the same two on sandboxConfigs, environmentVariables.ciphertext, accountEnvVars.ciphertext, agentRuntimeSecrets.ciphertext, agentDeployments.apiKeyCiphertext, channelEndpoints.tokenCiphertext, connections.ciphertext, mcp.connectionCiphertext, auditSinks.encryptedSecret.
  • The CLI compares env values by plain SHA-256, which listEnvBySecretHash computes per request from the decrypted value. The stored digest is keyed, so a table dump cannot be brute-forced.
  • Convex runs the cipher on Web Crypto and builds the keyring per request (model/accountKeys.ts). Core runs it on node:crypto (src/shared/node-aead.ts) and caches unwrapped keys per account for 5 minutes (src/shared/convex/account-keys.ts), refreshing once when a row names an unknown key. The format is the same on both sides.

Rotation runbook​

Both are internal mutations, run as the deployment admin. Each pages through its rows with a self-reschedule and is idempotent. Call it with no continuation arguments and wait for the scheduled batches to drain; the returned value covers only the first batch, so isDone: false is normal. A batch that throws stops the walk and shows as a failed scheduled function.

Rotate one account's DEK

  1. Run bunx convex run account/keys:rotateAccountKey '{"accountId": "<id>"}'.
  2. It mints a new key, which every write uses from then on, and rewrites every blob of the account under it, table by table.
  3. When the walk finishes it retires the older keys. Re-running it before then, or after a failed batch, resumes the same rotation.
  4. A config an HTTP action sealed under a retired key is refused with 409, and the caller retries.

Rotate the KEK

  1. Set ACCOUNT_CONFIG_ENCRYPTION_SECRET=old,new on core and on Convex. Deploy both.
  2. Set new,old on both. Deploy both.
  3. Run bunx convex run account/keys:rewrapAllKeys and wait for the batches to drain.
  4. Check that no accountKeys row still carries the old kekId.
  5. Set new alone on both. Deploy both.

No blob is rewritten in a KEK rotation; only the wrapped keys change. The two-step order exists because core and Convex pick up an env change at different moments.

Redaction​

Reads redact secret-like field names (token, secret, privateKey, apiKey, cookie, authorization) and every headers value as ********. One rule, isSecretName in packages/convex/model/secretNames.ts, decides secret names for config reads, MCP header refs, logs and policies. A value made only of ${NAME} refs, optionally after an auth scheme word, is shown as is. Sending ******** back in a patch keeps the stored value. Logs: see observability.

Credentials​

PrefixCredentialScope
bask_Account keyWhole tenant
bask_Scoped account keyIts own policy and optional stage pin, until expiresAt. Only the hash is stored
bcli_CLI login tokenAn org owner or admin, re-checked against membership on every request. Also the account API, never key rotation
bsk_Runtime keyOne account, project, stage and endpoint. Encrypted, recoverable by its owner
bpdk_Project keyOne project and stage
brole_RoleNever used directly. Exchanged for a session
bsts_Role sessionThe role's policy. Default 1 hour, max 12. Only the hash is stored
bdts_Stage session ticket15 minutes, signed by Convex with STAGE_TICKET_SECRET
bdob_Logs ticket5 minutes at most, never past its login. Reads one stage's logs and traces, nothing else
brt_Run tokenReads its own agent's runs. Signed by core, never stored

Every credential starts with b, and the broods-credential rule in .gitleaks.toml matches each full shape (brole_ ids are not secrets). Core resolves a bearer in resolveBearerAuth() (apps/core/src/shared/auth.ts):

Every branch that names an account requires it active, except the account key on DELETE /v1/account, so an owner can retry a deletion. bcli_ and bpdk_ never reach core; the config plane checks them in packages/convex/cli/http.ts, and resolveBearerAuth in config/routes/shared.ts takes a bcli_ as account auth for /v1/... in one cli/auth.resolveAccountLogin call, audited as cli with the user's auth id, which outlives the pruned token row.

CredentialLimits the code enforces
Runtime key bsk_Own stage only (another stage's agentId answers 404). Needs publicAccess: true. No system or model override without allowRunOverrides (403 run_overrides_disabled). continue: true only into direct-API conversations. With a user token, only that user's namespace; without one, no user's (see below). An agent with requireUserToken refuses it without a token. Opens its stage's agent WebSocket, where every frame passes these same checks; never the observability socket.
Stage ticket bdts_Every agent of its stage. continue: true re-enters channel sessions too.
Project key bpdk_Syncs its own stage. Cannot replace a skill or hook another stage manages. Sets and lists env vars but never reads a value. Rotates its stage's user token signing secret.
Role session bsts_Cannot mint sessions, rotate the account key or touch /v1/roles. A pinned session's resource must resolve to its stage first (authorize() in packages/convex/model/apiAuthorization.ts). A disabled role kills every live session.
Scoped key bask_Every role session limit, on the key's own policy and pin. The config plane and core resolve it only after the account key misses, so the CLI plane, assume-role, connections and skill downloads, which look up the account key alone, answer 401. Revoking deletes the row. The audit actor is accountApiKey with the key's id.
Login bcli_broods login binds the one-time code with S256 PKCE. broods env get reveals are recorded in environmentVariableReveals. Refused for POST /v1/account/rotate-secret. The dashboard's agent dock mints one per session for an owner or admin only (the tiers that hold the whole account), expiring with its user token and marked lentToAgent. The CLI plane never resolves a lent login, and managesCredentials in config/http.ts refuses it (403) on role writes, assume-role, connection writes, PATCH /v1/account, audit writes and key rotation. broods logout revokes its own token (DELETE /v1/account/auth/token). cli/auth.pruneExpiredTokens deletes expired rows daily.

End-user tokens​

The runtime key is embeddable, so it carries no end-user identity. A buat_ user token rides beside it in X-Broods-User-Token (or the broods.user.<token> subprotocol on the agent socket, which the gateway forwards as that header on every core call).

  • Format. The same signed ticket as bdts_ (sealSignedTicket in packages/convex/model/stageSessionTicket.ts): base64url JSON {userId, conversations?, credentialSealed?, expiresAt} and an HMAC-SHA256 over it, keyed by SHA-256 of broods-user-token: plus the stage's bsig_ secret. The context string keeps it apart from every other ticket key. openUserToken (packages/convex/model/userToken.ts) verifies with WebCrypto, which compares in constant time, and refuses an expiresAt in the past or more than 60 minutes out.
  • Secret. One per stage on its agentDeployments row, sealed under the account DEK in userTokenSecretCiphertext (an ENVELOPE_COLUMNS entry, so key rotation rewrites it). getByApiKeyHash hands core the sealed blob with the deployment; core opens it with decryptAccountBlob only when a request carries a token. rotateStageUserTokenSecret (agent/deployments.ts) mints and audits it for both callers: the dashboard's rotateUserTokenSecret (keys:write), and the CLI route POST /v1/account/projects/{project}/stages/{stage}/user-token-secret through cli/sync.rotateUserTokenSecretBySecretHash, where an account or project key holds keys:write on its stage by issue and a bcli_ login token is checked against its member's policies. Either shows it once; afterwards only its hint, which the manifest sync's deployment.userTokenSecretHint carries so the CLI can tell whether its .env.local copy is current. The CLI mints one only when the stage has none, and only warns when its copy is missing or stale, so a CI deploy or a second checkout never voids the secret a running backend signs with.
  • Check. resolveRuntimeKeyAuth (apps/core/src/shared/auth.ts) refuses the whole credential when a token does not open, so a forged token is a 401 and the gateway's auth-failure limiter counts it. A valid one lands on the deployment auth as user, and its userId heads the run's principal chain, so MCP servers see it in X-Broods-Principal.
  • Namespace. parseDirectPayload puts the conversation key and event id under bsu:<userId>: (USER_CONVERSATION_PREFIX, reserved for every other caller in runtime-keys.ts). A user id cannot contain :, so the split is unique. embeddableKeyReaches (apps/core/src/harness/status-access.ts) is the one rule for the runtime key: with a user only that user's keys, without one no key in any user namespace. continue, run reads and subagent reads through a parent all go through it; steering and answers go through parseDirectPayload with the namespaced key.
  • Files. Session.userConversationKey is set for a bsu: conversation (isUserConversationKey) and handed to its subagents. isolatedWorkspaceNamespace then mounts users/<hash> under whatever folder the workspace's isolation picks, and reservedAgentSandbox gives a persistent agent sandbox one reservation per user conversation unless options.reservationKey pins a shared one. Without this a public agent's end users would share the workspace root.
  • Requirement. agents.requireUserToken, a row field rather than config so no sync rewrites it. Both inserts (agent/agents.create, ensureAgentsRowForConfig) set it; older rows have none and answer without a token until the owner flips agent/config.setRequireUserToken. A token is honored either way.
  • Socket. The scope lookup returns conversationPrefix when a token rode the upgrade, and the gateway subscribes NATS on the namespaced key core publishes to.
  • Credential. An optional credential (printable ASCII, at most 512) is sealed into credentialSealed: AES-256-GCM, key HKDF-SHA-256 from the bsig_ secret (info broods-user-token-credential), base64url of a random 12-byte IV then the ciphertext. A token whose credential does not decrypt does not open. Once opened it rides DirectInboundEvent.userCredential and the Session's private field, never an ingress envelope, ledger row or log. mcpConnection adds it as X-Broods-User-Credential beside the principal headers, for hosted rows only, outside the tool-listing cache key. Subagents inherit it in memory; a rebuilt or queued run has none.
  • Limits. A token is a bearer until it expires: it is not bound to a client and can be replayed within its lifetime, which is why it is short. Rotating the secret voids every token at once.

Apply tickets​

The dashboard's Apply deploys approved files without handing a deploy key to anyone. agentDock.apply checks the member can write the stage, mints a bdap_ ticket (model/applyTicket.ts: {accountId, projectId, stageId, authId, expiresAt}, 5 minutes, signed with STAGE_TICKET_SECRET under its own key context so it never opens as bdts_), and asks core's in-cluster POST /v1/apply-runs (service token; refused through the gateway) to run bun install and broods deploy --prune in the APPLY_SANDBOX sandbox with the ticket as BROODS_TOKEN.

  • The config plane takes a bdap_ only on its own stage's manifest, mcp-bundle-uploads and skill-uploads routes, and re-checks the member's write access at each use (resolveApplyTicketAuth). It syncs fenced like a project key, and the audit row names the member.
  • Core runs no user code: the files run only in that sandbox, reserved per stage and bun.lock hash so two stages never share one. The dock agent's own sandbox never sees the ticket.
  • The ticket is a bearer for its 5 minutes: it reaches one stage's manifest, which the member could already deploy.

The dock agent reading logs​

Logs carry end users' chats, and the runtime key cannot open them. The dock agent reads them with a login lent by the signed-in owner or admin, under rules enforced in code, not in its prompt. verification/Broods/LentObservability.lean proves them over any sequence of steps.

  • A separate ticket. POST /v1/account/observability-session trades a login for a bdob_ ticket (model/observabilityTicket.ts) under its own key context, for one stage. Core's auth resolves it to nothing, so every route refuses it except the log socket's scope check. It cannot start a run, reach a sandbox or touch the account.
  • It dies with the login. Its expiry is 5 minutes or the login's, whichever comes first, and the socket closes then.
  • Reading makes the conversation read-only. Text in logs may try to steer the agent, so the mint marks the login's conversation in the same mutation, before any ticket exists. From then on the config plane refuses every non-GET from any lent login of that conversation (observedLoginRefusal, code read_only_after_logs), including one re-minted later. A write already past that check when logs open can still land: the agent decided on it before it could see a log line. A lent login with no conversation cannot open logs at all. dock__propose_changes still works, since the signed-in user reviews and applies those.
  • No way out for what it read. The dock agent has no web tools, and its sandbox has no network and no credential.
  • Audited. A conversation's first opening of each stage writes an observability.opened event, and so does each mint by a person's own login.

Not covered: the agent's answer text and links. A crafted log line can still try to mislead what it tells the user, which is why it must say what it read and treat logs as data.

Hosted MCP servers​

Account-uploaded MCP bundles never run in the core process. They run on the tool-runner Lambda or the Cloudflare Worker; both paths are drawn in tools and MCP.

Lambda (apps/lambda/handler.mjs):

  • One shared function, tool-runner, outside a VPC, so egress is open internet. Warm environments can serve several accounts. The separation is the child process plus reapStrays, which kills every other process of the function's user before each spawn.
  • The child gets a scrubbed env (PATH, HOME, TMPDIR, NODE_ENV, its timeout) and a fresh scratch dir per batch. It is containment, not a trust boundary: same OS user, so it can read the function's environment. The execution role grants only CloudWatch Logs.
  • The bundle arrives by presigned URL valid 120 s, reaches the child on fd 3, is sha256-checked and imported from memory. The function holds no S3 or data-plane access.
  • A child serves one accountId:agentId:sha256 only, so two agents of one account never share one. Module-level state persists across that key's calls.
  • MCP_TENANT_ISOLATION=true on both SST and core creates mcp-runner with PER_TENANT tenancy and sends accountId:agentId as the Lambda tenant id. It is off because AWS rejects tenancy config for this AWS account.

Cloudflare (apps/cloudflare-mcp/src/index.ts):

  • Each tenantId:sha256 gets its own Dynamic Worker isolate with env: {}, no compatibility flags and no bindings, so tenant code reads no secret.
  • Egress goes through TenantOutbound, which refuses non-http(s) schemes, the runtime's own host and the bundle store's host. There is no connect().
  • The R2 copy is keyed by content hash and re-hashed against the row on every load.

Code hooks​

Inline code hooks run in a V8 isolated-vm isolate. Bun cannot load isolated-vm, so core spawns Node runners (src/harness/isolate/runner/runner.mjs) and talks NDJSON frames (src/harness/frames.ts).

  • Pool of ISOLATE_WORKER_POOL_SIZE (4) warm workers, reaped after ISOLATE_WORKER_IDLE_SECONDS (120). ISOLATE_POOL=0 spawns per call, at most ISOLATE_RUNNER_CONCURRENCY (8).
  • 128 MB and 30 s per isolate, 1 MiB output. A runner gets only PATH and the isolate settings from core's env.
  • No filesystem, no imports. Network only through ctx.fetch (runner/pinned-fetch.mjs): resolves, refuses private and metadata ranges, connects to the validated address, caps bodies at 5 MB and requests at 30 s.
  • A pooled runner that never says ready (missing addon, broken runner, or silent for 5 s) is spawned 3 times in all, 100 ms then 200 ms apart, then that hook is skipped. The next fire tries again. The 30 s run limit starts once the runner is ready. A one-shot runner (ISOLATE_POOL=0) has no ready step: its 30 s starts at spawn.
  • A bundle is read from S3 once per account, storage key and sha256 and reused across fire points, up to 8 MiB of encoded source, least recently used out first. Only bytes matching the sha256 are kept, and the runner checks it again.
  • A hook that throws or times out is skipped, so a hook deny is best effort. Anything that must hold belongs in an enforced policy.

Sandboxes​

  • Runs start from a cleared env: declared envVars plus the image's reserved vars. No account key enters a sandbox; background jobs call back with a per-job token.
  • Workspace mounts get one-hour STS credentials whose session policy allows only the workspace prefix. The session carries the agent or account in its name and, on the platform role, SourceIdentity and session tags, so CloudTrail attributes every S3 call. See sandboxes and storage.
  • isolation: "agent" gives each agent its own namespace, sandbox and mount credentials.
  • File tools normalize paths and reject traversal before a provider sees the command.
  • E2B, Daytona and Vercel run outside the AWS boundary: isolated mounts, minimal env vars, provider-side egress controls.

More in sandboxes.

Outbound requests​

User-supplied URLs must be public https before core sends credentials to them: MCP url and OAuth tokenUrl, lifecycle webhook url, channel apiUrl overrides and storage endpoint. Private, loopback, link-local and metadata addresses are refused. MCP and webhook delivery do not follow redirects.

Audit ledger​

One append-only, hash-chained ledger per account in Convex (auditEvents, written only through appendAuditEvent in packages/convex/model/auditEvents.ts). It holds config mutations from every surface plus two runtime rows: run.completed, appended by the usage write in the same mutation so the per-turn Convex budget is unchanged, and tool.denied, appended when an enforcing policy stops a tool. There is no run.started, and a run cut off at pod shutdown writes no row. Rows never carry tool input, config blobs or secrets; detailsJson is capped at 8 KB.

PartRule
Chainhash is sha256 over canonical JSON (keys sorted, no whitespace, absent optionals left out) of every other field. The head is read and patched in the same mutation, so Convex OCC serializes appends.
VerifyRecomputes hashes and links over a range; ok covers only checkedFrom to checkedTo. A gap is reported at its seq. Without toSeq the last row must match the head, so a deleted tail shows.
ExportPUT /v1/audit/sink stores one webhook, its secret sealed under the account key. The cron posts up to 10 batches of 200 rows per sink per tick with X-Broods-Signature: sha256=<hmac>, 10 s timeout each, and advances exportedSeq on 2xx.
RetentionauditRetentionDays, 90 by default, set through PATCH /v1/account. A row the sink has not exported is never dropped. The head is never deleted, and rows go oldest first, so the kept range always verifies.
AccessThe account key, or a role with audit:read (ledger) and audit:write (sink). Setting retention also needs account:write.

Agent principal and run tokens​

Every run acts as one agent of one account. Core builds a Principal (apps/core/src/shared/domain/principal.ts), { kind: "agent", accountId, agentId, chain }, where the chain records who asked, oldest first:

RunChain
Channel turn[{ kind: "user", id, name?, channel }]
Direct API[{ kind: "api", keyKind: "account" | "deployment" }]
Direct API with a user token[{ kind: "user", id: userId }, { kind: "api", keyKind: "deployment" }]
Cron firing[{ kind: "api", keyKind: "cron" }]
Subagent, or a session-messaging runthe parent's chain, then { kind: "agent", agentId: parent }
Requester not knownno chain

Core never guesses a link: a rebuilt queued envelope, an async-tool continuation or a channel re-entry without a stored sender runs with no chain, since the hash-chained ledger could never correct a guess. The link shape is principalLinkValidator in packages/convex/model/principal.ts. The principal reaches:

X-Broods-Principal is base64url JSON of the chain without display names. The hosted path puts both headers in each requests[].mcpRequest.headers, since one batch mixes requesters. A row header of either name is dropped. Neither is part of the tool-listing cache key.

A run token lets sandbox code read its agent's runs:

  • The payload is { accountId, agentId, exp }, signed, not encrypted: no chain, user id or display name. HKDF info broods-run-token keeps it from ever opening a stage ticket.
  • TTL is WORKER_TIMEOUT_BUDGET_MS plus 5 minutes, capped at 2 hours. Minting is lazy, so a run with no exec signs none.
  • mergeSandboxEnv drops the four BROODS_* names from account envVars and per-call env, so nothing configured can spoof them. They ride each exec, never a persistent sandbox's create-time env, and background jobs get none.
  • The machine socket and the account verbs refuse a run token. Log redaction strips brt_ bearers.
  • Not yet: starting runs with a run token. That needs the holder's narrowing (channel denyTools and policies, parent policies, the cron flag), a depth cap and stage scope first.

Limits​

  • Upload URLs for hosted MCP bundles and workspace files: 20 open grants per account per hour. Unregistered blobs are deleted after a day. A workspace file's size is read from the stored blob and refused over 512 KB.
  • Anyone with ACCOUNT_CONFIG_ENCRYPTION_SECRET and table access can decrypt config. This guards against accidental table exposure, not compromised application code.
  • By design there is no Secrets Manager object per account and no KMS call per config read.