Sandboxes
How core runs sandbox tools on each provider. Users configure sandboxes through the sandboxes guide. Paths are relative to apps/core/src/harness/sandbox/ unless noted.
Where sandboxes run
Core owns every call. Providers only run commands; reservations, ids, env and limits stay in core and Convex.
lambdais the provider string for the AWS Lambda MicroVM backend. Its guest image is the Rust server in the sibling repo../lambda-sanbdox.- Only MicroVM guest logs take the CloudWatch path. The forwarder labels a line by tenant only when core signed its stream name, see observability.
Executor model
Every sandbox tool compiles to one run on the provider's executor. index.ts maps provider name to factory in EXECUTORS. The names are SANDBOX_PROVIDERS in packages/convex/model/sandboxProviders.ts, the one list the Convex validator, core and the SDK derive from.
- The budget wrapper also guards
runBackground,resumeandpostReserved. SandboxCapacityErrorcomes only from a refused create: MicroVMInsufficientCapacityException,ServiceQuotaExceededException,ThrottlingException,TooManyRequestsException, and the workdir and Daytona admission refusals. Validation refuses a fallback on apersistentconfig, a fallback equal toprovider, and amachineorcustomfallback.- Limits come from
packages/convex/model/sandboxRules.ts:timeout30 s default, capped byWORKSPACE_SANDBOX_MAX_TIMEOUT_SECONDSandWORKSPACE_SANDBOX_LAMBDA_MAX_TIMEOUT_SECONDS; output capped byWORKSPACE_SANDBOX_MAX_OUTPUT_LIMIT_BYTES. A blocking call also stays insideREQUEST_TIMEOUT_BUDGET_MSinsrc/server.ts. Background jobs are bound by neither. - Env:
mergeSandboxEnv()inutils.tsstarts from the account'senvVars. Only a harness shell command onlambdaadds a per-call layer (requestEnv), andRESERVED_SANDBOX_ENV_KEYS(PATH,HOME,LD_PRELOAD,NODE_OPTIONS, the__CB_*callback slots and the rest) are dropped from it. TheBROODS_*identity names are dropped from both, then the run'sBROODS_*identity is set on top. - The exec wire contract a MicroVM image, the Cloudflare bridge and a
customserver share isSandboxExecRequest/SandboxExecResponseinsrc/shared/domain/sandbox-config.ts.parseExecResponseandexecRunResultinutils.tsread it. - CPU:
sandboxreports CPU from the workdir cgroup andlambdafrom the image'sgetrusage.src/harness/usage-metering.tssums it per task into thesandboxUsagefield of the run'staskUsagerow and putstool.compute.typeandtool.compute.cpu_usecon thetool.callspan. Other providers report none.
Contribute a provider
- Write
src/harness/sandbox/<name>-executor.tsimplementingSandboxExecutorfromtypes.ts. Onlyrunis required; reservation, jobs and lifecycle methods are feature-detected. - Add the name to
SANDBOX_PROVIDERS. Add it toSTATELESS_SANDBOX_PROVIDERStoo if Broods never reserves it. - Import the file in
index.tsand add it toEXECUTORS. The build fails until you do, and the import is what pulls the file into the compiled binary. - Put option validation in
sandboxRules.ts, and say inrunsOnOwnCredentials(src/shared/workspaces.ts) whether the platform meters it.
Capability matrix
| Provider | Executor | Workspace mount | Persistent | Suspend, resume | Snapshot | Background jobs |
|---|---|---|---|---|---|---|
lambda | microvm-executor.ts | mount-s3 in the /run hook | idle suspend, 8 h max | yes | yes | yes, logs and stop |
sandbox | workdir-executor.ts | mount-s3 per run | native pause, resume, standby | yes | yes | yes, logs and stop |
daytona | daytona-executor.ts | mount-s3 with mountAwsS3Buckets | native autoStopInterval | no | yes | yes, logs and stop |
e2b | e2b-executor.ts | rejected | native pause on timeout | no | yes | launch and callback, no logs, stop |
vercel | vercel-executor.ts | mount-s3 per run | named sandbox | no | yes | yes, logs and stop |
cloudflare | cloudflare-executor.ts | rejected | same Container while warm, no disk | no | no | no |
worker | worker-executor.ts | rejected | files stored in R2 between calls | no | no | no |
machine | machine-executor.ts | rejected | no | no | no | no |
custom | http-executor.ts | rejected | no | no | no | no |
The detached workspace job needs a persistent workspace sandbox. E2B instead runs background: true natively on a persistent sandbox, with a completion callback but no logs or stop.
Stateless call
A run with persistent unset creates a machine, runs once and tears it down. A platform-paid one gets an ephemeral sandboxInstances row for the length of the call, which is what the meter bills.
- Every write for an ephemeral row goes through its row lane (see Instance row), so a removal never lands before the upsert.
- Cloudflare writes the row before the exec that starts the Container, so its start is billed too. The MicroVM writes it after
RunMicrovmreturns. - A lost row is billed no further than the provider's max call timeout plus 2 minutes (
ephemeralSandboxMaxMsinpackages/convex/model/usageMeter.ts).sweepStaleEphemeral, started by the hourlyaccrueRecentcron, deletes it after that. - An
workerrow bills no time at all. Nothing runs between its commands, so the executor meters each command as it ends, the way a hosted MCP invoke is: its time on the Worker, waits included, asworkerMsand oneworkerRequests, priced at Cloudflare's CPU-ms and request rates (packages/convex/model/pricing.ts, a slightly high estimate since a Worker cannot read its own CPU time) and shown inside Sandbox time. A call the Worker refused before its isolate ran (a 400, 401 or 404, a 410 for a released sandbox, or Cloudflare's own error) is not metered; a 409 means the command ran and lost the store race, so it is. - A sandbox on the account's own credentials gets no row. No lifecycle verb accepts an ephemeral row.
Reservations
A persistent: true config reserves one machine per key. Keys are hashed and account-scoped in src/shared/workspaces.ts, so no key an author writes can name another account's machine.
Harness adapters pick their own key, see harness adapters.
First call and resume
The MicroVM executor (#acquire in microvm-executor.ts) as the example. Workdir, Daytona, E2B, Vercel and Cloudflare follow the same claim and race rules.
- A slow resume or a control-plane error propagates. Only a VM unknown to the provider or terminal is replaced, since replacing a suspended one leaks it against the account's memory quota.
- A persistent MicroVM counts against that quota while running or suspended. Too many make every launch fail with
ServiceQuotaExceededException, and a reserved sandbox has no fallback. lifecycle.maxLifetimeSecondsis checked on acquire, never on a timer, so it cannot interrupt a running command.
Reservation row
sandboxReservations is the authoritative key to provider id map. Every write is conditional on the expected id, so a stale caller never removes a machine another run replaced.
- TTL: 7 days after last use (
SANDBOX_RESERVATION_TTL_SECONDSinpackages/convex/runtime.ts), one day for a harness reservation on a conversation key (ISOLATED_SANDBOX_RELEASE_SECONDSinsrc/harness/ai-sdk-harness/session.ts). src/shared/sandbox-sweeper.tsruns hourly (SANDBOX_SWEEP_INTERVAL_SECONDS) under a 5 minute lease, first after a random delay under 30 s. It pages expired reservations, plus mirror rows no reservation names, throughreleaseExpiredSandboxes()insrc/shared/sandbox-cleanup.ts. The orphan scan reads idle mirrors oldest first, page by page, until it has a page of orphans or reaches rows still in use, so mirrors whose deferred reservation keeps them reserved cannot hide the orphans behind them.- A release goes through the provider that reserved the machine, not the config's current one, so switching provider or turning
persistentoff strands nothing. It uses the reserving config's credentials when the instance row saysownCredentials, else the platform's. - A sandbox config that still holds an instance is kept when its CLI resource or canvas card goes, so an instance never names a missing config.
Instance row
sandboxInstances mirrors each reserved machine for the dashboard. Core writes the steady states. The dashboard's suspend action parks the row in suspending (packages/convex/sandbox/public.ts).
- Refresh removes the row when the provider reports nothing or
terminating, soterminatingis never stored by core. The removal names the machine it read, so a replacement's row under the same key stays. - Core sends every row write through
src/harness/sandbox/mirror-writes.ts: executor upserts on start and on every command, E2B's size patch, MicroVM burst reports, theexec,terminal,refresh,suspendandresumestatus writes, and every removal. Executors never wait for their writes. - Each row has a lane, keyed on the account and the reservation key (an ephemeral row's key is its sandbox id). A row's writes land in the order they were queued. A write of the same kind to the same machine as the lane's last pending write takes its place (an upsert keeps the run fields the newer one lacks; a status write never does, so a suspend and the resume after it both land), so a backlog holds only the latest; nothing else is dropped or reordered. Memory follows rows, not writes.
- An unchanged refresh is sent at most once a minute per row (
MIRROR_REFRESH_EVERY_MS), or once per fifteenth of the sandbox's idle timeout when that is shorter. A Convex write costs about the same CPU whatever it changes (about 25 ms on the local stack), so at one command every 5 s core asks for 12 times fewer writes. 1,000 such sandboxes on one account would ask for 200 writes a second: sent one at a time, about 60 a second land and Convex uses about 1.5 cores; with unchanged refreshes skipped, 17 a second land and it uses about 0.4. A refresh that differs in any field (a new run, machine or size), or that follows a status write or a removal, is sent at once. A refresh counts as sent only once Convex wrote the row: one that failed, timed out or was refused is sent again on the next command. The row's last-used time can trail the last command by up to that window, so an idle tail bills up to that much less. - An account sends one write at a time. Parallel writes on one account conflict on its shared usage rows in Convex (
OptimisticConcurrencyControlFailure) and fail: on the local stack 158 of 200 first writes sent at once were lost, 3 of 200 with two in parallel, none one at a time (about 15 ms each). The account's lanes take turns, and a lane holding a status, a removal or a MicroVM burst report goes first, so a terminate waits for about two writes and a burst reaches its row before a removal queued after it. Under overload a row's refreshes get staler; the queue never grows past a few writes per row. Accounts run in parallel. - A write that has not answered after 10 s (
MIRROR_WRITE_TIMEOUT_MS) frees the account's sender and logs, with the lane count, pending writes and oldest pending age; so does one that waited over 2 s. Shutdown'sdrainInFlightwaits for every account's sender. - The sender only orders one pod's writes. The reservation check in
upsertis what keeps a late write from another pod, such as the old pod of a rolling update, or one that timed out but landed, from bringing a released row back. A late write for an ephemeral row, which has no reservation, can bring it back untilsweepStaleEphemeral. - A reserved row is written only while its reservation still names the machine (
upsertinpackages/convex/sandbox/instances.ts). Every release drops the reservation before the row goes, so a write that lands after it, such as a Cloudflare or Worker command that answered core after a terminate, is dropped instead of bringing back a row that bills a machine that is gone, or pointing a key reserved again at its old machine. Ephemeral rows need no reservation. specsis what the meter bills.specsVerifiedis set only when the size is real: a provider report, or the size Broods sets on workdir, MicroVM and Cloudflare. Without it the dashboard shows?.- Executors queue a refresh on every command, and core sends the unchanged ones once a minute. A refresh of a machine still running at the size, credentials and project it was last billed at bills at most once a minute (
REFRESH_BILL_EVERY_MSinpackages/convex/sandbox/instances.ts), so an owner's concurrent tasks write the shared usage rows about once a minute each, not on every command. The next bill, status change or removal covers the stretch, so nothing is lost; like the idle tail, it lands on the day and month of the bill that covers it, and live usage can trail by up to a minute.
Lifecycle actions and terminal
The dashboard and the account API call POST /v1/sandboxes/:id/:action in src/accounts/handler.ts, with action suspend, resume, terminate, snapshot, refresh, exec or terminal. Each writes a sandboxAuditEvents row naming the actor. Only the config plane, on the service token, may name the dashboard user it acts for in the body's actor; any other caller is recorded as its own credential, service, with the role id for a scoped key or role session. exec takes at most 20,000 characters; the dashboard calls it with a 30 s timeout and 64 KiB output.
terminal opens a real TTY on sandbox, lambda and cloudflare. Other providers answer it as unsupported and the dashboard keeps the bounded exec runner.
| Provider | Upstream | Credential in the ticket |
|---|---|---|
sandbox | workdir PTY WebSocket | one-time terminal token the workdir host minted, as bearer |
lambda | MicroVM native shell (SHELL_INGRESS) | CreateMicrovmShellAuthToken JWE, 30 min, X-aws-proxy-auth |
cloudflare | bridge GET /v1/sandboxes/:id/terminal | terminal token for that sandbox as bearer, ticket TTL |
- The ticket is AES-256-GCM sealed (
src/shared/terminal-ticket.ts), so the browser holds only an opaque token. If NATS is down the upgrade answers 502 rather than skip the spend check. - A Cloudflare Container that is not running answers 409; a command has to start it first.
- The Cloudflare terminal token (
apps/cloudflare-sandbox/src/terminal-token.ts) is an HMAC of the sandbox id and expiry, signed with the bridge API key. The bridge accepts it only on/terminalfor that sandbox id and until it expires, so the gateway never holds the key that reaches every sandbox. - For workdir, core asks the host for a token with
POST /v1/sandboxes/:id/pty-token, on the same URL and API key as its other calls (an account's ownworkdirUrlandapiKeywhen set). The host keeps the token only in memory and accepts it once, on that sandbox's PTY, for 120 seconds; the ticket keeps its own two-minute life, whatever the host answers. A terminal token lets its holder run commands in that one sandbox for up to two minutes, once. A host without the route answers 404, and the terminal answers 409 saying the workdir host needs updating; core never falls back to the API key.
Background jobs
bash { background: true } on a persistent workspace sandbox launches a detached setsid session and returns a statusId, the model-facing name of the runtimeAsyncToolResults row's resultId.
The row exists before the launch, so a fast job's callback never arrives first. A settled row answers 409; an unknown id or wrong token answers 404, so the route is not a token oracle. The continuation queues behind any live turn, see queue and steer.
The job as statusScript in jobs.ts reads its marker files:
The runtimeAsyncToolResults row the model sees through async_status:
lambda, workdir, Daytona and Vercel keep markers under<workspace root>/.fp-jobs/<reservation key>, at most 10 running (MAX_CONCURRENT_BACKGROUND_JOBS). E2B uses its nativebackground: truecommand, so it has callback delivery but no logs or stop.- A MicroVM resume keeps the boot id, so a resumed job still reads as running. A recreate or scale-to-zero changes it, so the job reads as failed.
- The callback runs the image's
python3and needs egress toPUBLIC_BASE_URL. Without it the job still runs and polling still works. Idle scale-down never pauses a sandbox with a running job.
Lambda MicroVM
Each reserved key runs in one AWS Lambda MicroVM, a Firecracker VM booting the lambda-sandbox image from ../lambda-sanbdox as a long-lived HTTP server with bash, python3, Node 22, uv and ripgrep.
- The auth token lives 15 minutes; core caches it per VM and port and reuses it until 5 minutes before expiry.
- A proxy 502 or 503 means the VM is still restoring its snapshot, so
/execretries for up to 30 s. A 504 fails the call, because the guest may already be running the command. Request errors come back as HTTP 200 withok: false. maximumDurationInSecondsis the call timeout plus 60 s for an ephemeral VM, andmin(maxLifetimeSeconds, 28800)for a persistent one. A persistent VM also gets anidlePolicywith auto-resume.- The exec answer carries
burst, vCPU-seconds and GiB-seconds above baseline since boot. Core forwards its growth tosandbox.instances.recordBurst, and the meter bills it at baseline rates.
A persistent VM as the control plane sees it. Core never extends a VM; a terminal one is replaced on the next call, so local disk and processes are gone while the workspace stays.
Lifecycle hooks and the workspace mount
The image serves hooks on port 9000 under /aws/lambda-microvms/runtime/v1/:hook. They are internal to the image, not account config.
| Hook | When | Image does |
|---|---|---|
/ready, /validate | image build | return 200 |
/run | each VM start | mount-s3 the workspace from the runHookPayload |
/resume | resume | reconnect and refresh |
/suspend | before snapshot | sync(2) |
/terminate | teardown | unmount and final sync |
The harness's own credentials never enter the VM. Stateless runs skip the mount and work in /tmp. Mountpoint for S3 was chosen over S3 Files because it works on every network mode without a NAT gateway and is the same path workdir and Daytona use.
Image, snapshots and network
- The image is built by AWS from an S3 zip of a Dockerfile (
create-microvm-image), a versioned Firecracker snapshot of memory and disk. Core selects it withMICROVM_IMAGE_IDENTIFIER, optionally pinned byMICROVM_IMAGE_VERSION.image: "obscura"boots the-obscuravariant next to the default. AsnapshotARN must be in the same account and region, and abroods-snapshot-*one boots only for a sandbox in the account and stage of itssandboxSnapshotsrow (see Snapshot stage lock). apps/core/sst.config.tsprovisions theMicrovmArtifactsbucket, themicrovm-buildandmicrovm-executionroles, the/broods/:stage/microvmslog group with its forwarder subscription, and the NAT-lessSandboxNetworkVPC with themicrovm-egressconnector. Not inap-southeast-1(microvmPrereqsEnabled()).- Network:
allow-alluses default internet egress.deny-allandrestrictedattach the shared egress connector, which has no NAT; its S3 gateway endpoint reaches only the managed workspace bucket. Per-account allowlists cannot apply, soallowDomainsandallowCidrsare rejected. WithoutMICROVM_EGRESS_NETWORK_CONNECTOR_ARN,deny-allfails to launch.
There is no API to promote a running VM into an image, so a snapshot is an image build:
The capture script is microvm-snapshot.ts. It skips /proc, /sys, /dev, /run, /tmp, /mnt and the workspace root, refuses a clock more than 5 minutes off, changes over 4 GiB and a zip over 5 GiB. The tar only adds files, so a file deleted in the VM comes back from the source image.
Isolation levels
A workspace's isolation picks the namespace a run mounts (isolatedWorkspaceNamespace in src/shared/workspaces.ts). The full namespace is the reservation key, the S3 prefix and the STS session policy scope, so each level is its own VM, prefix and credentials.
| Level | Namespace | Who shares it |
|---|---|---|
| unset | fs-<hash(account:workspace)> | every agent and conversation on the workspace |
conversation | <base>/<alias>/<hash(conversation)> | one conversation, per the channel's partition |
agent | <base>/agent/<hash(agentId)> | one agent, across all of its conversations |
- The folders stay under the base prefix, so a workspace purge, the storage meter and teardown by namespace prefix still cover them.
assumeScopedMountCredentialsins3-mount.tsnames the sessionfp-sandbox-mount-<agentId>on anagentmount andfp-sandbox-mount-acct-<accountId>on any other, since those outlive one run. On the platformsandbox-s3mountrole it also setsSourceIdentityand thebroods:accountandbroods:agentsession tags, so CloudTrail ties each S3 call to an account and, on anagentmount, an agent. A bring-your-own role gets the session name only.- The in-VM mount directory uses the base namespace by design; reservation, endpoint cache and prefix key on the full one, so one VM holds one workspace folder.
Security notes
- Child processes start from
env_clear(), which does not hide IMDS. Code in the VM can read the execution role, which can only write logs in this stage's MicroVM group. It can create a stream named after another tenant, which is why the forwarder labels only stream names core signed. - Persistent VMs attach
HTTP_INGRESSandSHELL_INGRESSatRunMicrovm. Connectors cannot be added to a live VM, so an instance reserved before them must be terminated and re-reserved; the terminal route says so.
Workdir (sandbox provider)
- Core reaches the workdir control plane at
WORKDIR_URLwithWORKDIR_API_KEY, unlessoptions.workdirUrlandoptions.apiKeyoverride them. Images are referenced by name inconfig.snapshot. One node serves every account, so an image id another account or stage registered is refused before the create (see Snapshot stage lock). - Sizes are create-time resources with vCPU clamped to 0.5, 1, 2 or 4; explicit
options.cpu,memoryMbanddiskGbwin.options.docker: trueenables docker-in-sandbox, on this provider only. - Every command runs as
timeout -k 5 :seconds bash -c ..., since the workdir API has no exec timeout. - A self-hosted node needs the
mountpoint-s3binary in the rootfs, a guest kernel withCONFIG_FUSE_FS=y(the prebuilt Firecracker CI kernels up to v1.13 ship without it), andbashplus GNUtimeout. Without them every file tool fails withmount-s3: not found.
Daytona, E2B and Vercel
Credentials fall back to deployment env: DAYTONA_API_KEY, DAYTONA_ORGANIZATION_ID, DAYTONA_API_URL, DAYTONA_TARGET; E2B_API_KEY; VERCEL_TOKEN, VERCEL_TEAM_ID, VERCEL_PROJECT_ID.
| Topic | Daytona | E2B | Vercel |
|---|---|---|---|
| Idle | idleTimeoutSeconds to autoStopInterval | sandbox timeout with onTimeout: "pause" | session extended to idle past each call, 24 h cap (45 min on Hobby) |
| Max lifetime | maxLifetimeSeconds or 7 days to autoDeleteInterval | not mapped | checked at acquire, retires and recreates |
config.snapshot | snapshot | template | source snapshot for snap_, else image |
| Snapshot verb | sandbox.createSnapshot, broods-<uuid> name | Sandbox.createSnapshot | sandbox.snapshot({ expiration: 0 }) |
| Network | networkBlockAll, domain lists ignored with a warning | allow-all only | allow-all, deny-all, or domain and CIDR allowlists |
- Daytona needs a snapshot with
mount-s3, built bybun run daytona:s3-snapshot(apps/core/scripts/daytona-s3-snapshot.ts). It assumes thesandbox-s3mountrole (SANDBOX_MOUNT_ROLE_ARN) for prefix-scoped credentials. - Each snapshot verb finishes before it answers, so the row is
activeat once.
Snapshot stage lock
A snapshot is a copy of one sandbox's disk, so it boots only where it was taken.
- The snapshot verb writes the
sandboxSnapshotsrow withscope: the{projectId, stageId}of the sandbox config it ran on, or"account"for a config with no stage. Names are unique per account and scope. The row is the lock, so a capture whose row write fails answers an error rather than an unregistered image. Each image also gets asandboxSnapshotImagesrow (account, scope, base image) on its first capture. A re-capture under the same name moves the snapshot row to the new image and leaves the old image's lock in place, and account deletion keeps the locks, so an image id never falls back to booting anywhere as a plain provider image.findByImageanswers from the lock, asinactive, once no snapshot row names the image. snapshotBootRefusal(packages/convex/model/sandboxRules.ts) is the one rule: the row's account and scope must equal the booting sandbox's. A row with noscope(taken before the lock) is refused everywhere. No caller is exempt; an account key and a stage-pinned role session get the same answer.- Writes: the config API (
POSTandPATCH /v1/sandboxes), a dashboard canvas save, a CLI sync and an env var refresh that changessnapshotall runassertSnapshotInScopeand answer400with the refusal. An image id no account registered is a provider image or template and passes. - Boots: every executor that creates a machine from
config.snapshotcallsassertSnapshotBootablefirst (harness/sandbox/utils.ts): lambda in#runInput, workdir in#create, Daytona indaytonaCreateOptions, E2B and Vercel beforeSandbox.create.findByImagelooks the image up across accounts (by_externalImageId_and_accountId, the caller's own row first), so another account's image is refused rather than booted as a provider image. A config with no control plane (no known account or stage) boots no registered snapshot. The dashboard lifecycle verbs (exec,terminaland the rest) build their executor with the sandbox config's control plane, so a machine they create boots the config's own snapshot. Cloudflare, machine and custom rejectsnapshotat validation. - Vercel names a persistent sandbox with a prefix from the reservation key plus a random generation, stored as
externalId, so a sweeper deleting an old machine never hits a new one under the same key.onCreateandonResumerun as one script guarded by.fp-lifecycle-created.
Cloudflare
The Container API only answers inside a Durable Object, so cloudflare-executor.ts calls the bridge Worker in apps/cloudflare-sandbox (src/index.ts). Each sandbox id is one Sandbox Durable Object owning one Container booted from the Worker's Dockerfile. The bridge stays dumb: ids, env, working directory and output limit all come from core.
| Route | Does |
|---|---|
POST /v1/sandboxes/:id/exec | Starts the Container if needed, runs one argv, answers a SandboxExecResponse |
GET /v1/sandboxes/:id | { running } |
DELETE /v1/sandboxes/:id | Destroys the Container |
GET /v1/sandboxes/:id/terminal | PTY WebSocket, raw bytes both ways, for a terminal ticket |
- A Durable Object exists once named, so a persistent reservation is only the claim. A run that loses the race uses the winner's id. An existing reservation is reused by key alone, which is how the dashboard console reaches it.
- A stopped Container lost its disk, so
getInstanceInforeports it gone, not suspended. There is no suspend, snapshot, workspace mount or background job, and harness adapters refuse the provider. network.mode: "allow-all"starts withenableInternet: true, anything else without;restrictedis rejected.sizemaps to the nearest instance type,standard-1tostandard-4(CLOUDFLARE_INSTANCE_TYPESinapps/core/src/shared/sandbox-sizes.ts).- Limits are in
apps/cloudflare-sandbox/src/limits.ts, which core imports to clamp: 1 MiB output per stream, 15 minute timeout, 6 hour idle (thesetInactivityTimeoutceiling). An ephemeral Container idles only a minute past its command, so one whoseDELETEfailed stops soon. - Exit 124 or 137 counts as a timeout only once the deadline passed; earlier it is the command's own exit or an OOM kill.
sst deployships the bridge per stage with wrangler, ascloudflareWorkers.sandbox, when the deploy hasCLOUDFLARE_API_TOKEN. Its Container image is built only when the registry lacksbroods-sandbox:<Dockerfile hash>(the first deploy to an account, or aDockerfilechange), shared by every stage, so only that deploy needs Docker. The infra repo generatesSANDBOX_API_KEY, puts it on the Worker and in core'sCLOUDFLARE_SANDBOX_API_KEY, and pointsCLOUDFLARE_SANDBOX_URLat it. Each stage's bridge has its own Containers.
Worker
Bash in a V8 isolate, no container and no Durable Object. worker-executor.ts calls the Worker in apps/cloudflare-worker (src/index.ts), which loads one Dynamic Worker per sandbox id running just-bash, a bash written in TypeScript with its own in-memory disk and 81 commands (cat, grep, sed, awk, jq, find, sort, gzip, and curl with internet). No processes, no python, no git, no eval: a command is either a built-in or command not found. The Worker stays dumb: ids, env, working directory and output limit all come from core.
| Route | Does |
|---|---|
POST /v1/sandboxes/:id/exec | Runs one script in the sandbox's isolate, answers a SandboxExecResponse |
GET /v1/sandboxes/:id | { exists }: whether a stored disk is under the id |
DELETE /v1/sandboxes/:id | Drops the stored disk, leaving an empty released marker that later calls get 410 for |
- Nothing runs between commands. A persistent sandbox is its stored disk: one gzip of a JSON header and the files' raw bytes, one R2 object per id, restored into a fresh shell on every call. The runtime packs the disk again after the command and answers "unchanged" when nothing differs, so a read-only command stores nothing. An ephemeral call runs on an empty disk and stores nothing. There is no idle, no suspend, no snapshot, no terminal, no workspace mount and no background job.
- Billing follows: each command is metered as it ends (its time on the Worker plus one request), an ephemeral call keeps no instance row, and a reserved sandbox's row reads
suspendedwhile its disk is stored and bills nothing. - The isolate is keyed by sandbox id plus what is fixed at load: its network (
globalOutbound: nullwithout internet, socurlis not even registered) and its CPU limit,limits.cpuMsfrom the call's timeout. That limit is the real guard for a busy command, because inside an isolate the clock moves only while the shell waits; just-bash's own deadline (exit 124) catches asleepor a slowcurl. - The isolate runs with
nodejs_compatfornode:zlib(gzip and file sniffing, and the stored disk) and theBufferglobal; nothing else from Node is in the build.bun run buildin the workspace writes.generated/runtime.txtfromsrc/runtime.ts, and the host ships it as a text module. - Limits are in
apps/cloudflare-worker/src/limits.ts, which core imports to clamp: 1 MiB output per stream, 60 s timeout, 16 MiB stored disk compressed. A disk past the cap stays as it was and the call fails with what to clean up. sst deployships the Worker per stage with wrangler, ascloudflareWorkers.worker, when the deploy hasCLOUDFLARE_API_TOKEN. The infra repo generatesWORKER_API_KEY, puts it on the Worker and in core'sCLOUDFLARE_WORKER_API_KEY, and pointsCLOUDFLARE_WORKER_URLat it.bun testthere runs the Worker in Miniflare with a real Worker Loader and R2.- A
diskworkspace (storage.provider: "disk",sandboxHoldsWorkspaceinpackages/convex/model/sandboxProviders.ts) is this disk: the workspace sandbox carries nostorage, so the executor reserves on the workspace namespace and each folder, partition and user conversation is its own R2 object.SHELL_ONLY_SANDBOX_PROVIDERSmarks the shell with nonodeand nosync:editreads the file withbase64, replaces in core and writes it back;globlists withfindand matches withBun.Glob;writeskipssync.harness/conversation-files.tsreads a folder as one gzipped listing (the answer is capped at 1 MiB) and replaces it in one command.
Machine
machine-executor.ts runs bash, the computer tool and local MCP servers on a user's computer over the WebSocket the broods machine daemon holds. Frames are defined once in src/shared/machine-socket.ts, which the CLI bundles; the gateway relays them unchanged.
- One daemon holds a record. A second is closed with
4423naming the holder's host;--forcereplaces the holder, which gets4409. Bad credentials close with4401, an unknown sandbox name with4404, a malformed frame with4400. - A stage ticket's socket lives as long as its ticket.
readytells the daemon how long that is (expiresInMs, relative so a skewed clock cannot move it), and the daemon sends{ type: "credential", token }with a freshly minted ticket at half the time left, retrying at half of what remains until core answers. Core resolves that ticket and, only if it is for the socket's own account and stage and may still write sandboxes, moves the deadline and the bearer each heartbeat checks, then sendsreadyagain with the newexpiresInMs. Any other ticket closes the socket with4401; one core cannot check for an outage keeps the old deadline. Without a refresh the socket closes with4419the moment the ticket expires, and core sends no frame to it after that, even if the timer runs late.4419is not fatal: the daemon (and a CLI older than the refresh) reconnects with a freshly minted ticket and reclaims its record, and a command still running then fails like any disconnect. Every heartbeat also resolves the daemon's bearer again; one that no longer resolves to the account (key rotated, role session ended or revoked, account suspended) closes with4401. A heartbeat that cannot reach Convex keeps the daemon. - The socket accepts a stage ticket minted from
broods login, the account key, or a role session withsandboxes:writeon the record. Never the runtime key or the dashboard's member ticket. - Frames cap at 4 MiB. A computer action has 30 s, an MCP call 60 s,
bashits timeout plus 5 s. - Validation rejects
persistent,size,snapshotand any network mode butallow-all, and a machine cannot back a workspace.
Harness adapters
An agent with harness set runs its adapter (Claude Code, Codex, Deep Agents, OpenCode or Pi) inside its first sandbox through @broods/ai-sdk-sandbox. Only a persistent lambda or sandbox qualifies; src/harness/ai-sdk-harness/sandbox.ts picks microvm-harness-driver.ts or workdir-harness-driver.ts. The bridge listens on port 4321.
- The key is stored on
runtimeHarnessSessions, so a conversation resumes on the machine it started on. A session ending on a shared machine neither suspends nor deletes it. - Claude Code, Codex, Deep Agents and OpenCode keep the conversation on the machine (each CLI's own session store), and the checkpoint holds only its id. When the machine is released, or an adapter version bump moves the conversation to a new one, the next turn starts a fresh session.
- Pi keeps its conversation in Convex: Pi appends each session entry through
sessionStorage(adapters/pi.ts), the BroodsSessionholds the entries untilsaveHarnessSession, and the checkpoint holds only the leaf entry id and the last entry sequence. A turn usually costs one save and one paged read whatever the history length; a turn over one transaction's budget (4 MiB or 100 entries) stages the rest inappendHarnessSessionEntriesfirst. When Pi's machine is released,openAiSdkHarnessSessionresumes the same session on a fresh machine and logs it; the other adapters start a fresh session. An entry over 512 KiB is stored with every image swapped for[attachment not retained]and every long string truncated, in one pass. - Neither provider streams processes, so
harness-shell-process.tsruns the adapter detached and file-spooled; short shell calls poll its output in 180 KiB chunks every 25 ms. - Core holds a whole file in memory for each harness file read, so a read of a file over 25 MiB (a symlink counts as the file it points to) fails with
SandboxFileTooLargeErrorbefore its bytes reach core; the harness reads a part of it with a shell command instead. - Core also holds a harness command's stdout and stderr whole, because adapters parse them (Pi's
findandls,listDirectory). Each is capped at 16 MiB for core's memory, not the model's context: past it core kills the command, stops polling it, and fails the call withSandboxOutputTooLargeErrorinstead of returning a part. The cap bounds one stream, not the pod: several large commands at once can still add up. - On
lambda, WebSocket ingress needs the AWS token and port as subprotocols, somicrovm-websocket-proxy.tshands the adapter a loopback URL and adds them upstream. The token stays in core and the upstream header, never a URL or log.
- Live state (size, other runs holding the machine, and on a harness turn CPU, memory and disk from one probe) goes at the end of
<environment>.live-status.tskeeps occupancy in memory, which is correct because core runs one replica.
Opt-in live tests, each cleaning up its own sandboxes:
cd apps/core
MICROVM_HARNESS_TEST=1 bun test tests/sandbox-microvm-harness.integration.test.ts
WORKDIR_TEST_URL=... WORKDIR_TEST_KEY=... bun test tests/workdir-harness.integration.test.ts
Snapshot status model
sandboxSnapshots.status as core writes it:
The schema also allows pending, pulling, inactive and error, and the dashboard has a tone for each, but no core path writes them today.
Security review notes (2026-09)
Findings from the host-boundary review, each closed in the harness:
options.dockermust be a boolean and is accepted on thesandboxprovider only.lambdarefusesrestrictedallowlists at validation instead of launching on the shared connector with a warning.- Every executor merges env through one helper that drops reserved keys.
- A BYO bucket needs
storage.prefix, and mount credentials are scoped tobucket/prefix/*. - The background-job callback token rides the exec environment, not the script text.
- One MicroVM holds one workspace.
- Workspace
bashrejects literal..traversal as a guardrail only. A shell expands$'\x2e\x2e'or$(printf ..)after the check. Containment comes from the VM and the prefix-scoped mount credentials. Writes to absolute paths outside the mount are refused, except/tmpand/var/tmp, unless the workspace runs on the agent's own persistent sandbox.