Skip to main content

Sandboxes

How core runs sandbox tools on each provider. Users configure sandboxes through the sandboxes guide. Paths are relative to apps/core/src/harness/sandbox/ unless noted.

Where sandboxes run​

Core owns every call. Providers only run commands; reservations, ids, env and limits stay in core and Convex.

  • lambda is the provider string for the AWS Lambda MicroVM backend. Its guest image is the Rust server in the sibling repo ../lambda-sanbdox.
  • Only MicroVM guest logs take the CloudWatch path. The forwarder labels a line by tenant only when core signed its stream name, see observability.

Executor model​

Every sandbox tool compiles to one run on the provider's executor. index.ts maps provider name to factory in EXECUTORS. The names are SANDBOX_PROVIDERS in packages/convex/model/sandboxProviders.ts, the one list the Convex validator, core and the SDK derive from.

  • The budget wrapper also guards runBackground, resume and postReserved.
  • SandboxCapacityError comes only from a refused create: MicroVM InsufficientCapacityException, ServiceQuotaExceededException, ThrottlingException, TooManyRequestsException, and the workdir and Daytona admission refusals. Validation refuses a fallback on a persistent config, a fallback equal to provider, and a machine or custom fallback.
  • Limits come from packages/convex/model/sandboxRules.ts: timeout 30 s default, capped by WORKSPACE_SANDBOX_MAX_TIMEOUT_SECONDS and WORKSPACE_SANDBOX_LAMBDA_MAX_TIMEOUT_SECONDS; output capped by WORKSPACE_SANDBOX_MAX_OUTPUT_LIMIT_BYTES. A blocking call also stays inside REQUEST_TIMEOUT_BUDGET_MS in src/server.ts. Background jobs are bound by neither.
  • Env: mergeSandboxEnv() in utils.ts starts from the account's envVars. Only a harness shell command on lambda adds a per-call layer (requestEnv), and RESERVED_SANDBOX_ENV_KEYS (PATH, HOME, LD_PRELOAD, NODE_OPTIONS, the __CB_* callback slots and the rest) are dropped from it. The BROODS_* identity names are dropped from both, then the run's BROODS_* identity is set on top.
  • The exec wire contract a MicroVM image, the Cloudflare bridge and a custom server share is SandboxExecRequest / SandboxExecResponse in src/shared/domain/sandbox-config.ts. parseExecResponse and execRunResult in utils.ts read it.
  • CPU: sandbox reports CPU from the workdir cgroup and lambda from the image's getrusage. src/harness/usage-metering.ts sums it per task into the sandboxUsage field of the run's taskUsage row and puts tool.compute.type and tool.compute.cpu_usec on the tool.call span. Other providers report none.

Contribute a provider​

  1. Write src/harness/sandbox/<name>-executor.ts implementing SandboxExecutor from types.ts. Only run is required; reservation, jobs and lifecycle methods are feature-detected.
  2. Add the name to SANDBOX_PROVIDERS. Add it to STATELESS_SANDBOX_PROVIDERS too if Broods never reserves it.
  3. Import the file in index.ts and add it to EXECUTORS. The build fails until you do, and the import is what pulls the file into the compiled binary.
  4. Put option validation in sandboxRules.ts, and say in runsOnOwnCredentials (src/shared/workspaces.ts) whether the platform meters it.

Capability matrix​

ProviderExecutorWorkspace mountPersistentSuspend, resumeSnapshotBackground jobs
lambdamicrovm-executor.tsmount-s3 in the /run hookidle suspend, 8 h maxyesyesyes, logs and stop
sandboxworkdir-executor.tsmount-s3 per runnative pause, resume, standbyyesyesyes, logs and stop
daytonadaytona-executor.tsmount-s3 with mountAwsS3Bucketsnative autoStopIntervalnoyesyes, logs and stop
e2be2b-executor.tsrejectednative pause on timeoutnoyeslaunch and callback, no logs, stop
vercelvercel-executor.tsmount-s3 per runnamed sandboxnoyesyes, logs and stop
cloudflarecloudflare-executor.tsrejectedsame Container while warm, no disknonono
workerworker-executor.tsrejectedfiles stored in R2 between callsnonono
machinemachine-executor.tsrejectednononono
customhttp-executor.tsrejectednononono

The detached workspace job needs a persistent workspace sandbox. E2B instead runs background: true natively on a persistent sandbox, with a completion callback but no logs or stop.

Stateless call​

A run with persistent unset creates a machine, runs once and tears it down. A platform-paid one gets an ephemeral sandboxInstances row for the length of the call, which is what the meter bills.

  • Every write for an ephemeral row goes through its row lane (see Instance row), so a removal never lands before the upsert.
  • Cloudflare writes the row before the exec that starts the Container, so its start is billed too. The MicroVM writes it after RunMicrovm returns.
  • A lost row is billed no further than the provider's max call timeout plus 2 minutes (ephemeralSandboxMaxMs in packages/convex/model/usageMeter.ts). sweepStaleEphemeral, started by the hourly accrueRecent cron, deletes it after that.
  • An worker row bills no time at all. Nothing runs between its commands, so the executor meters each command as it ends, the way a hosted MCP invoke is: its time on the Worker, waits included, as workerMs and one workerRequests, priced at Cloudflare's CPU-ms and request rates (packages/convex/model/pricing.ts, a slightly high estimate since a Worker cannot read its own CPU time) and shown inside Sandbox time. A call the Worker refused before its isolate ran (a 400, 401 or 404, a 410 for a released sandbox, or Cloudflare's own error) is not metered; a 409 means the command ran and lost the store race, so it is.
  • A sandbox on the account's own credentials gets no row. No lifecycle verb accepts an ephemeral row.

Reservations​

A persistent: true config reserves one machine per key. Keys are hashed and account-scoped in src/shared/workspaces.ts, so no key an author writes can name another account's machine.

Harness adapters pick their own key, see harness adapters.

First call and resume​

The MicroVM executor (#acquire in microvm-executor.ts) as the example. Workdir, Daytona, E2B, Vercel and Cloudflare follow the same claim and race rules.

  • A slow resume or a control-plane error propagates. Only a VM unknown to the provider or terminal is replaced, since replacing a suspended one leaks it against the account's memory quota.
  • A persistent MicroVM counts against that quota while running or suspended. Too many make every launch fail with ServiceQuotaExceededException, and a reserved sandbox has no fallback.
  • lifecycle.maxLifetimeSeconds is checked on acquire, never on a timer, so it cannot interrupt a running command.

Reservation row​

sandboxReservations is the authoritative key to provider id map. Every write is conditional on the expected id, so a stale caller never removes a machine another run replaced.

  • TTL: 7 days after last use (SANDBOX_RESERVATION_TTL_SECONDS in packages/convex/runtime.ts), one day for a harness reservation on a conversation key (ISOLATED_SANDBOX_RELEASE_SECONDS in src/harness/ai-sdk-harness/session.ts).
  • src/shared/sandbox-sweeper.ts runs hourly (SANDBOX_SWEEP_INTERVAL_SECONDS) under a 5 minute lease, first after a random delay under 30 s. It pages expired reservations, plus mirror rows no reservation names, through releaseExpiredSandboxes() in src/shared/sandbox-cleanup.ts. The orphan scan reads idle mirrors oldest first, page by page, until it has a page of orphans or reaches rows still in use, so mirrors whose deferred reservation keeps them reserved cannot hide the orphans behind them.
  • A release goes through the provider that reserved the machine, not the config's current one, so switching provider or turning persistent off strands nothing. It uses the reserving config's credentials when the instance row says ownCredentials, else the platform's.
  • A sandbox config that still holds an instance is kept when its CLI resource or canvas card goes, so an instance never names a missing config.

Instance row​

sandboxInstances mirrors each reserved machine for the dashboard. Core writes the steady states. The dashboard's suspend action parks the row in suspending (packages/convex/sandbox/public.ts).

  • Refresh removes the row when the provider reports nothing or terminating, so terminating is never stored by core. The removal names the machine it read, so a replacement's row under the same key stays.
  • Core sends every row write through src/harness/sandbox/mirror-writes.ts: executor upserts on start and on every command, E2B's size patch, MicroVM burst reports, the exec, terminal, refresh, suspend and resume status writes, and every removal. Executors never wait for their writes.
  • Each row has a lane, keyed on the account and the reservation key (an ephemeral row's key is its sandbox id). A row's writes land in the order they were queued. A write of the same kind to the same machine as the lane's last pending write takes its place (an upsert keeps the run fields the newer one lacks; a status write never does, so a suspend and the resume after it both land), so a backlog holds only the latest; nothing else is dropped or reordered. Memory follows rows, not writes.
  • An unchanged refresh is sent at most once a minute per row (MIRROR_REFRESH_EVERY_MS), or once per fifteenth of the sandbox's idle timeout when that is shorter. A Convex write costs about the same CPU whatever it changes (about 25 ms on the local stack), so at one command every 5 s core asks for 12 times fewer writes. 1,000 such sandboxes on one account would ask for 200 writes a second: sent one at a time, about 60 a second land and Convex uses about 1.5 cores; with unchanged refreshes skipped, 17 a second land and it uses about 0.4. A refresh that differs in any field (a new run, machine or size), or that follows a status write or a removal, is sent at once. A refresh counts as sent only once Convex wrote the row: one that failed, timed out or was refused is sent again on the next command. The row's last-used time can trail the last command by up to that window, so an idle tail bills up to that much less.
  • An account sends one write at a time. Parallel writes on one account conflict on its shared usage rows in Convex (OptimisticConcurrencyControlFailure) and fail: on the local stack 158 of 200 first writes sent at once were lost, 3 of 200 with two in parallel, none one at a time (about 15 ms each). The account's lanes take turns, and a lane holding a status, a removal or a MicroVM burst report goes first, so a terminate waits for about two writes and a burst reaches its row before a removal queued after it. Under overload a row's refreshes get staler; the queue never grows past a few writes per row. Accounts run in parallel.
  • A write that has not answered after 10 s (MIRROR_WRITE_TIMEOUT_MS) frees the account's sender and logs, with the lane count, pending writes and oldest pending age; so does one that waited over 2 s. Shutdown's drainInFlight waits for every account's sender.
  • The sender only orders one pod's writes. The reservation check in upsert is what keeps a late write from another pod, such as the old pod of a rolling update, or one that timed out but landed, from bringing a released row back. A late write for an ephemeral row, which has no reservation, can bring it back until sweepStaleEphemeral.
  • A reserved row is written only while its reservation still names the machine (upsert in packages/convex/sandbox/instances.ts). Every release drops the reservation before the row goes, so a write that lands after it, such as a Cloudflare or Worker command that answered core after a terminate, is dropped instead of bringing back a row that bills a machine that is gone, or pointing a key reserved again at its old machine. Ephemeral rows need no reservation.
  • specs is what the meter bills. specsVerified is set only when the size is real: a provider report, or the size Broods sets on workdir, MicroVM and Cloudflare. Without it the dashboard shows ?.
  • Executors queue a refresh on every command, and core sends the unchanged ones once a minute. A refresh of a machine still running at the size, credentials and project it was last billed at bills at most once a minute (REFRESH_BILL_EVERY_MS in packages/convex/sandbox/instances.ts), so an owner's concurrent tasks write the shared usage rows about once a minute each, not on every command. The next bill, status change or removal covers the stretch, so nothing is lost; like the idle tail, it lands on the day and month of the bill that covers it, and live usage can trail by up to a minute.

Lifecycle actions and terminal​

The dashboard and the account API call POST /v1/sandboxes/:id/:action in src/accounts/handler.ts, with action suspend, resume, terminate, snapshot, refresh, exec or terminal. Each writes a sandboxAuditEvents row naming the actor. Only the config plane, on the service token, may name the dashboard user it acts for in the body's actor; any other caller is recorded as its own credential, service, with the role id for a scoped key or role session. exec takes at most 20,000 characters; the dashboard calls it with a 30 s timeout and 64 KiB output.

terminal opens a real TTY on sandbox, lambda and cloudflare. Other providers answer it as unsupported and the dashboard keeps the bounded exec runner.

ProviderUpstreamCredential in the ticket
sandboxworkdir PTY WebSocketone-time terminal token the workdir host minted, as bearer
lambdaMicroVM native shell (SHELL_INGRESS)CreateMicrovmShellAuthToken JWE, 30 min, X-aws-proxy-auth
cloudflarebridge GET /v1/sandboxes/:id/terminalterminal token for that sandbox as bearer, ticket TTL
  • The ticket is AES-256-GCM sealed (src/shared/terminal-ticket.ts), so the browser holds only an opaque token. If NATS is down the upgrade answers 502 rather than skip the spend check.
  • A Cloudflare Container that is not running answers 409; a command has to start it first.
  • The Cloudflare terminal token (apps/cloudflare-sandbox/src/terminal-token.ts) is an HMAC of the sandbox id and expiry, signed with the bridge API key. The bridge accepts it only on /terminal for that sandbox id and until it expires, so the gateway never holds the key that reaches every sandbox.
  • For workdir, core asks the host for a token with POST /v1/sandboxes/:id/pty-token, on the same URL and API key as its other calls (an account's own workdirUrl and apiKey when set). The host keeps the token only in memory and accepts it once, on that sandbox's PTY, for 120 seconds; the ticket keeps its own two-minute life, whatever the host answers. A terminal token lets its holder run commands in that one sandbox for up to two minutes, once. A host without the route answers 404, and the terminal answers 409 saying the workdir host needs updating; core never falls back to the API key.

Background jobs​

bash { background: true } on a persistent workspace sandbox launches a detached setsid session and returns a statusId, the model-facing name of the runtimeAsyncToolResults row's resultId.

The row exists before the launch, so a fast job's callback never arrives first. A settled row answers 409; an unknown id or wrong token answers 404, so the route is not a token oracle. The continuation queues behind any live turn, see queue and steer.

The job as statusScript in jobs.ts reads its marker files:

The runtimeAsyncToolResults row the model sees through async_status:

  • lambda, workdir, Daytona and Vercel keep markers under <workspace root>/.fp-jobs/<reservation key>, at most 10 running (MAX_CONCURRENT_BACKGROUND_JOBS). E2B uses its native background: true command, so it has callback delivery but no logs or stop.
  • A MicroVM resume keeps the boot id, so a resumed job still reads as running. A recreate or scale-to-zero changes it, so the job reads as failed.
  • The callback runs the image's python3 and needs egress to PUBLIC_BASE_URL. Without it the job still runs and polling still works. Idle scale-down never pauses a sandbox with a running job.

Lambda MicroVM​

Each reserved key runs in one AWS Lambda MicroVM, a Firecracker VM booting the lambda-sandbox image from ../lambda-sanbdox as a long-lived HTTP server with bash, python3, Node 22, uv and ripgrep.

  • The auth token lives 15 minutes; core caches it per VM and port and reuses it until 5 minutes before expiry.
  • A proxy 502 or 503 means the VM is still restoring its snapshot, so /exec retries for up to 30 s. A 504 fails the call, because the guest may already be running the command. Request errors come back as HTTP 200 with ok: false.
  • maximumDurationInSeconds is the call timeout plus 60 s for an ephemeral VM, and min(maxLifetimeSeconds, 28800) for a persistent one. A persistent VM also gets an idlePolicy with auto-resume.
  • The exec answer carries burst, vCPU-seconds and GiB-seconds above baseline since boot. Core forwards its growth to sandbox.instances.recordBurst, and the meter bills it at baseline rates.

A persistent VM as the control plane sees it. Core never extends a VM; a terminal one is replaced on the next call, so local disk and processes are gone while the workspace stays.

Lifecycle hooks and the workspace mount​

The image serves hooks on port 9000 under /aws/lambda-microvms/runtime/v1/:hook. They are internal to the image, not account config.

HookWhenImage does
/ready, /validateimage buildreturn 200
/runeach VM startmount-s3 the workspace from the runHookPayload
/resumeresumereconnect and refresh
/suspendbefore snapshotsync(2)
/terminateteardownunmount and final sync

The harness's own credentials never enter the VM. Stateless runs skip the mount and work in /tmp. Mountpoint for S3 was chosen over S3 Files because it works on every network mode without a NAT gateway and is the same path workdir and Daytona use.

Image, snapshots and network​

  • The image is built by AWS from an S3 zip of a Dockerfile (create-microvm-image), a versioned Firecracker snapshot of memory and disk. Core selects it with MICROVM_IMAGE_IDENTIFIER, optionally pinned by MICROVM_IMAGE_VERSION. image: "obscura" boots the -obscura variant next to the default. A snapshot ARN must be in the same account and region, and a broods-snapshot-* one boots only for a sandbox in the account and stage of its sandboxSnapshots row (see Snapshot stage lock).
  • apps/core/sst.config.ts provisions the MicrovmArtifacts bucket, the microvm-build and microvm-execution roles, the /broods/:stage/microvms log group with its forwarder subscription, and the NAT-less SandboxNetwork VPC with the microvm-egress connector. Not in ap-southeast-1 (microvmPrereqsEnabled()).
  • Network: allow-all uses default internet egress. deny-all and restricted attach the shared egress connector, which has no NAT; its S3 gateway endpoint reaches only the managed workspace bucket. Per-account allowlists cannot apply, so allowDomains and allowCidrs are rejected. Without MICROVM_EGRESS_NETWORK_CONNECTOR_ARN, deny-all fails to launch.

There is no API to promote a running VM into an image, so a snapshot is an image build:

The capture script is microvm-snapshot.ts. It skips /proc, /sys, /dev, /run, /tmp, /mnt and the workspace root, refuses a clock more than 5 minutes off, changes over 4 GiB and a zip over 5 GiB. The tar only adds files, so a file deleted in the VM comes back from the source image.

Isolation levels​

A workspace's isolation picks the namespace a run mounts (isolatedWorkspaceNamespace in src/shared/workspaces.ts). The full namespace is the reservation key, the S3 prefix and the STS session policy scope, so each level is its own VM, prefix and credentials.

LevelNamespaceWho shares it
unsetfs-<hash(account:workspace)>every agent and conversation on the workspace
conversation<base>/<alias>/<hash(conversation)>one conversation, per the channel's partition
agent<base>/agent/<hash(agentId)>one agent, across all of its conversations
  • The folders stay under the base prefix, so a workspace purge, the storage meter and teardown by namespace prefix still cover them.
  • assumeScopedMountCredentials in s3-mount.ts names the session fp-sandbox-mount-<agentId> on an agent mount and fp-sandbox-mount-acct-<accountId> on any other, since those outlive one run. On the platform sandbox-s3mount role it also sets SourceIdentity and the broods:account and broods:agent session tags, so CloudTrail ties each S3 call to an account and, on an agent mount, an agent. A bring-your-own role gets the session name only.
  • The in-VM mount directory uses the base namespace by design; reservation, endpoint cache and prefix key on the full one, so one VM holds one workspace folder.

Security notes​

  • Child processes start from env_clear(), which does not hide IMDS. Code in the VM can read the execution role, which can only write logs in this stage's MicroVM group. It can create a stream named after another tenant, which is why the forwarder labels only stream names core signed.
  • Persistent VMs attach HTTP_INGRESS and SHELL_INGRESS at RunMicrovm. Connectors cannot be added to a live VM, so an instance reserved before them must be terminated and re-reserved; the terminal route says so.

Workdir (sandbox provider)​

  • Core reaches the workdir control plane at WORKDIR_URL with WORKDIR_API_KEY, unless options.workdirUrl and options.apiKey override them. Images are referenced by name in config.snapshot. One node serves every account, so an image id another account or stage registered is refused before the create (see Snapshot stage lock).
  • Sizes are create-time resources with vCPU clamped to 0.5, 1, 2 or 4; explicit options.cpu, memoryMb and diskGb win. options.docker: true enables docker-in-sandbox, on this provider only.
  • Every command runs as timeout -k 5 :seconds bash -c ..., since the workdir API has no exec timeout.
  • A self-hosted node needs the mountpoint-s3 binary in the rootfs, a guest kernel with CONFIG_FUSE_FS=y (the prebuilt Firecracker CI kernels up to v1.13 ship without it), and bash plus GNU timeout. Without them every file tool fails with mount-s3: not found.

Daytona, E2B and Vercel​

Credentials fall back to deployment env: DAYTONA_API_KEY, DAYTONA_ORGANIZATION_ID, DAYTONA_API_URL, DAYTONA_TARGET; E2B_API_KEY; VERCEL_TOKEN, VERCEL_TEAM_ID, VERCEL_PROJECT_ID.

TopicDaytonaE2BVercel
IdleidleTimeoutSeconds to autoStopIntervalsandbox timeout with onTimeout: "pause"session extended to idle past each call, 24 h cap (45 min on Hobby)
Max lifetimemaxLifetimeSeconds or 7 days to autoDeleteIntervalnot mappedchecked at acquire, retires and recreates
config.snapshotsnapshottemplatesource snapshot for snap_, else image
Snapshot verbsandbox.createSnapshot, broods-<uuid> nameSandbox.createSnapshotsandbox.snapshot({ expiration: 0 })
NetworknetworkBlockAll, domain lists ignored with a warningallow-all onlyallow-all, deny-all, or domain and CIDR allowlists
  • Daytona needs a snapshot with mount-s3, built by bun run daytona:s3-snapshot (apps/core/scripts/daytona-s3-snapshot.ts). It assumes the sandbox-s3mount role (SANDBOX_MOUNT_ROLE_ARN) for prefix-scoped credentials.
  • Each snapshot verb finishes before it answers, so the row is active at once.

Snapshot stage lock​

A snapshot is a copy of one sandbox's disk, so it boots only where it was taken.

  • The snapshot verb writes the sandboxSnapshots row with scope: the {projectId, stageId} of the sandbox config it ran on, or "account" for a config with no stage. Names are unique per account and scope. The row is the lock, so a capture whose row write fails answers an error rather than an unregistered image. Each image also gets a sandboxSnapshotImages row (account, scope, base image) on its first capture. A re-capture under the same name moves the snapshot row to the new image and leaves the old image's lock in place, and account deletion keeps the locks, so an image id never falls back to booting anywhere as a plain provider image. findByImage answers from the lock, as inactive, once no snapshot row names the image.
  • snapshotBootRefusal (packages/convex/model/sandboxRules.ts) is the one rule: the row's account and scope must equal the booting sandbox's. A row with no scope (taken before the lock) is refused everywhere. No caller is exempt; an account key and a stage-pinned role session get the same answer.
  • Writes: the config API (POST and PATCH /v1/sandboxes), a dashboard canvas save, a CLI sync and an env var refresh that changes snapshot all run assertSnapshotInScope and answer 400 with the refusal. An image id no account registered is a provider image or template and passes.
  • Boots: every executor that creates a machine from config.snapshot calls assertSnapshotBootable first (harness/sandbox/utils.ts): lambda in #runInput, workdir in #create, Daytona in daytonaCreateOptions, E2B and Vercel before Sandbox.create. findByImage looks the image up across accounts (by_externalImageId_and_accountId, the caller's own row first), so another account's image is refused rather than booted as a provider image. A config with no control plane (no known account or stage) boots no registered snapshot. The dashboard lifecycle verbs (exec, terminal and the rest) build their executor with the sandbox config's control plane, so a machine they create boots the config's own snapshot. Cloudflare, machine and custom reject snapshot at validation.
  • Vercel names a persistent sandbox with a prefix from the reservation key plus a random generation, stored as externalId, so a sweeper deleting an old machine never hits a new one under the same key. onCreate and onResume run as one script guarded by .fp-lifecycle-created.

Cloudflare​

The Container API only answers inside a Durable Object, so cloudflare-executor.ts calls the bridge Worker in apps/cloudflare-sandbox (src/index.ts). Each sandbox id is one Sandbox Durable Object owning one Container booted from the Worker's Dockerfile. The bridge stays dumb: ids, env, working directory and output limit all come from core.

RouteDoes
POST /v1/sandboxes/:id/execStarts the Container if needed, runs one argv, answers a SandboxExecResponse
GET /v1/sandboxes/:id{ running }
DELETE /v1/sandboxes/:idDestroys the Container
GET /v1/sandboxes/:id/terminalPTY WebSocket, raw bytes both ways, for a terminal ticket
  • A Durable Object exists once named, so a persistent reservation is only the claim. A run that loses the race uses the winner's id. An existing reservation is reused by key alone, which is how the dashboard console reaches it.
  • A stopped Container lost its disk, so getInstanceInfo reports it gone, not suspended. There is no suspend, snapshot, workspace mount or background job, and harness adapters refuse the provider.
  • network.mode: "allow-all" starts with enableInternet: true, anything else without; restricted is rejected. size maps to the nearest instance type, standard-1 to standard-4 (CLOUDFLARE_INSTANCE_TYPES in apps/core/src/shared/sandbox-sizes.ts).
  • Limits are in apps/cloudflare-sandbox/src/limits.ts, which core imports to clamp: 1 MiB output per stream, 15 minute timeout, 6 hour idle (the setInactivityTimeout ceiling). An ephemeral Container idles only a minute past its command, so one whose DELETE failed stops soon.
  • Exit 124 or 137 counts as a timeout only once the deadline passed; earlier it is the command's own exit or an OOM kill.
  • sst deploy ships the bridge per stage with wrangler, as cloudflareWorkers.sandbox, when the deploy has CLOUDFLARE_API_TOKEN. Its Container image is built only when the registry lacks broods-sandbox:<Dockerfile hash> (the first deploy to an account, or a Dockerfile change), shared by every stage, so only that deploy needs Docker. The infra repo generates SANDBOX_API_KEY, puts it on the Worker and in core's CLOUDFLARE_SANDBOX_API_KEY, and points CLOUDFLARE_SANDBOX_URL at it. Each stage's bridge has its own Containers.

Worker​

Bash in a V8 isolate, no container and no Durable Object. worker-executor.ts calls the Worker in apps/cloudflare-worker (src/index.ts), which loads one Dynamic Worker per sandbox id running just-bash, a bash written in TypeScript with its own in-memory disk and 81 commands (cat, grep, sed, awk, jq, find, sort, gzip, and curl with internet). No processes, no python, no git, no eval: a command is either a built-in or command not found. The Worker stays dumb: ids, env, working directory and output limit all come from core.

RouteDoes
POST /v1/sandboxes/:id/execRuns one script in the sandbox's isolate, answers a SandboxExecResponse
GET /v1/sandboxes/:id{ exists }: whether a stored disk is under the id
DELETE /v1/sandboxes/:idDrops the stored disk, leaving an empty released marker that later calls get 410 for
  • Nothing runs between commands. A persistent sandbox is its stored disk: one gzip of a JSON header and the files' raw bytes, one R2 object per id, restored into a fresh shell on every call. The runtime packs the disk again after the command and answers "unchanged" when nothing differs, so a read-only command stores nothing. An ephemeral call runs on an empty disk and stores nothing. There is no idle, no suspend, no snapshot, no terminal, no workspace mount and no background job.
  • Billing follows: each command is metered as it ends (its time on the Worker plus one request), an ephemeral call keeps no instance row, and a reserved sandbox's row reads suspended while its disk is stored and bills nothing.
  • The isolate is keyed by sandbox id plus what is fixed at load: its network (globalOutbound: null without internet, so curl is not even registered) and its CPU limit, limits.cpuMs from the call's timeout. That limit is the real guard for a busy command, because inside an isolate the clock moves only while the shell waits; just-bash's own deadline (exit 124) catches a sleep or a slow curl.
  • The isolate runs with nodejs_compat for node:zlib (gzip and file sniffing, and the stored disk) and the Buffer global; nothing else from Node is in the build. bun run build in the workspace writes .generated/runtime.txt from src/runtime.ts, and the host ships it as a text module.
  • Limits are in apps/cloudflare-worker/src/limits.ts, which core imports to clamp: 1 MiB output per stream, 60 s timeout, 16 MiB stored disk compressed. A disk past the cap stays as it was and the call fails with what to clean up.
  • sst deploy ships the Worker per stage with wrangler, as cloudflareWorkers.worker, when the deploy has CLOUDFLARE_API_TOKEN. The infra repo generates WORKER_API_KEY, puts it on the Worker and in core's CLOUDFLARE_WORKER_API_KEY, and points CLOUDFLARE_WORKER_URL at it. bun test there runs the Worker in Miniflare with a real Worker Loader and R2.
  • A disk workspace (storage.provider: "disk", sandboxHoldsWorkspace in packages/convex/model/sandboxProviders.ts) is this disk: the workspace sandbox carries no storage, so the executor reserves on the workspace namespace and each folder, partition and user conversation is its own R2 object. SHELL_ONLY_SANDBOX_PROVIDERS marks the shell with no node and no sync: edit reads the file with base64, replaces in core and writes it back; glob lists with find and matches with Bun.Glob; write skips sync. harness/conversation-files.ts reads a folder as one gzipped listing (the answer is capped at 1 MiB) and replaces it in one command.

Machine​

machine-executor.ts runs bash, the computer tool and local MCP servers on a user's computer over the WebSocket the broods machine daemon holds. Frames are defined once in src/shared/machine-socket.ts, which the CLI bundles; the gateway relays them unchanged.

  • One daemon holds a record. A second is closed with 4423 naming the holder's host; --force replaces the holder, which gets 4409. Bad credentials close with 4401, an unknown sandbox name with 4404, a malformed frame with 4400.
  • A stage ticket's socket lives as long as its ticket. ready tells the daemon how long that is (expiresInMs, relative so a skewed clock cannot move it), and the daemon sends { type: "credential", token } with a freshly minted ticket at half the time left, retrying at half of what remains until core answers. Core resolves that ticket and, only if it is for the socket's own account and stage and may still write sandboxes, moves the deadline and the bearer each heartbeat checks, then sends ready again with the new expiresInMs. Any other ticket closes the socket with 4401; one core cannot check for an outage keeps the old deadline. Without a refresh the socket closes with 4419 the moment the ticket expires, and core sends no frame to it after that, even if the timer runs late. 4419 is not fatal: the daemon (and a CLI older than the refresh) reconnects with a freshly minted ticket and reclaims its record, and a command still running then fails like any disconnect. Every heartbeat also resolves the daemon's bearer again; one that no longer resolves to the account (key rotated, role session ended or revoked, account suspended) closes with 4401. A heartbeat that cannot reach Convex keeps the daemon.
  • The socket accepts a stage ticket minted from broods login, the account key, or a role session with sandboxes:write on the record. Never the runtime key or the dashboard's member ticket.
  • Frames cap at 4 MiB. A computer action has 30 s, an MCP call 60 s, bash its timeout plus 5 s.
  • Validation rejects persistent, size, snapshot and any network mode but allow-all, and a machine cannot back a workspace.

Harness adapters​

An agent with harness set runs its adapter (Claude Code, Codex, Deep Agents, OpenCode or Pi) inside its first sandbox through @broods/ai-sdk-sandbox. Only a persistent lambda or sandbox qualifies; src/harness/ai-sdk-harness/sandbox.ts picks microvm-harness-driver.ts or workdir-harness-driver.ts. The bridge listens on port 4321.

  • The key is stored on runtimeHarnessSessions, so a conversation resumes on the machine it started on. A session ending on a shared machine neither suspends nor deletes it.
  • Claude Code, Codex, Deep Agents and OpenCode keep the conversation on the machine (each CLI's own session store), and the checkpoint holds only its id. When the machine is released, or an adapter version bump moves the conversation to a new one, the next turn starts a fresh session.
  • Pi keeps its conversation in Convex: Pi appends each session entry through sessionStorage (adapters/pi.ts), the Broods Session holds the entries until saveHarnessSession, and the checkpoint holds only the leaf entry id and the last entry sequence. A turn usually costs one save and one paged read whatever the history length; a turn over one transaction's budget (4 MiB or 100 entries) stages the rest in appendHarnessSessionEntries first. When Pi's machine is released, openAiSdkHarnessSession resumes the same session on a fresh machine and logs it; the other adapters start a fresh session. An entry over 512 KiB is stored with every image swapped for [attachment not retained] and every long string truncated, in one pass.
  • Neither provider streams processes, so harness-shell-process.ts runs the adapter detached and file-spooled; short shell calls poll its output in 180 KiB chunks every 25 ms.
  • Core holds a whole file in memory for each harness file read, so a read of a file over 25 MiB (a symlink counts as the file it points to) fails with SandboxFileTooLargeError before its bytes reach core; the harness reads a part of it with a shell command instead.
  • Core also holds a harness command's stdout and stderr whole, because adapters parse them (Pi's find and ls, listDirectory). Each is capped at 16 MiB for core's memory, not the model's context: past it core kills the command, stops polling it, and fails the call with SandboxOutputTooLargeError instead of returning a part. The cap bounds one stream, not the pod: several large commands at once can still add up.
  • On lambda, WebSocket ingress needs the AWS token and port as subprotocols, so microvm-websocket-proxy.ts hands the adapter a loopback URL and adds them upstream. The token stays in core and the upstream header, never a URL or log.
  • Live state (size, other runs holding the machine, and on a harness turn CPU, memory and disk from one probe) goes at the end of <environment>. live-status.ts keeps occupancy in memory, which is correct because core runs one replica.

Opt-in live tests, each cleaning up its own sandboxes:

cd apps/core
MICROVM_HARNESS_TEST=1 bun test tests/sandbox-microvm-harness.integration.test.ts
WORKDIR_TEST_URL=... WORKDIR_TEST_KEY=... bun test tests/workdir-harness.integration.test.ts

Snapshot status model​

sandboxSnapshots.status as core writes it:

The schema also allows pending, pulling, inactive and error, and the dashboard has a tone for each, but no core path writes them today.

Security review notes (2026-09)​

Findings from the host-boundary review, each closed in the harness:

  • options.docker must be a boolean and is accepted on the sandbox provider only.
  • lambda refuses restricted allowlists at validation instead of launching on the shared connector with a warning.
  • Every executor merges env through one helper that drops reserved keys.
  • A BYO bucket needs storage.prefix, and mount credentials are scoped to bucket/prefix/*.
  • The background-job callback token rides the exec environment, not the script text.
  • One MicroVM holds one workspace.
  • Workspace bash rejects literal .. traversal as a guardrail only. A shell expands $'\x2e\x2e' or $(printf ..) after the check. Containment comes from the VM and the prefix-scoped mount credentials. Writes to absolute paths outside the mount are refused, except /tmp and /var/tmp, unless the workspace runs on the agent's own persistent sandbox.