CI/CD
Every GitHub Actions workflow in .github/workflows/, what fires it and what it ships. For contributors and operators.
Branch flow
- Work lands on
devthrough pull requests. Do not deploy by hand; push todevand let the workflows run. mainis protected and only moves by fast-forward fromdev, throughpromote.yaml.- The promote push uses
GITHUB_TOKEN, which fires noon: pushworkflow, so promote dispatches each production workflow itself. ci.yamlandlean-verification.yamlhave no path filter onpull_request, so a docs-only pull request still gets every required check.
Promote
Convex goes first so no service runs ahead of the schema it calls. npm goes last so npm i broods never pulls a client ahead of the backend.
Workflows
| Workflow | Trigger | What it does |
|---|---|---|
ci.yaml | every pull request; pushes to dev | validate (lint, format, types, unit tests, build), secrets-scan (gitleaks), dependency-audit (high/critical bun audit advisories without a reviewed, dated exception in scripts/audit-gate.ts), performance, app-surfaces. PRs only: local-stack, docs |
real-model.yaml | the real-model label on a pull request; manual | local-stack verify with ANTHROPIC_API_KEY, so the model cases call Claude Haiku 5.5 (and Sonnet 5.5 in subagent-model). Billed per call, so it runs only on request |
jittest.yaml | the jittest label on a pull request or an issue; manual | bun run jittest: Claude Code writes tests that pass on the base and target what the change could break, the script runs them on both sides, and a failure that survives the false-positive rules and the judges becomes one question on the pull request. Every model call goes through a 9router tunnel, which spends the subscriptions connected to it, so the job fails when that router is off. Judges are asked in order and stop once two agree: a free opencode model, GPT-6.1 Sol, then Claude Sonnet 5.5. On an issue written by a maintainer it writes a test that fails on dev for the reported reason, names the commit that broke it, and answers on the issue. Never blocks. It runs only on request |
lean-verification.yaml | every pull request; pushes to dev; daily 03:23 UTC; manual | lake build of the models in verification/, the pinned statements and mirrors.json checks, then bun run verification:test. The daily run uses a random oracle seed and 50,000 cases |
agent-context-budget.yaml | every pull request, and its label changes | Fails when a value in apps/core/tests/agent-context-budget.json is above the base branch's, unless the pull request has the agent-context-increase label |
check-broods-sdk.yaml | PRs and dev pushes touching the SDK, demos, root package files or npm workflows | Typecheck, test, build and dry-run pack of the SDK |
codeql.yml | pushes and PRs to dev and main; Tuesdays 17:35 UTC | CodeQL analysis |
deploy-convex.yaml | dev or main pushes touching packages/convex or root package files; manual | convex deploy. main refuses a non-prod: key, or a self-hosted URL other than https://convex-api.broods.app |
deploy.yaml | dev or main pushes, except docs, dashboard and markdown only; manual | SST deploy of the AWS data plane and the Cloudflare Workers. See deploy pipeline |
build-core.yaml | dev or main pushes touching core, Convex or root package files; manual | Build broods-core, roll it after deploy-convex and deploy |
build-gateway.yaml | dev or main pushes touching the gateway or files it imports; manual | Check, test, build broods-gateway, roll it after core's rollout |
build-dashboard.yaml | dev or main pushes touching dashboard, SDK or Convex; manual | Build broods-dashboard with NEXT_PUBLIC_* args, roll it after deploy-convex |
build-discord-forwarder.yaml | dev or main pushes touching the forwarder or files it imports; manual | Build broods-discord-forwarder. Rolls from main, or a manual run with rollout ticked |
rollout.yaml | called by the build workflows | See rollouts |
e2e-dashboard.yaml | after "Build Dashboard Image" succeeds on dev or main; manual | Signed-in Playwright suites against the deployed dashboard |
opa-policy-check.yaml | dev or main pushes touching the rego or its cases; daily 06:17 UTC | Checks broods_authz.rego parses, the deployed OPA serves the same rego, and decides scripts/opa-decision-cases.ts as expected |
deploy-docs.yaml | main pushes touching apps/docs; manual | Build this site, sync to DOCS_S3_BUCKET, invalidate CloudFront |
publish-npm.yaml | main pushes touching the SDK or root package files; manual; promote | Publish packages/broods by npm Trusted Publishing when the version is new, tag broods-v*, cut a release |
crash-recovery.yaml | daily 01:37 UTC; manual | bun run local:crash on the local stack. A failure keeps one crash-recovery issue open and hands it to Claude Code, which opens a fix pull request or comments why not |
drift-cleanup.yaml | daily 03:00 UTC; manual | See drift cleanup |
dependency-watch.yaml | Mondays 06:00 UTC; manual | Reports AI SDK family and esbuild releases into one rolling issue |
promote.yaml | manual only | See promote |
Each forwarder is one release serving every config plane, prod included, so a dev push only builds and tests it. To try a forwarder change first, run its build workflow on the branch with rollout ticked; that serves prod traffic too. The forwarder reads channel/connections:listConnections from every config plane, and the rollout waits only for this branch's Convex deploy, so a ticked branch must not change that query.
Lean verification
The Lean models in verification/Broods/ cover selected gateway routing, run lifecycle, async result, cron and manifest sync contracts. They do not prove the whole product, nor that the models match the TypeScript. Change a mirrored implementation and update the model in the same PR.
Rollouts
A green image build deploys nothing by itself. The pods live in a k3s cluster owned by beeblastco/infra.
| Image | Infra workflow, dev / main | Waits for |
|---|---|---|
| core | deploy-core-dev.yaml / deploy-core.yaml | deploy-convex.yaml, deploy.yaml |
| gateway | deploy-gateway-dev.yaml / deploy-gateway.yaml | build-core.yaml, core's rollout |
| dashboard | deploy-dashboard-dev.yaml / deploy-dashboard.yaml | deploy-convex.yaml |
| discord-forwarder | deploy-discord-forwarder.yaml, main or ticked only | deploy-convex.yaml |
- A commit that did not trigger an awaited workflow is judged by the branch's newest push run of it, so a failing backend deploy blocks every image behind it.
- The gateway waits for core because it trusts core's answer about each observability and agent socket's credential, and refuses one missing a field it needs (an observability scope with no ticket expiry is a 502). The agent socket asks
/v1/internal/agent-socket-scope, which an older core does not serve. A gateway that rolled first would refuse every such socket until core caught up. - Images are
ghcr.io/beeblastco/broods-*, tagged with the commit sha plus a floatingdevormaintag. - Each build workflow runs one at a time per branch, in push order, and a newer push replaces a run still waiting, so the newest commit rolls last. An awaited run cancelled that way is judged by the newer run that replaced it, so an image whose prerequisite was replaced still rolls once the newer one succeeds. A manual
deploy.yamlrun titled for another stage (Deploy to <stage>) never counts, by the same rule as drift cleanup. - The rollout job has a 40 minute budget: 20 for prerequisites, 20 for the roll. It needs
INFRA_DISPATCH_TOKEN.
Deploy pipeline
| Source | SST stage | Region | GitHub environment |
|---|---|---|---|
push to dev | dev | DEV_AWS_REGION, default eu-west-1 | development |
push to main | production-eu-west-1 | eu-west-1 | production |
manual, stage: production | production-eu-west-1 | eu-west-1 | production |
| manual, other stage | that stage | DEV_AWS_REGION | development |
- Production stages deploy only from
main.production_targets()indeploy.yamllistseu-west-1alone.microvmPrereqsEnabledinsst.config.tsskips the MicroVM pieces inap-southeast-1. - The deploy job shares the
sst-<stage>concurrency group with drift cleanup, so the two never touch a stage at once. Neither runssst unlock: a lock left by a crashed run, or held by a deploy started outside CI, fails the deploy. Clear it by hand after checking nothing runs. - With a Cloudflare token,
sst deployitself ships the Cloudflare MCP runtime, the sandbox bridge and the V8 Worker, each named per stage, by running its workspace'sdeployscript (deployCloudflareWorkerinsst.config.ts). A Worker redeploys only when its files or a var it gets change, orbun.lockfor the MCP and V8 Workers. The MCP script creates thebroods-mcp-bundlesR2 bucket if missing, sets its 30-day expiry and setsBUNDLE_ORIGINto the stage's tool-bundles bucket; the V8 Worker's script createsbroods-worker-state; the bridge's script builds its Container image with the runner's Docker only when the registry lacksbroods-sandbox:<Dockerfile hash>(first deploy to the account, or aDockerfilechange), shared by every stage, and deploys it by digest. They ship code only: the infra repo generates each key and writes it to the Worker and to core. A failed Worker deploy fails the stage deploy like any other resource.
Secrets and variables
Environment-scoped values resolve from development or production by branch.
| Name | Kind | Used by |
|---|---|---|
AWS_ROLE_ARN | variable | deploy, drift-cleanup, deploy-docs |
AWS_ACCOUNT_ID, PROJECT_NAME, PROJECT_OWNER_EMAIL | variable | ci, deploy, drift-cleanup |
DEV_AWS_REGION | variable | ci, deploy |
CONVEX_URL, CONVEX_DEPLOY_KEY | secret, per env | deploy, drift-cleanup; the key also deploy-convex |
CONVEX_SELF_HOSTED_URL, CONVEX_SELF_HOSTED_ADMIN_KEY | secret, per env | deploy-convex, instead of a deploy key |
OTEL_EXPORTER_OTLP_HEADERS | secret | deploy, drift-cleanup. Unset skips the sandbox log forwarder |
SANDBOX_IMAGE_READY_DEV, SANDBOX_IMAGE_READY_PRODUCTION[_<REGION>], SANDBOX_IMAGE_SOURCE_REGION | variable | deploy; the readiness flags also drift-cleanup. SANDBOX_IMAGE_READY is the legacy fallback for the dev flag |
CLOUDFLARE_API_TOKEN, CLOUDFLARE_ACCOUNT_ID | secret, variable, per env | deploy, drift reconcile. Unset skips the Cloudflare Workers. A user token, not an account token |
INFRA_DISPATCH_TOKEN | secret | every build workflow. Fine-grained PAT on beeblastco/infra, Actions read and write |
NEXT_PUBLIC_CONVEX_URL, NEXT_PUBLIC_WORKOS_REDIRECT_URI, NEXT_PUBLIC_BROODS_BASE_URL | variable, per env | build-dashboard; the Convex URL also ci, the redirect URI's origin also deploy-convex (DASHBOARD_ORIGIN) |
STRIPE_PRO_PRICE_ID, STRIPE_SEAT_PRICE_ID | variable, per env | deploy-convex, onto the Convex env. Unset leaves the backend's value alone |
WORKOS_API_KEY, WORKOS_CLIENT_ID, WORKOS_COOKIE_PASSWORD | secret | ci signed-in dashboard tests |
DASHBOARD_E2E_EMAIL, DASHBOARD_E2E_PASSWORD, DASHBOARD_E2E_PROJECT_ID | secret, variable | ci, e2e-dashboard |
ANTHROPIC_API_KEY | secret | real-model, crash-recovery |
JITTEST_ROUTER_URL, JITTEST_ROUTER_KEY | variable, secret | jittest: the 9router tunnel that serves the generator and every judge |
OPA_BASE_URL, OPA_API_TOKEN | variable, secret | opa-policy-check |
DOCS_S3_BUCKET, DOCS_DOMAIN, DOCS_AWS_REGION | variable | deploy-docs |
Runtime secrets for the containers are not GitHub secrets. They live in k8s secrets referenced by the infra repo and in the Convex deployment env. See self-hosting.
SDK versioning
Nobody hand-edits or commits the SDK version. publish-npm.yaml runs packages/broods/scripts/next-version.ts over the conventional-commit subjects since the newest broods-v* tag, of every commit that touched the package or the core and Convex files bundled into it (SHIPPED_PATHS, the bundled paths check-broods-sdk.yaml watches):
| Subject | Bump on 0.x | Bump from 1.0 |
|---|---|---|
! or BREAKING CHANGE | minor | major |
feat: | minor | minor |
| anything else | patch | patch |
The subject line is the release note. A package.json version above the last tag is released as is, which is how a major like 1.0.0 ships. The publish is skipped when no commit touched those paths or the version is already on npm. Trusted Publishing is configured for organization beeblastco, repository broods, workflow publish-npm.yaml; never commit .npmrc files or npm tokens. See packages/broods/AGENTS.md.
Drift cleanup
drift-cleanup.yaml checks each stage against sst.config.ts every night, so resources whose code was removed stop charging.
| Stage | Checked out from | Environment | On drift |
|---|---|---|---|
dev | last successful deploy.yaml on dev | development | sst deploy |
production-eu-west-1 | last successful deploy.yaml on main | production | job fails |
- The commit is the newest successful
deploy.yamlrun on the branch that shipped this stage, never the branch head, so a commit that failed validation or is still deploying is neither reported as drift nor deployed here. Push runs count (dev), and so do manual runs (promote shipsmainthat way) unless they named another stage, whichdeploy.yamlputs in the run title (Deploy to <stage>). No such run fails the job. - A production dispatch from any ref but
mainfails fast. Reconcile production drift throughdeploy.yamlfrommain. - The job resolves
SANDBOX_IMAGE_READYper stage the waydeploy.yamldoes, sosst.config.tsimports the sandbox ECR repo in both. SST ignores tag changes on an imported resource; a plan without the import reports the repo's tags as drift every night. - Each job takes the same
sst-<stage>lock asdeploy.yaml. A lock left by a crashed run fails the job; clear it by hand after checking nothing runs. - The plan stays in the job log, with the AWS account id masked. The repo is public, so there is no artifact.
- It sees only resources in Pulumi state and never bootstraps sandbox images; the mirror step lives in
deploy.yamlalone. - A new stage must be added to
STAGESin the workflow and to its dispatch options, or drift cleanup never sees it.