Skip to main content

CI/CD

Every GitHub Actions workflow in .github/workflows/, what fires it and what it ships. For contributors and operators.

Branch flow​

  • Work lands on dev through pull requests. Do not deploy by hand; push to dev and let the workflows run.
  • main is protected and only moves by fast-forward from dev, through promote.yaml.
  • The promote push uses GITHUB_TOKEN, which fires no on: push workflow, so promote dispatches each production workflow itself.
  • ci.yaml and lean-verification.yaml have no path filter on pull_request, so a docs-only pull request still gets every required check.

Promote​

Convex goes first so no service runs ahead of the schema it calls. npm goes last so npm i broods never pulls a client ahead of the backend.

Workflows​

WorkflowTriggerWhat it does
ci.yamlevery pull request; pushes to devvalidate (lint, format, types, unit tests, build), secrets-scan (gitleaks), dependency-audit (high/critical bun audit advisories without a reviewed, dated exception in scripts/audit-gate.ts), performance, app-surfaces. PRs only: local-stack, docs
real-model.yamlthe real-model label on a pull request; manuallocal-stack verify with ANTHROPIC_API_KEY, so the model cases call Claude Haiku 5.5 (and Sonnet 5.5 in subagent-model). Billed per call, so it runs only on request
jittest.yamlthe jittest label on a pull request or an issue; manualbun run jittest: Claude Code writes tests that pass on the base and target what the change could break, the script runs them on both sides, and a failure that survives the false-positive rules and the judges becomes one question on the pull request. Every model call goes through a 9router tunnel, which spends the subscriptions connected to it, so the job fails when that router is off. Judges are asked in order and stop once two agree: a free opencode model, GPT-6.1 Sol, then Claude Sonnet 5.5. On an issue written by a maintainer it writes a test that fails on dev for the reported reason, names the commit that broke it, and answers on the issue. Never blocks. It runs only on request
lean-verification.yamlevery pull request; pushes to dev; daily 03:23 UTC; manuallake build of the models in verification/, the pinned statements and mirrors.json checks, then bun run verification:test. The daily run uses a random oracle seed and 50,000 cases
agent-context-budget.yamlevery pull request, and its label changesFails when a value in apps/core/tests/agent-context-budget.json is above the base branch's, unless the pull request has the agent-context-increase label
check-broods-sdk.yamlPRs and dev pushes touching the SDK, demos, root package files or npm workflowsTypecheck, test, build and dry-run pack of the SDK
codeql.ymlpushes and PRs to dev and main; Tuesdays 17:35 UTCCodeQL analysis
deploy-convex.yamldev or main pushes touching packages/convex or root package files; manualconvex deploy. main refuses a non-prod: key, or a self-hosted URL other than https://convex-api.broods.app
deploy.yamldev or main pushes, except docs, dashboard and markdown only; manualSST deploy of the AWS data plane and the Cloudflare Workers. See deploy pipeline
build-core.yamldev or main pushes touching core, Convex or root package files; manualBuild broods-core, roll it after deploy-convex and deploy
build-gateway.yamldev or main pushes touching the gateway or files it imports; manualCheck, test, build broods-gateway, roll it after core's rollout
build-dashboard.yamldev or main pushes touching dashboard, SDK or Convex; manualBuild broods-dashboard with NEXT_PUBLIC_* args, roll it after deploy-convex
build-discord-forwarder.yamldev or main pushes touching the forwarder or files it imports; manualBuild broods-discord-forwarder. Rolls from main, or a manual run with rollout ticked
rollout.yamlcalled by the build workflowsSee rollouts
e2e-dashboard.yamlafter "Build Dashboard Image" succeeds on dev or main; manualSigned-in Playwright suites against the deployed dashboard
opa-policy-check.yamldev or main pushes touching the rego or its cases; daily 06:17 UTCChecks broods_authz.rego parses, the deployed OPA serves the same rego, and decides scripts/opa-decision-cases.ts as expected
deploy-docs.yamlmain pushes touching apps/docs; manualBuild this site, sync to DOCS_S3_BUCKET, invalidate CloudFront
publish-npm.yamlmain pushes touching the SDK or root package files; manual; promotePublish packages/broods by npm Trusted Publishing when the version is new, tag broods-v*, cut a release
crash-recovery.yamldaily 01:37 UTC; manualbun run local:crash on the local stack. A failure keeps one crash-recovery issue open and hands it to Claude Code, which opens a fix pull request or comments why not
drift-cleanup.yamldaily 03:00 UTC; manualSee drift cleanup
dependency-watch.yamlMondays 06:00 UTC; manualReports AI SDK family and esbuild releases into one rolling issue
promote.yamlmanual onlySee promote

Each forwarder is one release serving every config plane, prod included, so a dev push only builds and tests it. To try a forwarder change first, run its build workflow on the branch with rollout ticked; that serves prod traffic too. The forwarder reads channel/connections:listConnections from every config plane, and the rollout waits only for this branch's Convex deploy, so a ticked branch must not change that query.

Lean verification​

The Lean models in verification/Broods/ cover selected gateway routing, run lifecycle, async result, cron and manifest sync contracts. They do not prove the whole product, nor that the models match the TypeScript. Change a mirrored implementation and update the model in the same PR.

Rollouts​

A green image build deploys nothing by itself. The pods live in a k3s cluster owned by beeblastco/infra.

ImageInfra workflow, dev / mainWaits for
coredeploy-core-dev.yaml / deploy-core.yamldeploy-convex.yaml, deploy.yaml
gatewaydeploy-gateway-dev.yaml / deploy-gateway.yamlbuild-core.yaml, core's rollout
dashboarddeploy-dashboard-dev.yaml / deploy-dashboard.yamldeploy-convex.yaml
discord-forwarderdeploy-discord-forwarder.yaml, main or ticked onlydeploy-convex.yaml
  • A commit that did not trigger an awaited workflow is judged by the branch's newest push run of it, so a failing backend deploy blocks every image behind it.
  • The gateway waits for core because it trusts core's answer about each observability and agent socket's credential, and refuses one missing a field it needs (an observability scope with no ticket expiry is a 502). The agent socket asks /v1/internal/agent-socket-scope, which an older core does not serve. A gateway that rolled first would refuse every such socket until core caught up.
  • Images are ghcr.io/beeblastco/broods-*, tagged with the commit sha plus a floating dev or main tag.
  • Each build workflow runs one at a time per branch, in push order, and a newer push replaces a run still waiting, so the newest commit rolls last. An awaited run cancelled that way is judged by the newer run that replaced it, so an image whose prerequisite was replaced still rolls once the newer one succeeds. A manual deploy.yaml run titled for another stage (Deploy to <stage>) never counts, by the same rule as drift cleanup.
  • The rollout job has a 40 minute budget: 20 for prerequisites, 20 for the roll. It needs INFRA_DISPATCH_TOKEN.

Deploy pipeline​

SourceSST stageRegionGitHub environment
push to devdevDEV_AWS_REGION, default eu-west-1development
push to mainproduction-eu-west-1eu-west-1production
manual, stage: productionproduction-eu-west-1eu-west-1production
manual, other stagethat stageDEV_AWS_REGIONdevelopment
  • Production stages deploy only from main. production_targets() in deploy.yaml lists eu-west-1 alone. microvmPrereqsEnabled in sst.config.ts skips the MicroVM pieces in ap-southeast-1.
  • The deploy job shares the sst-<stage> concurrency group with drift cleanup, so the two never touch a stage at once. Neither runs sst unlock: a lock left by a crashed run, or held by a deploy started outside CI, fails the deploy. Clear it by hand after checking nothing runs.
  • With a Cloudflare token, sst deploy itself ships the Cloudflare MCP runtime, the sandbox bridge and the V8 Worker, each named per stage, by running its workspace's deploy script (deployCloudflareWorker in sst.config.ts). A Worker redeploys only when its files or a var it gets change, or bun.lock for the MCP and V8 Workers. The MCP script creates the broods-mcp-bundles R2 bucket if missing, sets its 30-day expiry and sets BUNDLE_ORIGIN to the stage's tool-bundles bucket; the V8 Worker's script creates broods-worker-state; the bridge's script builds its Container image with the runner's Docker only when the registry lacks broods-sandbox:<Dockerfile hash> (first deploy to the account, or a Dockerfile change), shared by every stage, and deploys it by digest. They ship code only: the infra repo generates each key and writes it to the Worker and to core. A failed Worker deploy fails the stage deploy like any other resource.

Secrets and variables​

Environment-scoped values resolve from development or production by branch.

NameKindUsed by
AWS_ROLE_ARNvariabledeploy, drift-cleanup, deploy-docs
AWS_ACCOUNT_ID, PROJECT_NAME, PROJECT_OWNER_EMAILvariableci, deploy, drift-cleanup
DEV_AWS_REGIONvariableci, deploy
CONVEX_URL, CONVEX_DEPLOY_KEYsecret, per envdeploy, drift-cleanup; the key also deploy-convex
CONVEX_SELF_HOSTED_URL, CONVEX_SELF_HOSTED_ADMIN_KEYsecret, per envdeploy-convex, instead of a deploy key
OTEL_EXPORTER_OTLP_HEADERSsecretdeploy, drift-cleanup. Unset skips the sandbox log forwarder
SANDBOX_IMAGE_READY_DEV, SANDBOX_IMAGE_READY_PRODUCTION[_<REGION>], SANDBOX_IMAGE_SOURCE_REGIONvariabledeploy; the readiness flags also drift-cleanup. SANDBOX_IMAGE_READY is the legacy fallback for the dev flag
CLOUDFLARE_API_TOKEN, CLOUDFLARE_ACCOUNT_IDsecret, variable, per envdeploy, drift reconcile. Unset skips the Cloudflare Workers. A user token, not an account token
INFRA_DISPATCH_TOKENsecretevery build workflow. Fine-grained PAT on beeblastco/infra, Actions read and write
NEXT_PUBLIC_CONVEX_URL, NEXT_PUBLIC_WORKOS_REDIRECT_URI, NEXT_PUBLIC_BROODS_BASE_URLvariable, per envbuild-dashboard; the Convex URL also ci, the redirect URI's origin also deploy-convex (DASHBOARD_ORIGIN)
STRIPE_PRO_PRICE_ID, STRIPE_SEAT_PRICE_IDvariable, per envdeploy-convex, onto the Convex env. Unset leaves the backend's value alone
WORKOS_API_KEY, WORKOS_CLIENT_ID, WORKOS_COOKIE_PASSWORDsecretci signed-in dashboard tests
DASHBOARD_E2E_EMAIL, DASHBOARD_E2E_PASSWORD, DASHBOARD_E2E_PROJECT_IDsecret, variableci, e2e-dashboard
ANTHROPIC_API_KEYsecretreal-model, crash-recovery
JITTEST_ROUTER_URL, JITTEST_ROUTER_KEYvariable, secretjittest: the 9router tunnel that serves the generator and every judge
OPA_BASE_URL, OPA_API_TOKENvariable, secretopa-policy-check
DOCS_S3_BUCKET, DOCS_DOMAIN, DOCS_AWS_REGIONvariabledeploy-docs

Runtime secrets for the containers are not GitHub secrets. They live in k8s secrets referenced by the infra repo and in the Convex deployment env. See self-hosting.

SDK versioning​

Nobody hand-edits or commits the SDK version. publish-npm.yaml runs packages/broods/scripts/next-version.ts over the conventional-commit subjects since the newest broods-v* tag, of every commit that touched the package or the core and Convex files bundled into it (SHIPPED_PATHS, the bundled paths check-broods-sdk.yaml watches):

SubjectBump on 0.xBump from 1.0
! or BREAKING CHANGEminormajor
feat:minorminor
anything elsepatchpatch

The subject line is the release note. A package.json version above the last tag is released as is, which is how a major like 1.0.0 ships. The publish is skipped when no commit touched those paths or the version is already on npm. Trusted Publishing is configured for organization beeblastco, repository broods, workflow publish-npm.yaml; never commit .npmrc files or npm tokens. See packages/broods/AGENTS.md.

Drift cleanup​

drift-cleanup.yaml checks each stage against sst.config.ts every night, so resources whose code was removed stop charging.

StageChecked out fromEnvironmentOn drift
devlast successful deploy.yaml on devdevelopmentsst deploy
production-eu-west-1last successful deploy.yaml on mainproductionjob fails
  • The commit is the newest successful deploy.yaml run on the branch that shipped this stage, never the branch head, so a commit that failed validation or is still deploying is neither reported as drift nor deployed here. Push runs count (dev), and so do manual runs (promote ships main that way) unless they named another stage, which deploy.yaml puts in the run title (Deploy to <stage>). No such run fails the job.
  • A production dispatch from any ref but main fails fast. Reconcile production drift through deploy.yaml from main.
  • The job resolves SANDBOX_IMAGE_READY per stage the way deploy.yaml does, so sst.config.ts imports the sandbox ECR repo in both. SST ignores tag changes on an imported resource; a plan without the import reports the repo's tags as drift every night.
  • Each job takes the same sst-<stage> lock as deploy.yaml. A lock left by a crashed run fails the job; clear it by hand after checking nothing runs.
  • The plan stays in the job log, with the AWS account id masked. The repo is public, so there is no artifact.
  • It sees only resources in Pulumi state and never bootstraps sandbox images; the mirror step lives in deploy.yaml alone.
  • A new stage must be added to STAGES in the workflow and to its dispatch options, or drift cleanup never sees it.