Skip to main content

Judge (Proposal)

Not implemented

This is a design note, not shipped behavior. No judge config key exists, ctx.judge is not on the hook context, and nothing in core resolves an evaluation model. The page records a proposed surface and the decisions behind it so the shape can be argued with before any of it is built.

The gap​

Broods makes three kinds of runtime model call, and none of them is a decision. Every classify, route, gate, or rank today is either the full agent model spending a turn in its tool loop, or a regex.

That leaves the cheap middle empty. A hook can already drop a message that matches a keyword, but dropping one that is spam by meaning costs a whole agent run. Deciding which of four agents should answer costs a whole agent run. Scoring a tool result before it goes back into context costs a whole agent run, or it does not happen.

What Jev is​

Jev is TypeSafe AI's System One model. It answers a question instead of writing text. No tokens stream back, and the response is a calibrated number. Roughly 100ms per call.

It replaces nothing the agent model does. It fills the tier below it.

Question typeCriteriaAnswer
booleanoptional true and false descriptionsprobability, 0 to 1
choiceup to 255 labeled optionschoice, plus the full distribution
scorean ordered list of 2 to 10 levels, from index 0score, fractional

Every question also carries instructions, the question itself.

Questions in one request evaluate independently and in parallel, so asking five things costs about what asking one costs. That matters for the shape of the API below: batch the questions, do not chain the calls.

The Vercel AI SDK already ships this kind of call as experimental_evaluate against an EvaluationModelV4 provider, and ai plus @ai-sdk/gateway are already installed in apps/core. The Gateway route needs no new dependency.

Configuring it​

One block, shaped like config.model, resolved the same way.

import { defineAgent } from "broods";

export const agent = defineAgent({
name: "guarded-agent",
model: { provider: "minimax", modelId: "MiniMax-M3" },
judge: {
provider: "typesafe",
modelId: "jev-1",
apiKey: "${TYPESAFE_AI_API_KEY}",
},
});

Set it once on the account and every agent inherits it. Set it on an agent only when that agent needs a different model or a different key. ${ENV_REF} resolves at sync time, the same as every other credential.

Agent config replaces the account default, it does not merge into it. A half-merged credential, a new baseURL still carrying the old key, fails in a way nobody finds until production.

There is no separate judges table and no judge id to reference. A table that exists to hold one row is not worth its CRUD.

Calling it​

In a hook, judge arrives on ctx next to fetch. It has to. The isolate rejects bare imports, so hook code cannot import { experimental_evaluate } from "ai" itself. Core holds the model and the key, and bridges one function across, exactly as it does for ctx.fetch. The questions take the SDK's own shape, so core passes them straight through.

hooks: {
onMessageReceived: async (ctx, event) => {
const { answers } = await ctx.judge({
state: event.text,
questions: {
spam: {
type: "boolean",
instructions: "Is this spam or an automated promotion?",
},
},
});

return answers.spam.probability > 0.9 ? { drop: true } : undefined;
},
}

Several questions, one round trip:

const { answers } = await ctx.judge({
state: event.text,
questions: {
spam: { type: "boolean", instructions: "Is this spam?" },
language: {
type: "choice",
instructions: "Which language is this written in?",
criteria: { en: "English", vi: "Vietnamese", other: "Anything else" },
},
urgency: {
type: "score",
instructions: "How soon does this need an answer?",
criteria: [
"Can wait days",
"Should be handled today",
"Needs an answer now",
],
},
},
});

Tighten, never loosen​

A judge answer may narrow what the agent does. It may never grant capability the agent did not already have. This is the same rule a channel record follows, and it exists for the same reason: state is attacker-controlled text, and Jev is a classifier, not a guard. It does not treat its input as hostile, so a message that argues with the criteria can move the number.

AllowedRefused
drop a messageun-drop one the policy dropped
deny a tool callallow a tool the policy denies
lower a confidence or a scoreapprove an action awaiting a human
block a reply before it sendswiden a subagent's visibility

Bounded that way, a manipulated answer costs availability at worst. The agent declines something it would have done. It never does something it could not.

Hooks are best effort by design: one that throws or times out is skipped, and the run continues unmutated. A judge deny inherits that, so it is a filter, not a control. Anything that must hold belongs in an enforced policy, which fails closed.

Where it fires​

Judge is available wherever ctx is, so every hook in Hooks can call it. What the answer is allowed to change is whatever that hook could already change.

HookUseful judge question
onMessageReceivedis this worth waking the agent for
onMessageSendingdoes this reply leak something it should not
onStartwhich of these skills matter for this conversation
onToolCalldoes this argument look destructive
onToolResultis this result relevant enough to keep in context
onSubagentFinishdid the child actually answer the question
onStepFinish, onError, onApprovalobserve only, so judge can log or alert, not steer

There is no sandbox hook and no connection hook. AGENT_HOOK_EVENT_NAMES has twelve entries and none of them covers sandbox lifecycle or a channel connection opening. Sandbox startup lives in core's account handler; connections live in the forwarders. Judging either means adding hook events first, which is its own piece of work and not assumed here.

Open decisions​

QuestionOptionsLeaning
Judge timeoutfixed in core, or configurable per agentfixed. A judge slower than the decision it saves is a bug, not a setting
Failure shapethrow like ctx.fetch, or return nullnull. Tighten-only means a missing answer is just "do not tighten", and every call site would wrap a throw anyway
Per-run call budgetnone, or a cap like the byte capsnone until a use case shows a loop calling it per token
Retries0, or the SDK default0. A retry costs more than the decision is worth
Observabilitynew spans, or reuse the model-call span attributesreuse, and record response.modelId so a model swap is visible in history

Before this gets built​

The design has no use case attached to it yet, which is why it reads as abstract. One named path, end to end, with a measured baseline from bench/, decides whether any of this is worth building. Shadow mode against those baselines is the cheap way to find out: run the judge, record what it would have decided, change nothing.

Prior research and the interactive mock: