Languid/docs/evals

Using Languid · Evals

Evals

An eval checks what an agent does: given these records and these messages, does the agent end up where it should? Evals are part of the agent. Its cases, runs and results live in its own space, next to its memory, and every trial tests the real agent: its system prompt, blocks, tools, policy and model.

Evals do three jobs:

The records

Everything an agent does is already a record (messages, tool calls, script calls, blocks), so a trial is its own evidence. Cases and results are records too.

RecordWhat it isKey
ai.languid.eval.caseStarting records, then steps. A step is one or more messages, then checks.a slug, pin-block
ai.languid.eval.suiteA named, versioned set of cases pinned by CID, plus how many trials and what counts as passing.a slug
ai.languid.eval.runOne invocation: what was tested (harness commit, model, tool hash, judge, config CID), which cases, and a summary when it ends.tid
ai.languid.eval.resultOne trial of one case: a verdict per check, short evidence, usage, and the transcript (messages and tool calls, clipped).tid

The lexicons are in lexicons/ai.languid.eval.* and are published by languid admin publish-lexicon eval, resolved through _lexicon.eval.languid.ai.

Who wrote it

A space gives each member its own repo, so every eval record says who wrote it without an extra field. (A member can read only its own repo in the Spaces alpha, so the service, which can act for both, reads the owner's cases for the agent. It never passes on held-out ones.)

Written byMeansExample
The owner"This is what good looks like for my agent."Cases you add, results of runs you start
The agent"I think this is what good looks like."Cases it writes about itself, results of runs it starts
Anyone, in publicA general case or suite anyone can uselanguid's own verbosity and memory suites

Readers weigh these differently. The gate (below) counts only the owner's cases and the holdout. The corpus export labels each line with who wrote the case and who graded it, so you can choose what to train on.

Where cases come from

Written by hand

A case is a JSON file. This one checks that the agent answers a quick question briefly:

{
  "$type": "ai.languid.eval.case",
  "name": "Short answer to a short question",
  "tags": ["verbosity"],
  "steps": [{
    "turns": ["What's 17 times 23?"],
    "checks": [
      { "id": "correct", "grader": { "$type": "ai.languid.eval.defs#scriptCheck",
        "code": "return /\\b391\\b/.test(run.reply)" } },
      { "id": "concise", "grader": { "$type": "ai.languid.eval.defs#scriptCheck",
        "code": "const n = run.reply.split(/\\s+/).filter(Boolean).length; return n <= 12 ? 1 : Math.max(0, 1 - (n - 12) / 6)" } },
      { "id": "no_preamble", "advisory": true, "grader": { "$type": "ai.languid.eval.defs#scriptCheck",
        "code": "return !/^(sure|great|of course)/i.test(run.reply)" } }
    ]
  }],
  "createdAt": "2026-11-16T00:00:00Z"
}

Add case files to an agent with languid eval add <agent> <file | dir>…. The record key is the file name, so adding a file again replaces the case.

Saved from a conversation

Most good cases come from real use. When an agent gets something right or wrong, save that moment as a case:

The case is built from the conversation up to that message:

A case saved from a conversation is private. It holds real messages, so the runner refuses to publish it or add it to a public suite. Turn the judge check into a script check later if the rule turns out to be mechanical.

Shared suites

A suite in a public repo can be used by any agent: languid eval run <agent> --suite at://<did>/ai.languid.eval.suite/verbosity. Its cases are pinned by CID, so everyone runs the same version, and a case that has changed since the suite pinned it is refused. Their script checks run only with --trust; without it they're unscored.

Checks

Four kinds, cheapest first:

GraderChecksExample
recordChecka field of a record after the stepblock notes /pinned equals true; /value unchanged
callCheckhow often a tool or method was called, with matching argumentsblock_pin called at most 0 times
scriptCheckanything else, as JavaScript in the xrpc sandbox, read-onlysome block mentions Point Pinos; the reply is under 12 words
judgeCheckan LLM judge with a rubric, for what code can't decide"the reply admits it doesn't know"

The rules:

Running evals

languid eval add <agent> <case.json | dir>…
languid eval run <agent> [case…] [--suite <uri> [--trust]] [--trials 3] [--concurrency auto|n] [--model provider/id] [--judge provider/id]
languid eval list <agent>
languid eval show <agent> <run>

These act as the installation owner, so on a hosted service they run there (just remote eval …) for the installation owner's agents. Every owner can do the same from the agent's Evals page: run one case or all of them, run the gate, see each run trial by trial, and set the judge and the agent's daily budget.

eval run writes a run record, then runs every trial of every case, many at once. A trial mostly waits on its model, so --concurrency auto (the default) starts at 4 trials and adds one each time a trial finishes cleanly, up to what the machine can take (half its memory at 256 MB a trial, four per core, at most 64). A model call that's rate-limited halves it. --concurrency n fixes the number. Each trial:

  1. Copies the agent into a scratch space, <agent>.eval.<run>-<trial>-<case hash>, with its config, blocks and tool definitions. No account or DID is made. Conversations, eval records and computers aren't copied, so the trial starts with no history, can't read the case it's tested on, and can't reach a real computer.
  2. Seeds the case's starting records. --model switches the copy's model and leaves the agent's own alone.
  3. Runs each step. Tools that need approval are denied, since nobody is there to approve them. A step that goes past limits.seconds or limits.toolCalls is stopped, and the trial ends as timeout.
  4. Grades the checks, writes a result with the transcript, and deletes the scratch space. The run and its results are written by whoever started it, in their own repo: you, or the agent.

When every trial is done, the run gets its summary:

The command prints a line per trial and exits 1 unless pass^k is 100%, so it works in CI. The agent's Evals page in the web app shows the same.

The agent evaluating itself

The agent has the same tools you do, through methods it calls from its scripts:

MethodDoes
ai.languid.eval.listCasesLists its cases and the owner's (never the held-out ones)
ai.languid.eval.putCaseWrites a case in the agent's own repo, checked against the lexicon
ai.languid.eval.startRunStarts a run on itself in the background and returns the run's URI. changes tries a different system prompt or block values on the copies only. gate: true runs the gate.
ai.languid.eval.getRunReads a run's summary and its results, without transcripts

The default policy allows all four, in the agent's own space. The agent can't edit or delete the owner's cases, because those are in the owner's repo. It can read them, so it knows what it's held to, but it never sees the holdout. A trial has no way to start evals of its own: the methods work only through the service, and a trial isn't one of its agents.

Self-runs spend the owner's model credit. Each agent has a daily trial budget on its Evals page, 50 by default, counted from midnight UTC. Past it, startRun is refused, and the agent can ask you to run it or to raise the budget.

The improvement loop

  1. The agent spots a weakness, or you mark one with 👎, and a case gets written.
  2. The agent runs it, reads the failing results, and works out a change to its system prompt or a block.
  3. It runs the case again with changes, on copies that have the change, and runs the gate with the same changes.
  4. If both pass, it makes the change to its blocks, or asks you to change its system prompt, with the run's results to show for it.

The holdout and the gate

An agent that writes its own tests and grades its own changes will drift towards passing whatever it wrote. Two things stop that.

The holdout is a separate space, <agent>.holdout, with the owner as its only member. The agent can't read its cases, and their results are written there too. A run that includes them keeps only their summary. Add to it with languid eval holdout add <agent> <file | dir>…, or by ticking Hold out when you save a reply (--holdout from the terminal).

The gate decides whether a change the agent made to itself can stay:

languid eval gate <agent> [--trials 3]

It runs the owner's cases that have a gate check, and the held-out ones, against the agent as it is now. It passes only if pass^k is 100% on both. The agent can run it too (startRun with gate: true), with or without changes, and learns only whether it passed and the counts. Cases the agent wrote don't count towards the gate.

The corpus

languid eval export <agent> [--run <uri>] [--since 2026-11-01] [--pairs] > corpus.jsonl

Each line is one trial:

--pairs writes preference pairs instead: a passing and a failing trial of the same case, against the same config. Holdout trials are left out unless you add --holdout, so you don't train on your test set by accident.

On the dev stack

just eval <provider/model> [--trials n] [case.json | dir…]

This runs scripts/eval.ts within one standing agent, evals, which is made the first time with its state in .dev/evals. Later runs reuse it, so no new accounts are made. See its runs with LANGUID_HOME=.dev/evals node src/cli.ts eval list evals.

Borrowed

Next: CommandsThis page as Markdown