When the caller is a coin flip

You cannot unit-test your way out of the anxiety of shipping something an LLM will call. Unit tests answer did this function return the right value for these inputs? The question that keeps you up at night is different: will the model, in a phrasing we didn’t anticipate, hallucinate an argument, or reformat our output in a way that changes what it means? You can raise coverage to 100% and that question stays exactly as loud.

So we built the smallest thing that made the question quieter. It’s a few hundred lines of Python across two MCP servers, and it’s built on three constraints that turned out to matter more than any single design choice.

Two failure classes, two tests

The first constraint: separate the failures that are deterministic from the failures that require judgment.

Formatter regressions, missing fields, upstream shape drift — all deterministic. You give the server a stubbed upstream that returns known data, run a fixed sequence of tool calls, and assert that specific substrings appear in the joined output. Exact match, case sensitive. If the formatter starts capitalizing team names differently, the test fails immediately and the failure message tells you what changed. Cheap, fast, runs in CI on every PR without an API key.

Hallucination triggers, semantic drift, “the output is technically correct but no longer answers the question” — not deterministic. For those, the same scenario declares one or more criteria in plain English, and each criterion is sent to Claude Haiku 4.5 as its own PASS/FAIL grading task. The judge gets exactly one criterion and the joined tool output; it returns exactly two lines, VERDICT: PASS|FAIL and REASON: one sentence. Anything else — a preamble, markdown, a hedged verdict — is treated as FAIL.

Both assertion modes live in the same YAML scenario file. Adding a new scenario is thirteen lines of text, not a Python class:

name: find_events_basic
tool_sequence:
  - tool: find_events
    args: {}
expected_contains:
  - "North Atlantic"
  - "2025-26"
expected_judge:
  - "Output lists at least one ECNL event with a conference and a season."

A contributor who’s never seen the harness can copy the file, change the strings, and open a PR. That was the whole point.

The judge is a black box; the parser is the contract

The second constraint, and the one that shaped the most decisions: we do not try to test the model’s judgment. We test our parser’s contract with the model’s response format.

The judge system prompt is a string constant checked into the repo. It says “be strict; if the criterion is not clearly satisfied, return FAIL,” specifies the two-line format, and forbids preambles. Change the prompt, get a code review, run the tests. There is no tuning loop, no RLHF feedback, no dashboard tracking judge accuracy over time.

What is exhaustively tested is the response parser. The unit tests cover leading whitespace, mixed case in verdict tokens, missing reason lines, trailing junk, and truncated responses. Every path returns a well-formed JudgeVerdict object. Every unparseable response is a FAIL with the raw text preserved as the reason, so a human reading the failure email sees exactly what the model returned.

This division falls out of a simple observation: the parser is deterministic and we own it, so we test it like any other code. The model is probabilistic and we don’t own it, so we treat its output like data from an external service — validate the shape, fail closed on anything unexpected, log the raw response, move on. Attempting to prove the judge is “correct” would be a research program; treating it as a black box with a strict contract is a Tuesday.

Same suite, two transports

The third constraint: the same scenario file has to run against a hermetic stub in CI and against a live production instance overnight.

The mechanism is one environment variable. When MCP_EVAL_REMOTE_URL is unset, the test opens an in-process FastMCP server wired to stubs — no network, no credentials, no API cost. When it’s set, the same test connects to a live instance over Streamable HTTP with a bearer token forwarded from another environment variable. Only scenarios that declare live: true in their YAML participate in the second mode, so stub-specific expectations don’t leak into live-instance runs.

The nightly workflow schedules NWSL at 09:00 UTC and ECNL at 09:15 UTC — enough offset that when both are eventually promoted to live-instance replay, they don’t hammer the same upstream simultaneously.

No dashboard

The result of every nightly run is pytest output in a GitHub Actions log. Failures generate an email. A human reads the email and decides what to do.

We considered every layer above that: a metrics store, a trend dashboard, an automatic rollback trigger on failure rate, a Slack integration with per-scenario status. We built none of them. Every one is more infrastructure to maintain than signal to gain — the eval suite is small enough that a failure is legible on first read, and the escalation path is short enough that “human sees email, opens repo, investigates” is not a bottleneck.

The dashboard is the temptation. Refusing to build it kept the signal loop tight.

What actually changed

The specific fear got quieter. Not the general anxiety of shipping something to an agent — that’s structural — but the specific worry that a formatter change would silently degrade a tool’s usefulness, or that upstream drift would produce responses that pass every unit test and still mean the wrong thing. Those are exactly the failures the two-mode harness catches.

What it does not catch: whether the tool contract itself is well-designed for the caller. That’s the earlier essay — tools, not endpoints. Evals verify a server behaves; they don’t tell you the server should exist in that shape. Those are separate disciplines and the second one has to come first.