Assistant Test Adapters Design

Date: 2026-08-18 Status: Draft

Goal

Author a test once, against no runner in particular, and let any runner that satisfies the contract execute it.

Solution

A test definition that names a situation and never a mechanism, one outcome shape every runner reports into, and an adapter at each edge that translates both ways. A runner's job ends at the outcome; our assertions are evaluated against it afterwards, and each consumer reads a verdict out of those results under its own criteria. The contract is claim / report / release rather than run() → results, because most runners are vendors that execute remotely, may answer hours later, and may never answer at all.

This is the execution half of the data model. That document decided what a result is; this one decides how something that is not ours produces one.

The problem

A test is authored against a runner, so it can only ever run there. A Simulation v2 case is VAPI-shaped. A Hamming case is Hamming-shaped. The same scenario — a caller with a broken water heater who is not in the service area — has to be written twice to run in both places, and the two copies drift the day after they are written.

The verdicts cannot be compared even when the scenario matches. Hamming returns its guardrail's judgment. VAPI-native returns a scorer. Our judge returns a rubric verdict. Three different measurements of three different things, stored as three different blobs, so "did this behaviour regress" cannot be asked across them.

Adding a platform meant adding a lane, not an adapter. ElevenLabs support in Simulation v2 arrived as a parallel path beside the VAPI one, sharing an authoring surface and nothing underneath. The third platform would have been a third lane.

The cost compounds: every new runner multiplies the authoring surface, and no runner's results can be trended against another's.

Why now

The data model is landing, and it already assumes this shape. assistant_test_runs carries lease_expires_at, deadline_at, provider, provider_ref and a timed_out outcome — columns that are meaningless for an in-process function and exist only because a runner can die. assistant_test_assertion_results.evaluator exists because the same rubric judged by us and by a vendor are two different measurements.

The model was designed against this contract. Writing the contract down is finishing a decision already made, and doing it before the adapters are built is cheaper than reconciling two of them afterwards.

There is also a working prototype to design from rather than guess at: apps/web/lib/simv2-integration/ already separates IntegrationTestDefinition from IntegrationCaseResult, with the VAPI translation isolated in its own lane.

Decisions

A definition names a situation, never a mechanism. It carries who is calling, what they want, what they must be told, and what is true in the world. It does not carry a platform, a model, a squad id, or a vendor's request shape. If a field only means something to one runner, it does not belong in the definition.

An adapter translates in both directions, and both halves are its job. Outbound turns a definition into whatever that vendor accepts. Inbound turns whatever the vendor returns into an outcome — its output, and any evaluations it made along the way. An adapter that only does the outbound half has moved the problem rather than solved it.

claim / report / release, not run() → results. Three of four runners execute somewhere we do not control. A function signature that returns results assumes the runner is alive when the results exist, and for Hamming it is not.

A runner declares which mechanisms it can evaluate, and an assertion it cannot evaluate fails with a reason. It never skips, and it never quietly degrades to a model judgment. A skipped assertion reads as a pass in every aggregate; a silent downgrade reports a rubric verdict as if it were a config check.

Every result records its evaluator. Not for provenance decoration — because results from different evaluators are not automatically comparable, and any aggregate that mixes them without saying so is wrong.

A runner produces an outcome; it does not decide whether the test passed. Its output and its own evaluations both land in the outcome, and our assertions are judged against that afterwards. This is what lets a vendor's verdict be an input rather than the answer, and what lets the same conversation be re-judged by a better rubric without re-running the call.

The contract

A runner's job ends at an outcome. Assertions are evaluated against that outcome afterwards, as a separate act. Keeping those two apart is what makes one definition runnable anywhere: a runner has to produce an outcome, and nothing else, whether it is Hamming, a platform chat, or a function that reads a compiled config.

The outcome is what a runner produces

An outcome is one pass over one situation, and it holds two things:

• Output — what came back. Transcript, tool calls, the compiled artifact, whatever evidence that runner produces.
• The runner's own evaluations — Hamming's guardrail verdicts, a VAPI-native scorer's output, anything the runner judged for itself along the way.

Both belong to the outcome. A vendor that judges is not reporting assertion results; it is reporting an outcome that happens to contain judgments. That distinction is the whole reason a vendor with its own opinion can be adapted at all — we do not have to accept its judgment as our verdict, and we do not have to throw it away either.

A runner with no opinion produces an outcome with output and no evaluations. A runner that only has an opinion — a vendor scorer with no transcript to hand back — produces an outcome with evaluations and thin output. Both are valid.

Assertions run against the outcome

Our assertions are evaluated against a stored outcome and produce one result each. This is a separate act from producing the outcome, and separating them buys three things:

• A vendor's judgment is an input, not the answer. An assertion may be satisfied by a runner's evaluation, or judged independently of it, or contradict it — and all three are recordable.
• Re-judging costs nothing. A better rubric is new assertion results over a stored outcome, not another call.
• The same outcome answers new questions. An assertion written next month is evaluated against outcomes produced today.

Every result records its evaluator, because "our judge said this" and "the vendor's guardrail said this" are different measurements even when the sentence is identical.

From an outcome to a verdict

Pass criteria are applied to stored results, never decided by the runner. That is the shape of the whole thing: the runner says what happened, the assertions say what was true, and a consumer's criteria says what that adds up to — for them.

Three things this arrangement buys, each a decision rather than a consequence:

One execution serves every consumer. The expensive half — placing the call, collecting the transcript, judging each clause — happens once. Blueprint CI and the publish gate then read different verdicts out of the same result rows, under their own criteria. Nothing is re-run to satisfy a second opinion, which is the whole reason a runner must not decide whether a test passed: it does not know, and cannot know, who is asking.

An assertion can be advisory, per consumer. A key in a criteria's ignore list is still evaluated and recorded; it simply does not affect that consumer's verdict. So a new assertion can report against a live suite while only the nightly sweep counts it, and a badly-worded rubric does not fail forty teams on its first day. Advisory is a consumer's position, not a property of the assertion.

The criteria version is stamped on the run. criteria_version is recorded at start, because a verdict read under a bar that has since moved is uninterpretable — you cannot tell whether the run failed or the standard rose. And because results are rows, a new criteria can still be applied to historical results to ask "what would have passed under this": changing the definition of passing costs a query, not a test suite.

The runner sits outside all of this. It never sees a criteria, never learns which consumers exist, and cannot make a case pass by asserting that it did.

What a runner claims

A runner claims a run's lease before doing anything. The claim is what makes a run single-writer: a second runner cannot report into a run it does not hold, and a runner that dies stops renewing, so the lease expires and the run resolves timed_out rather than hanging forever.

deadline_at is separate from lease_expires_at on purpose. The lease answers "is anyone still working on this"; the deadline answers "is this worth waiting for at all". A voice run a vendor never returns needs the second one.

What a runner reports

The outcome, and then nothing. It reports its output and any evaluations it performed, renews its lease while working, and releases at the end. It does not decide whether the test passed.

A runner declares which mechanisms it can evaluate. An assertion whose mechanism no runner in the run can evaluate fails with a reason — it never skips, and it never quietly degrades to a model judgment. A skipped assertion reads as a pass in every aggregate; a silent downgrade reports a rubric verdict as though it were a config check.

What a runner must not do

It must not write assertion results directly — its evaluations belong to the outcome, and the mapping onto our assertions is ours to make. It must not write into a finalised run; a late vendor callback arriving after a timed_out resolution is dropped, loudly. It must not put the only copy of a judgment in provider_data; that column enriches an outcome, it never replaces one.

What crosses the boundary outbound

assertions.role tells the outbound translation which of our assertions belongs in a vendor's designated slot — Hamming's guardrail field takes one, and the rest still have to be judged somewhere. role is a translation hint, and it sits on the assertion rather than on a criteria for a structural reason: the run happens once and serves every consumer, so there is no single consumer whose opinion could pick the vendor's slot. It is not our pass policy, which lives entirely in the criteria a verdict is read under.

Runners

Runner	Executes	Evaluates	Adapter's hard part
Hamming	remote, voice	remote — its own guardrails	asynchronous return; may never come back; mapping their verdict scale onto met + value
Platform chat (VAPI / ElevenLabs)	remote, text	local — our judge	two vendors, one definition; provisioning and tearing down scratch objects per run
VAPI-native scorers	remote	remote — its own scorers	their scorer set is fixed; only some of our assertions have a home in it
Ours	nothing — a structural check	local	there is no transcript and no fixture, so most of the contract is unused

The local runner is the useful stress test. It executes nothing, has no transcript, and still has to satisfy the contract — if it cannot, the contract has assumed a conversation exists, which would exclude every structural check.

What it unlocks

• One scenario, every runner. The same situation runs as text on a platform chat for pennies, and as voice through Hamming when it matters, without being authored twice.
• Comparable results across vendors — or at least honestly incomparable ones, since evaluator makes the mixing visible instead of silent.
• A new vendor is an adapter. Two translations and a mechanism declaration, with no change to the model, the authoring surface, or existing runners.
• Re-judging without re-running. Evidence and judgment are separate writes, so a better rubric is a new outcome over stored evidence rather than another call.

How we get there

1. Write the contract and a runner that executes nothing. The config_check reference runner is deliberately the least capable thing that can satisfy it.
2. A second, trivial runner with fixed verdicts. If it cannot satisfy the contract without touching the first, the contract is wrong, and that is the finding.
3. The platform chat adapter, ported from apps/web/lib/simv2-integration/ — the closest thing to a real one that exists.
4. The Hamming adapter, which is where the asynchronous half of the contract gets tested for real.

Steps 1–4 are Phases 2, 4 and 5 of assistant-test-impl.

The first extension

A structural check has no conversation. It runs against a compiled config, returns in under a second, and is exactly the kind of assertion the behavioural runners cannot evaluate. If the contract holds, it is a runner that declares one mechanism and reports through the same path as Hamming.

If accommodating it requires special-casing the model, the seam is in the wrong place.

Unknowns and unsures

• Whether one definition genuinely spans voice and text. The prototype's IntegrationTestDefinition was built for text. A voice run may need timing, interruption, or audio properties that text has no opinion about — and if it does, "author once" gets an asterisk.
• How a vendor's verdict scale maps onto met + value. Recorded as an open decision in the Hamming phase. A boolean loses information a score has; a score invents information a boolean does not.
• What happens when a vendor changes its own shape. The adapter absorbs it, but the stored provider_data from before the change does not migrate.
• Whether role is enough to decide which assertion goes in a vendor's one designated slot when several are eligible.