Avoca — Internal

Appearance

Chat-Based Unit Testing Design ​

Date: 2026-08-18 Status: Draft

Goal ​

Answer "did my change break this?" in seconds, against real production behaviour, for the price of one model call.

Solution ​

Take one turn of a real call — the exact compiled prompt, the exact prior context, the exact tool results the model had — and replay it against a candidate config. One exchange, judged by a rubric. No phone, no audio, no full conversation, no waiting.

A voice suite answers "does the whole journey work" in half an hour. This answers "does this specific moment still behave" in about a second, which makes it the only tier that can sit inside an edit loop rather than after one.

The problem ​

There is no fast feedback on a blueprint edit. Every existing check is a conversation: a voice run takes minutes, a chat run takes tens of seconds, and both exercise a whole journey to tell you about one clause. So the loop is edit, deploy, dial, listen — and nobody runs it more than a couple of times.

Regression is invisible. A prompt edit intended to fix one behaviour changes another, and nothing catches it, because nothing knows what the agent used to do in that situation. The only record of prior behaviour is a call recording nobody replays.

The expensive part of a test is almost never the part under test. Reproducing one turn does not require dialling, transcribing, or holding a conversation. It requires the model's input at that moment — and that is a document.

Why now ​

A quick spike showed the mechanism works — see Single-turn tests are possible. The scale was deliberately small: three real calls across both platforms, eleven turns between them, replayed at temperature 0.

On that sample, every turn reproduced the real behaviour, and five of the eleven were character-exact. Repeated replays produced identical output, so the mechanism is at least deterministic.

Eleven turns is an encouraging signal, not a measured reproduction rate — it is enough to say the approach is worth building on, and not enough to quote as a fidelity figure. What it does establish is that the non-exact cases were not noise: each traced to ElevenLabs injecting an estimated six to ten thousand tokens of hidden runtime content — audio emotion-tag steering, silence-event framing — that no public API exposes. That figure came from comparing llm_usage token counts, so it is inferred rather than observed directly.

The missing prerequisite has also landed. Replaying a turn requires the prompt that call actually ran, and under many scenarios that was not retained. VAPI hands back the wire-format request it sent, so a single-agent call was already close to replayable from capture alone — but its artifact keeps only the last active sub-agent's prompt, measured at one of eleven on a real squad whose members did not even share a model. ElevenLabs returns no prompt at all.

So on a multi-agent call most turns had nothing faithful to replay against, and recompiling from the current config is not a substitute: once the knowledge base has moved, it silently reproduces something the call never ran. #16352 closed that gap — every sub-agent's compiled prompt is now content-addressed in S3 and retrievable from a call id.

Decisions ​

A fixture is immutable and committed. Everything the model saw at one moment — prompt, transcript, recorded tool results, tool schemas, model parameters — is pinned in a file. Resolving inputs live at run time was rejected for a specific reason: a baseline that can move cannot distinguish "my edit fixed it" from "the inputs changed." The whole value is a fixed point to measure against.

Replay stops after one completion. The model may ask for a tool; the request is recorded and never executed. A unit run therefore cannot book a job, write to a CRM, or dial a transfer — which is what makes it safe to run continuously, against production-derived fixtures, on a laptop.

Only turns with a real completion behind them can be pinned. A static greeting has no model call and nothing to reproduce. On the one call where this was counted, 23 transcript rows yielded 10 usable completions — a rough sense of the ratio, from a single example.

Assertions are LLM-judged rubric checklists. Near-binary clauses, each phrased to own its own variations, because a model rewording itself must not register as a regression — and because a failure that names its clause is actionable where a similarity score is not.

This tier does not traverse assistant-request. A unit run replays against a prompt assembled for replay. It is deliberately not the place to assert anything about what production sends — that belongs to the publish gate, which compiles the candidate, or to chat-based sim testing, which takes the real path.

Why this is the one that changes the loop ​

Cost is the whole argument, and it is a difference in kind rather than degree:

Voice suiteChat runUnit replay
wall clockminutestens of seconds~1 second
costdollarscentsfractions of a cent
side effectsrealreal dispatchnone — replay stops before tools
answersdoes the journey workdoes the path workdid this moment change
fits insidea releasea pipelinean edit

Anything that runs in a second against a hundred pinned turns is a different tool from anything that runs in a minute. It stops being a gate you pass and becomes a thing you consult — the difference between a test suite and a compiler.

Most of what a voice test costs is not the question. Provisioning, dialling, speech synthesis, holding a conversation in real time, transcribing it back — call it ninety percent ceremony, none of which is the assertion. That overhead is worth paying when the thing under test is the call. It is pure waste when the thing under test is whether one instruction still produces one behaviour.

Which is why the scale argument is the real one: only a test this light can run often enough, and widely enough, to actually insulate every team from regressions. A check that is too expensive to run against the fleet does not protect the fleet — it protects whoever remembered to run it.

And because every fixture is derived from a real call, the corpus is free: any production call that exposed a behaviour worth keeping becomes a regression test by being pinned.

The loop this unlocks ​

The shape ​

A group of teams shares a base config. Each team runs a production config that inherits from it, plus whatever branch configs they are working on. An override is a team's local divergence from what they inherit.

Everything below follows from one fact about that picture: a defect in the base is a defect everywhere, but it surfaces on one team's call. Today it gets fixed for that team and stays broken for the rest, because nobody can cheaply ask who else has it.

The loop ​

Three actors, and the whole point is that the test changes hands.

There are two different greens, and confusing them is the current failure mode. Team A goes green at step 4 because they patched themselves; the fleet goes green at step 9 because the defect is gone. Today only the first one happens, and it looks enough like success that the second never gets scheduled.

Promoting the test is what makes it everyone's problem. Attached to the base, it runs against every inheritor in one pass, and that pass is the blast radius — a list of exactly which teams have this defect, produced before anyone has argued about whether it matters. Two teams affected and forty teams affected are different decisions; this is the step that tells them apart.

The fix goes in the base, so its size does not scale with the damage. Every inheritor picks it up by inheriting. The test is what proves it reached them rather than assuming it did.

Any team still failing after step 9 is failing for its own reason — a local override, or a knowledge base that has drifted. That is a notification, not another base-config change.

Only one tier can do this. Steps 6 and 10 run a test against every production config in the group, twice. A voice suite across forty teams is not a slow version of that — it is not something anyone would attempt. A conversation run per team is arguable but expensive enough that nobody re-runs it casually. A one-second replay makes the measurement unremarkable, which is what lets it be routine rather than an incident response.

What this does not show ​

Whether the call ends well. A single turn says the model behaved correctly at one moment, given exactly the context it had. It cannot say whether the conversation reaches a booking ten turns later.

That gap is not a rounding error. A conversation compounds: the model's own output becomes the next turn's context, so a change that looks harmless at turn three can send the trajectory somewhere else by turn ten, and no amount of single-turn confidence would have caught it. Replay holds the prior context fixed precisely so the measurement is clean — which is the same reason it cannot tell you what a different prior context would have produced.

So the tiers are complementary, not ranked. Single-turn replay answers "did this moment change," at a price that permits asking constantly. Chat-based sim testing answers "does the path still work end to end," and voice answers "does the journey work for a person." A green unit suite is evidence that nothing regressed at the moments you pinned. It is not evidence that the call succeeds.

How we get there ​

  1. The engine — a plain async function over a fixture, its assertions, and a set of configs. Every runner is a thin caller, so the same engine serves a script, a job, and a route.
  2. Fixtures from real calls, built off the stored compiled-prompt artifact rather than recompiled live.
  3. A corpus that grows from incidents — a reported problem becomes a pinned turn, so the same defect cannot return unnoticed.
  4. Wire it as a consumer — a suite of pinned turns plus a criteria, so a config change is checked before it ships.

Unknowns and unsures ​

  • Exact-text replay on ElevenLabs looks capped, by an estimated six to ten thousand tokens of runtime content no API exposes. Behaviour matched on all eleven turns tried, but a rubric that depends on precise wording will be measuring something the platform may not let us reproduce. Worth re-testing at a larger sample before anyone relies on it.
  • Rubric stability is the harness's own failure mode. A near-binary clause is meant to survive rewording; whether a given rubric does is a property of how it was written, and a flaky assertion is worse than none.
  • How large a corpus stays useful. A hundred pinned turns is a fast, cheap regression suite. Whether it stays meaningful at a thousand, or becomes noise nobody reads, is unmeasured.
  • Fixtures pin a moment, and the world moves. A turn pinned against a knowledge base that has since changed still replays faithfully, but may be asserting behaviour nobody wants any more. Nothing yet says when a fixture should be retired.