Avoca — Internal

Appearance

Chat-Based Integration Testing Design ​

Date: 2026-08-18 Status: Draft

Goal ​

Start a conversation from a real call's midpoint and let it run forward, to answer the question a single turn cannot: will this call book?

Solution ​

Take a real call's history up to some turn — the messages, the tool calls, the results the agent actually received — seed it as the conversation's starting state, and let a simulated caller continue from there against a candidate config.

This is not a replay. A replay reproduces one turn and stops. This resumes: everything before the seam really happened; everything after it is generated now, against the config under test.

Built on ElevenLabs, whose system__conversation_history dynamic variable carries prior turns as structured entries — agent messages, user messages, tool_requests and tool_results. Populate it and the agent behaves as though that history is its own. The seam already exists in elevenlabs-lane.ts; it currently starts every run empty.

The question it answers ​

Unit replay tells you the model said the right thing at turn seven. It cannot tell you whether turn seven leads to a booking, because it stops there.

A full simulated call answers the outcome question, but from turn zero — so it spends most of its length re-establishing a situation you already had, and it tests the opening far more often than the part you changed.

Resuming gets both. The expensive, specific, hard-to-fabricate context is real, and only the part under test is generated. A call that took seventeen turns to reach an availability offer can be resumed at turn seventeen a hundred times, with a hundred different caller reactions, and every one of them starts from the same genuine ground.

So the assertions are outcome-shaped rather than phrasing-shaped:

  • did the call reach a booking
  • did it hold the appointment time that was offered
  • did it escalate instead of inventing an answer
  • did it get there without asking for something it already had

How it is used ​

Fork a real call at the moment before the thing you care about. A customer about to be offered slots, a caller who has just said something ambiguous, a transfer about to be decided.

Then vary one thing:

VaryAnswers
the configdoes my blueprint change still book this call
the caller's next movedoes it hold up when they push back, change their mind, go quiet
the tool resultswhat happens when availability comes back empty

The first is regression. The second is coverage a scripted test rarely reaches, because it explores from a real state rather than an imagined one. The third is how you test a branch that almost never occurs naturally.

What would be needed to run these ourselves ​

Today this depends on ElevenLabs' simulation to drive the caller and hold the loop. Owning it would mean building four things, and we already have two:

PieceState
The compiled prompt for a configHave it — content-addressed per sub-agent, retrievable from a call id (#16352)
Real tool dispatch from a non-telephony runHave it — the test-harness header path (#16217)
A simulated callerNeed it — a model given a persona and a goal, replying in character, deciding when the call is over
The turn loopNeed it — call the model, execute any tool it requests, append the result, repeat until the goal is met, the caller gives up, or a turn cap is hit

The loop is the smaller job. The caller is the harder one, and it is where a home-grown harness earns or loses its keep: a simulated caller that is too agreeable books every call, and one that is too difficult books none. Neither result measures the agent.

Two things would need care beyond that. Sub-agent transfers are handled by the platform today, so the loop would have to decide when a handoff occurs and which compiled prompt takes over. And ElevenLabs injects several thousand tokens of runtime content — emotion-tag steering, silence framing — that no API exposes, so a self-driven loop is a slightly different agent from the one production runs. That difference is acceptable for outcome questions and is not acceptable for exact-wording ones.

The honest reason to build it anyway: a harness we own runs anywhere, costs a model call, and does not depend on a vendor keeping an undocumented variable working. The reason not to, yet: the vendor's version exists now and this tier is not the one that needs to run ten thousand times a day.

Constraints ​

ElevenLabs only. VAPI exposes no equivalent seeding surface, and Hamming's voice API does not take prior turns — its separate text/chat API does. A config running on VAPI cannot be resumed this way today.

Read tools run live. Resuming re-enters real dispatch, so a resumed call queries the real CRM. The seeded history is fixed; the world it reads is not.