Avoca — Internal

Appearance

Single-turn tests are possible ​

TLDR: One real call turn — exact system prompt, exact prior context, exact tool results — can be replayed in isolation against the model directly, bypassing VAPI/EL entirely, and reproduces real agent behavior with high fidelity. This is a working unit-test mechanism for voice-agent logic today, not a future capability: 11/11 replayed turns matched real behavior, 5/11 character-exact.

What's proven ​

Two platforms, three real calls, replayed at temperature 0:

PlatformTurns replayedResult
VAPI21 exact match, 1 payout-figure-exact (computed value reproduced, phrasing differs)
ElevenLabs94 exact/near-exact, 5 same-behavior-different-wording

11/11 same behavior. 5/11 character-exact. Every non-exact case is on ElevenLabs and traces to one quantified cause (below) — not noise. Determinism confirmed: identical replay output across repeated runs.

Why it works ​

  • VAPI hands back artifact.messagesOpenAIFormatted — the literal OpenAI-wire-format request it sent, compiled prompt included. Replay is close to solved by capture alone.
  • ElevenLabs injects ~6–10k tokens of hidden runtime content (audio-emotion-tag steering, silence-event framing) no public API exposes — quantified via llm_usage token counts. This is the ceiling on exact-text EL replay; behavior still matched on every tested turn.

Foundation built ​

  • PR #16215 — additive capture of both platforms' per-turn LLM trace at the existing post-call conversion boundary (from-vapi.ts/from-elevenlabs.ts), plus a content-addressed compiled-prompt-per-sub-agent store — the one genuinely new piece of stored information. We don't save the compiled prompt actually sent to VAPI/EL today; recompiling an old call after a KB edit silently drifts and does not reproduce what that call actually ran. This PR closes that gap. No migration, no hot-path change beyond a fire-and-forget write; 30 unit tests green, full-repo sweep clean.
  • The actual unit-test runner: replay-turn-vapi.ts / replay-turn-elevenlabs.ts — given a call id and turn number, reconstruct the exact input and replay it live.

Still need ​

  • No pass/fail harness yet — today this is a script + manual eyeball diff, not an assertable test suite.
  • EL's hidden-context gap means exact-text assertions won't work for EL; behavior-equivalence checks (tool called, decision made) are the reliable bar there.
  • The config-swap test itself isn't built yet — the substrate (PR #16215) makes it possible to replay turn n with the candidate config's compiled prompt in place of the original and diff the output, but that comparison script doesn't exist yet.
  • No CI wiring; no backfill of historical calls.