Research Spikes

Empirical findings from Avoca FDE engagement work — things we proved, not things we're pitching. Distinct from Ideas (speculative, cross-engagement pitches) and musings (raw working drafts).

Two broad conclusions

1. We can and should use VAPI and ElevenLabs as test runners for chat-based regression testing.

2. We can and should build two classes of tests, analogous to unit and integration tests.

• Integration tests: chat-based tests run through VAPI and ElevenLabs, exercising the real agent — real routing, real tool dispatch — end to end.
• Unit tests: recreate the exact prompt sent to the model (GPT-4.1, GPT-5.2, whatever it is at the time) for a single turn — not the whole conversation — and run it directly against the model.

With minimal, value-accretive updates to our call-data collection (capturing the compiled prompt/blueprint, among other things we already receive but discard), we can accurately reconstruct a single turn of a conversation — the unit — and run it directly through the model to reproduce an issue. Swap in the blueprint under review instead of the original, and the same mechanism becomes a test.

Two PRs, one per conclusion

Both make the fundamental changes each capability needs — the substrate, not the finished testing product.

• #16217 — enables conclusion 1 (integration tests). VAPI has no way to resolve a team_id when a tool call comes from a test squad rather than a real call; adds the missing last-resort resolution path.
• #16215 — enables conclusion 2 (unit tests). Updates post-call data collection to capture info both platforms already send but we discard, plus the one genuinely new piece: we don't currently save the compiled prompt actually passed to VAPI or ElevenLabs. Recompiling an old call after a KB edit silently drifts — it does not reproduce what that call actually ran. This PR closes that gap with a content-addressed store, keyed at the moment our own code produces the compile output.

Findings, in depth

• VAPI + ElevenLabs chat as a real-dispatch test surface — proof for conclusion 1.
• Single-turn tests are possible — proof for conclusion 2, including the 11/11-turn replay demonstration built before #16215 was opened.
• Just-in-time service context — separate line of work, and inconclusive: injecting per-service knowledge by tool call instead of compiling it into every prompt produced no measured improvement in call time or accuracy. Prompt size does stop scaling with the catalog, but the quality gains came from better content, not delivery. Points at mid-call goal evaluation as the more promising direction. Build details in the implementation notes.

How entries get here

An investigation runs to ground truth — a real production or tunnel-routed call, cross-checked against ClickHouse or logs — not vendor docs or marketing claims. What's proven lands here; what's still open is named explicitly rather than implied.