Voice Agent Testing Primitive

A productizable primitive that emerged from Avoca FDE work. Cross-engagement vision; not a commit-to-build. The full analytical research lives at .indusk/research/voice-agent-testing-primitive.md; this is the publishable pitch + architecture.

Status

Vision / strategic option. Not in active development. The Avoca-specific implementation (mock AutoOps + replica enterprise + Hamming-driven variant testing) is the validation surface; productization would come after.

The headline

Instant sandbox for any CRM or SaaS integration. Point at an OpenAPI spec (or capture a few real calls). Get a stateful mock in hours, not weeks. Test thousands of interactions without touching prod.

Why this is 5-50x

CRMs traditionally don't have sandboxes. Getting access to a real test tenant takes weeks of back-and-forth with the vendor and the client, costs money, has rate limits, and risks contaminating real customer data. The alternative is hand-rolling a one-off mock per engagement, which is slow and doesn't reuse.

With the primitive: ingest the OpenAPI spec, prune to what the consumer actually uses (via path-tracing on the consumer codebase), fill in realistic response shapes from a few captured calls. Stateful mock running locally in hours.

Today	With the primitive
Weeks of back-and-forth to get a test tenant	Hours: ingest OpenAPI, generate mock
Each engagement reinvents the mock	One-time primitive, infinite reuse
Real tenants cost money, have rate limits, contaminate real data	Local mock, free, infinitely resettable
Test loop = real API calls (slow, real failure cost)	Test loop = local HTTP (fast, no consequences)

Who needs this

The headline framing fits voice agents because that's where we discovered it, but the primitive is bigger:

• Voice agent platforms building integrations with CRMs (Avoca, Hamming customers, Vapi / ElevenLabs customers)
• LLM-based tools that talk to backend systems (dispatchers, copilots, agents)
• CRM migration projects (validate the new system handles every observed-in-prod payload shape)
• Generic integration testers (Zapier-style products wrapping N CRMs)
• Engineers building against vendor APIs that don't ship a sandbox tier (most niche SaaS APIs)

Voice agents are the first vertical that needs this acutely. The primitive surface is bigger.

Architecture

Originally scoped to six bespoke components. Iteration on 2026-05-20 collapsed substantial pieces into off-the-shelf tools:

• Mock substrate: WireMock standalone (Docker container). OpenAPI ingest + Scenarios for state + REST admin API are all built in. Don't reinvent.
• Orchestration: Hamming Test Hooks. Lifecycle webhooks (Before Run / Before Case / After Case / After Run) with variable interpolation and response chaining are native. Don't reinvent.
• Grading: Hamming guardrails with "Wait for webhook validation data" toggle. The orchestration loop drives both state setup and post-call validation. Don't reinvent.

What remains custom:

Component	What it does	Custom build?
Spec-driven mapping generator	Ingest OpenAPI + path-tracing on consumer code + traffic capture → produce WireMock mappings + scenarios	Yes — this is the differentiated piece
Replica tenant provisioning	Spin up N test tenants in the host platform (e.g., Avoca's enterprise + N teams + mocked CRM credentials)	Yes, light scripting
Scenario authoring patterns	Express "what the user wants, what the agent should do, what post-call state should look like" in a shape the testing infra consumes	Convention + helper utilities; mostly schema design
Spec-driven mock framework	Generate own mock server	No — use WireMock
Mock orchestration	Build lifecycle hooks	No — use Hamming Test Hooks
Variant comparison harness	Build statistical aggregation	Use Hamming's batch + sampling primitives + light post-analysis scripting
Post-test analysis layer	Build dashboard	Use Hamming's run results UI + ad-hoc SQL against mock state when needed

The differentiator narrows to: smart spec-driven generation of WireMock mappings, scoped to actual consumer usage, with realistic response shapes from captured traffic. Everything else is config of existing tools.

Mock generation: three complementary sources

The differentiated insight (relative to existing tools like Stoplight Prism, openapi-mock-express-middleware) is combining three signals rather than relying on any one:

OpenAPI spec ───┐
                ├──→ [shape definitions]
Path-tracing ───┘                          ├──→ [generated mock]
                                           │
Traffic-capture ──→ [response examples] ───┘

Source	What it provides	When it shines
OpenAPI spec	Full surface contract	When the vendor publishes one. Fastest path.
Path-tracing	Which endpoints the consumer actually calls, with what shape	Always. Prunes the OpenAPI surface to what's used. Reduces mock scope by 80-95%.
Traffic-capture	Realistic response payloads, edge cases the spec doesn't describe	When semantic accuracy matters more than coverage.

Path-tracing is the most differentiated piece. Anyone can mock from OpenAPI; few think to scope it to actual usage by reading the consumer's code.

What's hard

Hard part	Why
Realistic state semantics	Multi-turn interactions must match real backend behavior (book → list → cancel works coherently). Stateless mocks are easy; stateful mocks that match real backend behavior are hard.
PII in captured traffic	Real traffic has customer phone numbers, addresses, payment info. Generation has to scrub or synthesize. Privacy review is non-trivial.
Cross-tenant isolation	Parallel tests against one mock must not bleed. Either separate mock instances per test (expensive) or scenario-tagged data with strict scoping.
Post-call expectations	Existing testing tools (Hamming, etc.) assert on transcript / tool-call sequence. Asserting on post-call backend state is the missing feature. A custom post-run assertions library is real work.
Variant-comparison statistics	Beyond "pass rate per variant," you want significance testing, failure-mode clustering, longitudinal regression tracking.

DSPY and automated optimization: v2, not headline

DSPY (and similar: TextGrad, Adalflow) automate prompt optimization given (a) a program description, (b) labeled examples, (c) a metric. They consume the evaluation surface; they don't provide it. Without mocks + scenarios + metrics, DSPY has nothing to optimize against.

Worth adding as a v2 feature once the mock primitive has traction. Not the headline.

"Productize" framing, narrowed

Refined 2026-05-20: "productize" in this context does NOT mean go build a SaaS. It means make the pattern reusable across Avoca teams who integrate with different CRMs. Each Avoca team (ServiceTitan, HCP, Acculynx, Jobber, AutoOps) has the same shape of testing problem; one well-built primitive serves them all.

The relevant audience is internal Avoca + the FDE-engaged Lazer team, not external customers.

Build-then-promote sequence (toolkit-first)

Following the FDE workbench philosophy (InDusk + proven primitives live in the toolkit, never in client repos until earned):

1. Stand it up in the toolkit at apps/api-mocker/ with WireMock + AutoOps spec. Runs in Docker, lives entirely outside avoca-next.
2. avoca-next side change is minimal: env var (AUTOOPS_BASE_URL) + DB row insert (the autoops_team_configs row). No PR. No client-repo coupling.
3. Run cross-shop tests against it via Hamming. Validates the design end-to-end.
4. Generalize once: add ServiceTitan or HCP spec. Proves multi-CRM reuse on second customer's surface.
5. Once proven, propose promotion to avoca-next. git mv apps/api-mocker/ from the toolkit into the avoca-next monorepo with the same structure. Pitch is concrete + already-validated rather than speculative.

Net: by the time we propose moving it into avoca-next, we have a working demo, a second CRM proven, and zero risk to other in-flight work.

Build-then-productize sequence (proposed)

1. Weeks 1-2: Build the Avoca-specific version (mock AutoOps + replica + first variant comparison) → validates the design with real failures.
2. Weeks 3-4: Generalize the mock-framework (extract spec-driven generator, path-tracing utility) → turn Avoca's version into reusable primitives.
3. Months 2-3: Second customer / second CRM → proves generalizability.
4. Months 3-6: Polish + GTM → what becomes the productized form.
5. Months 6+: Layer DSPY-style optimization on top once the primitive has traction.

Origin

Discovered during Avoca FDE work, 2026-05-20. After hitting the "no AutoOps admin UI" wall, the natural next thought ("what if we just don't wire to real AutoOps at all?") generalized into "what if we never wire to any real CRM during testing?" The 5x filter applied honestly surfaced the CRM-sandbox primitive as the strongest pitch; everything else (DSPY, automated regression, eval-at-scale) became secondary once the primitive existed.

Related

• Full analytical research: .indusk/research/voice-agent-testing-primitive.md
• Avoca-specific implementation: Mock AutoOps API research, EAS Replica how-to, Enterprise Test Fanout research
• Insights captured along the way: Two voice testing surfaces, Blueprint override drift, FDE replica before iteration