Deterministic Test Verification: The Transfer Problem and the Oversight Proposal

A briefing on why verifying a voice-agent transfer is hard for Hamming, what we built to make it testable, and the proposed direction: give every test case a second, deterministic verdict from Avoca's existing Oversight engine.

Thesis

Hamming operates on the transcript, and it judges non-deterministically: it is an LLM reading the call and returning a probabilistic pass/fail. That domain is broad. It covers not just "did it sound right" but any question you can ask of the transcript, including factual ones like "did the agent repeat the caller's phone number." But two things fall outside what it can do well:

1. Outcomes that are not in the transcript at all. Did it transfer to the correct destination, did it create the appointment, did it write the right CRM update. These live in Avoca's data, not the conversation. Hamming cannot see them.
2. Transcript facts that are actually deterministic. "Did the agent say the required disclosure," "did it repeat the phone number," "did it use a forbidden phrase." An LLM can answer these from the transcript, but it does so probabilistically and at the cost of inference, when a plain match settles them exactly. Using an LLM for a true-or-false that a string check answers is both flaky and wasteful.

So the distinction is not "sound versus outcome." It is non-deterministic (LLM over the transcript) versus deterministic (exact checks over the transcript and the data). LLM judgment still has a legitimate place for genuinely fuzzy questions (tone, coherence, whether an explanation was clear). The point is to match the tool to the question, and to stop paying LLM inference for facts that are deterministic.

The proposal: run every test-case call through Oversight (Avoca's existing deterministic-eval engine) to produce a second, deterministic verdict alongside Hamming's. Hamming judges what genuinely needs a judgment; Oversight settles what is a matter of fact.

How we got here

We set out to test transfer behavior on Simulation v2: a voice agent should transfer a caller to the right place under the right conditions, and should not transfer under the wrong ones. Testing transfers turned out to expose a structural gap.

On a test call (a Hamming test caller, or a team test phone), transfers were refused. The tool-call mock layer returned "Transfer is not available during test calls," so the agent took a message instead of transferring. Every "the agent should transfer here" assertion was therefore unpassable: the behavior under test could never happen. Separately, there was a historical leak (real CSR lines rung by test runs) rooted in the same detection layer failing open. See the Simulation v2 overview for the fuller mechanism map.

What we built: the transfer mock

To make a test transfer both safe and observable, the transfer handlers now do the following on a test call:

1. Resolve the real destination. The same server-side routing logic a production call uses (getTransferDestination reads the conversation and picks the destination). This is the thing under test, so it must run for real.
2. Record it. The chosen destination is written to transfer_destination_logs (its id, number, and reason), exactly as a production transfer would.
3. Narrate it, do not dial. Instead of placing the PSTN call, the handler returns a response naming the destination. No real number is dialed, so no human is rung.

The intercept sits after the destination is resolved and logged, before the dial fork, so a single branch covers every transfer type. This is implemented for both ElevenLabs and Vapi.

What the live test showed

Running it end to end confirmed the mechanism works: the call resolved to the correct destination (French) and recorded it in transfer_destination_logs, and nothing rang. But it also surfaced the real problem:

• On ElevenLabs, the agent did not reliably speak the destination. EL has no guaranteed-speech mechanism, so the platform cannot be forced to voice "you have reached the French line." When the mock returned a transfer-success status, the agent behaved as if a real handoff were in progress ("connecting you, please hold"), waited for a bridge that never came, and stalled.
• The routing was correct in the log the entire time. The truth was recorded; the transcript just did not reflect it.

What the demo does, and its relationship to the post-call workflow

A useful way to see this: we are bringing a post-call analysis step into the test flow. Two different "post-call" things must not be conflated:

• The post-call workflow (common-webhook / end-of-call orchestration) takes actions: booking, CRM writes, classification. On a test call this is mocked or skipped, so those actions do not really happen.
• Oversight / analyze-call is a post-call analysis layer that reads the call and computes evaluations. It takes no actions. This is what the demo pulls into the test flow.

That distinction bounds what a deterministic check can read on a test call. Transfer routing is the clean case because the routing decision is recorded in-call (the transfer handler writes transfer_destination_logs at transfer time), so it exists no matter what the post-call workflow does. A check like "did it book," by contrast, depends on an action the workflow would take, which is mocked on a test call, so there is no booking row to read; the honest deterministic check there is the intent (did it call the booking tool with the right arguments, which is recorded even when mocked), not the completed outcome.

The way it worked before

What changes in the demo

The change is the second half: a post-call analysis pass (analyze-call) runs the deterministic transfer_destination tag, which reads the in-call routing record and writes the exact destination. Hamming still judges the transcript; the deterministic check judges the routing.

Why Hamming cannot verify routing correctness

This is the core finding, and it generalizes far beyond transfers.

1. Hamming judges the transcript, probabilistically. Its assertions are natural-language pass/fail criteria scored by an LLM over the call. That is the right tool for a genuinely fuzzy question (was the explanation clear, did it handle the objection well). It is the wrong tool for a fact with a definite answer, and the destination is a fact.
2. On ElevenLabs we cannot control what the agent says. No guaranteed speech means the destination name and number may or may not appear in the transcript. So even if Hamming wanted to check the destination, the information may not be there to read.
3. Hamming cannot see Avoca's data. Its assertions have no database, webhook, or SQL access. The one place the routing is reliably recorded, transfer_destination_logs, is invisible to it.

The result: Hamming can, at best, coarsely tell that a transfer was attempted. It cannot tell that the transfer went to the correct destination. And the same limitation applies to any outcome that lives in the system rather than the conversation.

The failure this misses

The most dangerous case makes the gap concrete. An agent can say "you are all booked for Tuesday at 2" while the CRM write silently failed and no appointment exists. Hamming hears a confident, correct-sounding confirmation and passes the call. The customer never gets a technician. Only a check against the actual booking record catches it. A transcript judge, by construction, cannot.

The insight: match the judge to the question

A voice-agent test asks two kinds of question, split by whether the answer is a matter of judgment or a matter of fact, not by transcript versus data:

Kind of question	Example	Right judge	Nature
Genuinely fuzzy	Was the explanation clear, was the tone appropriate, did it handle the objection well	Hamming (LLM over transcript)	Probabilistic, subjective
A matter of fact	Transferred to destination X, created the appointment, said the disclosure verbatim, repeated the phone number, used a forbidden phrase	Deterministic check	True/false, exact

Note the deterministic column spans both transcript facts (did it say the disclosure, did it repeat the number: a string or pattern check) and data facts (did it book, where did it transfer: a record check). Hamming can answer the transcript-fact questions, but only probabilistically and at inference cost, which is the wrong trade for something exact. It cannot answer the data-fact questions at all.

Simulation v2 today only has the LLM judge. The proposal adds the deterministic one, which absorbs the transcript facts that were being judged probabilistically and reaches the data facts that were out of reach entirely.

Proposed solution: a second type of assertion

A test case is defined by its assertions. Today every assertion is a Hamming assertion: a natural-language criterion an LLM scores over the transcript. The proposal is to introduce a second type of assertion, the Oversight (deterministic) assertion, and let a test case carry both kinds. Each assertion is evaluated by the engine that fits it; the two do not merge.

• A Hamming assertion is a fuzzy criterion judged by an LLM over the transcript ("did the agent handle the objection well").

• An Oversight assertion is a deterministic check over the call's structured outcome ("the transfer destination equals French," "an appointment was created," "the disclosure phrase is present"). It encodes both the expectation and the check, and it returns exact true/false. Avoca already has the engine for it: Oversight runs deterministic evaluations per call (today on real production calls via insights.analyze-call), and writes structured results to tag_values / eval_results.

  Grounding this in the real implementation (lib/oversight/deterministic-registry.ts): a deterministic template is a run(context) function that is plain code, never an LLM guess. Its context already carries what a transfer check needs: callId, teamId, transcript, toolCalls (name, arguments, result), isTransferred, callDurationMs, and DB access (it can call getCallByIdX). There are two kinds: tags (record a value) and evals (record a pass/fail). Templates are gated per team via availability.teamIds and installed as tags / evals rows with execution_type = 'deterministic'.

  A transfer check is a near-clone of the one live deterministic template, the Neighborly "Franchise" tag: its run reads the value the workflow stamped on the call and returns { value, explanation }. A "Transfer Destination" tag is the same shape reading transfer_destination_logs by callId (the row the mock writes). So the check fits a tag (records which destination), not an eval, which is also the proven path: the tag registry has a working example, the eval-template list is currently empty.

So a test case selects the assertions it needs, of either type. A transfer case attaches an Oversight assertion "destination equals French" (and perhaps a Hamming assertion about the greeting); a booking case attaches "appointment created." Not every case runs every check; each case declares its own.

Each test case then produces a combined verdict from two kinds of assertion that are evaluated independently:

• Hamming assertions: LLM judgment over the transcript (the fuzzy questions).
• Oversight assertions: deterministic true/false over the structured outcome (the factual questions).

Authoring UX: a query builder, not a prose box

Today an assertion is authored in the v2 Assertions tab (AgentTestingGuardrailsPage → GuardrailDialog) as free text: name, trigger, assertionText, passCriteria, failCriteria, all typed, all handed to Hamming for an LLM to interpret. Even a hard fact ("transferred to French") is written as prose and judged probabilistically.

A deterministic assertion is authored as a query-builder condition instead: [field] [operator] [value], where each part is constrained so you can only build something checkable.

• Field is a deterministic check from the Oversight registry (Transfer Destination, Appointment Created, Call Reason, Call Duration, Disclosure Said). Each tag we add is one selectable field. The registry is the menu, and it is curated by construction: you cannot pick a field that has no real check behind it.
• Operator is constrained by the field's type: text/enum offer =, ≠, is set, is empty; boolean offers is true / is false; number offers =, <, >.
• Value is a typed control, open or constrained depending on the field:
  • constrained from a static list (option + allowedValues) → a fixed dropdown;
  • constrained from a live source (Transfer Destination → the team's transfer_destinations; Call Reason → the team's reasons) → a dropdown fetched per team;
  • open text → type any string;
  • open number → type a value (Call Duration < 5).

So a condition reads Transfer Destination = French, Appointment Created = true, Call Duration < 120. The negative cases are just operators: "did not transfer" is Transfer Destination is empty; "didn't book" is Appointment Created is false. Conditions can compose (Transfer Destination = French AND Call Duration < 60), though a single condition is enough for the first checks.

Configuring the value types

The field list, the operators, and the value control all derive from one per-field type declaration on the registry entry. The Oversight template already carries the start of it: valueType (text | boolean | option) and allowedValues. What each setting drives:

valueType	Input control	Operators
boolean	true/false toggle	is true, is false
option (static allowedValues)	dropdown from the fixed list	=, ≠, is set, is empty
option (live source)	dropdown fetched per team	=, ≠, is set, is empty
text	open text input	=, ≠, contains, is empty
number (net-new)	open numeric input	=, <, >

Net-new to configure: (a) a number value type for open numeric values (today only text/boolean/option exist), and (b) a way for a field to declare "my options are dynamic, fetch them from source X" rather than a hardcoded allowedValues list. Open values and constrained values are then just two settings of the same knob.

The contrast with today is the point: free text can express "the vibe was good," which no system can reliably judge; the query builder can only express facts the system can settle, because every field, operator, and value option is backed by a real deterministic check.

What deterministic evals unlock

The transfer check is the first of a class. Anything with a factual answer in the data or an exact answer in the transcript is a candidate, and each is something Hamming judges poorly or cannot judge at all.

Did the system actually do it? (DB / tool outcome, Hamming cannot see this)

• Did it create the appointment? (a booking row exists; catches silent CRM failure)
• Did it book the right service, date, and window?
• Did it transfer to the correct destination? (the first eval)
• Did the cancel or reschedule actually execute? (job status changed)
• Did it match the existing customer, or create a duplicate?
• Did it classify the call reason correctly? (team_call_reason_id)
• Did the post-call SMS or email actually send?

Did the agent say, or not say, a specific thing? (exact, often regulatory)

• Said the AI disclosure or recording notice verbatim. A "92% probably disclosed" is a compliance liability, not an answer; this is true or false.
• Did not say a forbidden phrase (a wrong price, a competitor, banned language).
• Used the correct company name and did not hallucinate a different one.
• Read back and confirmed the callback number.

Tool-call facts (structural)

• Called the lookup or availability tool before booking (did not fabricate times).
• Did not loop (bounded number of tool calls; would have caught the transfer hang above).
• Passed the right arguments (booked the service the caller asked for, not a default).

Negative cases (clean absence checks)

• Did not transfer when it should not have (no transfer log row).
• Did not book for an out-of-area caller.

Why deterministic is the right tool when the answer is a fact

This is not "deterministic beats LLM" in general. For a genuinely fuzzy question (was the explanation clear, was the tone right) an LLM judge is correct and a deterministic check is impossible. The claim is narrower: when the answer is a fact, whether it lives in the transcript or the data, a deterministic check is the right tool, and reaching for an LLM is the wrong trade.

• Reproducible: same call, same verdict, every time. An LLM judge drifts run to run, so a regression suite built on probabilistic checks for factual things is noisy by construction.
• Ground truth over inference: "did it book" has a factual answer in the data; inferring it from the transcript is strictly worse, and worst on the failure that matters. "Did it say the disclosure" is a match, not an opinion.
• Free and instant versus paid and slow, which matters when a coverage suite runs hundreds of cases and each fact would otherwise be an inference call.
• Auditable: a true/false plus the underlying row or transcript span is defensible to a customer; a confidence score is not.

What exists vs what is net-new

Already exists (do not rebuild):

• The Oversight deterministic-eval engine and its registry (runs on real calls, writes structured results).
• The transfer mock (this work): it produces the transfer_destination_logs record the eval would read, and never dials.

Net-new (the real work):

• The Oversight assertion type on a test case. A test case can today attach Hamming assertions; it needs to also attach deterministic assertions (each naming a check and its expected value, e.g. "destination equals French"). This is the main new data model and the bridge between the two engines. It is also the one real gap in the existing engine: a deterministic tag computes a value ("routed to French") with no notion of "expected"; the expected-vs-actual comparison is the test layer that does not exist yet.
• A transfer-destination deterministic tag in the registry, a near-clone of the live Neighborly franchise tag, reading transfer_destination_logs. Small and low-risk; the registry has a proven tag example already.
• Evaluating on test calls. analyze-call has no is_simulation / test-phone skip in its body, and insights.analyze-call has been observed firing on test calls. So the path is available; the remaining check is confirming the live per-call emitter fires for test traffic (the backfill and rerun emitters clearly do).
• Per-call to per-case correlation (test_case_id on the outcome record), so a deterministic assertion attaches to the right case even when a run reuses one caller number across cases.

Open questions and risks

• Does the live per-call emitter fire analyze-call on simulation / test calls? Evidence says yes (analyze-call has no test-call skip, and the event has been seen firing on test calls), but the live emit path (versus the backfill/rerun emitters) still needs a definitive confirmation.
• Where do "expected outcomes" live on a test case, and who authors them? For the transfer coverage suite these can be derived automatically from the team's reason-to-destination mapping; for other evals they may need manual authoring.
• Scope and ownership: Oversight is an Avoca-owned system. Wiring it into the v2 test flow is a cross-surface change that needs Avoca approval.

How the pieces fit

1. The transfer mock (done): a test transfer resolves, records the destination, and does not dial. It produces the record a deterministic assertion reads.
2. A transfer deterministic eval (small, next): the check an Oversight assertion points at, computing "transferred to destination X" from that record on the Oversight engine that already exists.
3. The Oversight assertion type plus the coverage suite (the larger follow-on): let a test case attach a deterministic "destination equals X" assertion, and generate one such assertion per destination automatically from the team's reason-to-destination mapping.

The mock is the mechanism. Oversight is the engine we do not have to build. The new assertion type is the bridge, and the coverage suite is what turns it into an automated, deterministic transfer-routing test. The same assertion type extends to booking, classification, disclosures, and every other factual check.