Avoca — Internal

Appearance

Post-call extraction nondeterminism — same transcript, divergent scheduledAt ​

What happened ​

Running a multi-shop fanout test (chore/service-matcher-fanout-test branch) that replays one captured transcript against six AutoOps tenants, six identical OpenAI extraction calls produced one off-by-154-days outcome. Same input, same prompt, same model — different output.

The fanout's pipeline per shop:

  1. Pick a real slot via client.getAvailability.
  2. Render transcriptTemplate with {{SLOT_FULL}} etc. for that slot (all six shops landed on the same first-available slot: Thursday, November 12th at 7:00 AM Mountain Time).
  3. Run extractCallData on the rendered transcript.
  4. convertScheduledAtToUtcIso(extracted.scheduledAt, teamTimezone).
  5. Assert the converted UTC matches the slot's UTC.

All six runs used identical rendered transcripts. Five extracted the correct date. One didn't.

The evidence ​

ShopPicked Slot (UTC)Extracted (naive)Converted UTCRound-trip
Chatfield2026-11-12T14:00:00.000Z2026-11-12T07:00:002026-11-12T14:00:00.000Z✅ exact
Crestline2026-11-12T14:00:00.000Z2026-11-12T07:00:002026-11-12T14:00:00.000Z✅ exact
South Park2026-11-12T14:30:00.000Z2026-11-12T07:30:002026-11-12T14:30:00.000Z✅ exact
Quebec2026-11-12T14:00:00.000Z2026-06-11T07:00:002026-06-11T13:00:00.000Z❌ 154 d
Platt2026-11-12T14:00:00.000Z2026-11-12T07:00:002026-11-12T14:00:00.000Z✅ exact
Ponderosa2026-11-12T14:00:00.000Z2026-11-12T07:00:002026-11-12T14:00:00.000Z✅ exact

Reproduce: apps/web/scripts/service-matcher-fanout/fanout.test.ts on chore/service-matcher-fanout-test, run with AUTOOPS_FANOUT_LIVE=1 AUTOOPS_FANOUT_BOOK=1.

The Quebec extraction returned 2026-06-11T07:00:00 for a transcript that said Thursday, November 12th at 7:00 AM. The output isn't a parse of the slot text — it's an unrelated date. Pure model sampling variance.

Why we care ​

In production, extractCallData runs once per call (lib/auto-service/extraction.ts:59). If that single call lands on a Quebec-shaped outcome, the post-call workflow books the appointment at the wrong date with no detection mechanism. Customer expects Nov 12; AutoOps records June 11.

Detection in prod today is reactive at best:

  • The shop sees an unexpected appointment land on their calendar.
  • The customer arrives on a day there's no record of them.
  • customerNotes retains the agent's spoken confirmation, which would not match scheduledStartAt — but no system compares the two.

We don't have a per-call sampling distribution, so the 1-in-6 rate from this test isn't a population statistic. It's an existence proof. Worth understanding the failure shape and whether existing safeguards (e.g., convertScheduledAtToUtcIso schema validation) catch enough of it.

Root cause hypothesis ​

extractCallData calls openai.chat.completions.create({ model: 'gpt-4o', messages, response_format: { type: 'json_object' } }) with no temperature, top_p, or seed parameter. Default sampling applies. Even with strict JSON-mode, the model can produce semantically-divergent date strings for identical inputs — particularly when the source transcript references multiple dates (initial-availability mentions, customer's "two weeks from now" framing, the substituted final-confirmation line).

The current transcript has at least four date-bearing lines:

  • Bot offering Wednesday availability for the initial day
  • User saying "two weeks from now"
  • Bot offering {{SLOT_DATE}} (rendered as Thursday Nov 12)
  • Bot confirming {{SLOT_FULL}} in the final wrap-up

Even when the placeholder lines align, ambient ambiguity in the transcript is enough for sampling to swing the model toward an unrelated date.

Mitigations to discuss ​

OptionTradeoff
Set temperature: 0 in the extraction OpenAI callCheapest. Reduces sampling variance significantly but doesn't eliminate it. Doesn't break anything in callers.
Cross-check extracted scheduledAt against the in-call autoOpsGetAvailability selectionStrongest. The agent already chose a slot via the in-call tool; that selection is in the call's audit trail. Compare against the post-call extraction; flag/reject on mismatch beyond a threshold.
Re-extract on inconsistencyRun extraction twice with temperature: 0. If outputs disagree, fall through to graceful failure + manual review. Doubles the OpenAI cost per call.
Schema-validate scheduledAt against sensible boundsAdd a Zod check: scheduledAt must be within N days of created_at. A 154-day skew would fail. Cheap, but doesn't catch in-window divergences (e.g., wrong-week-same-month).
Compare scheduledAt against customerNotes content via second LLM passValidates the agent said what we extracted. Expensive, but high signal.

The most surgical: temperature=0 plus a Zod bound on the extracted scheduledAt relative to the call's created_at. Captures the obvious skews without changing the workflow shape. Cross-check against the in-call tool selection is the architecturally correct answer but requires plumbing the in-call audit trail into the post-call workflow.

What we should mirror in our work ​

  • Any new LLM-driven extraction we add should set temperature: 0 by default.
  • Any boundary that converts model output to a real-world action (booking, customer creation, etc.) should validate the output's plausibility before committing.
  • Round-trip assertions like the one in fanout.test.ts are cheap and surfaced this. The pattern (test asserts extract(render(slot)) == slot) is reusable for any future extraction path we add.