Avoca — Internal

Appearance

Does the post-call LLM see the same context as the in-call agent? ​

Abstract ​

A common architectural objection to in-call "commit tools" (e.g., the proposed autoOpsConfirmAppointment in solutions/cross-shop-availability-and-commit-tool) is that they appear redundant: the post-call extraction LLM has access to the call's transcript, so it can re-derive the booking target the in-call agent already decided. Under that framing, the commit tool only adds latency without adding information. This is incorrect. In Avoca's current implementation, the post-call LLM sees a text transcript with tool-call entries filtered out, not the in-call agent's structured tool-result context. The commit tool's value is not new information; it is structural certainty about a decision the in-call agent already made, captured as JSON on the call record so the post-call workflow can act on it without further LLM inference.

Conclusion ​

The post-call extractor is given a flat user/bot transcript string. The structured outputs of in-call tool invocations (e.g., the shopClientId, serviceId, slotTimeUtc per slot returned by autoOpsGetAvailability) are not part of its prompt. The in-call agent, by contrast, sees those structured outputs natively in its message history. The commit tool closes this gap by attaching the in-call agent's decision to the call record as structured toolCalls[] data. The post-call workflow reads those args directly, eliminating LLM inference from the booking-target step. Latency is masked by Vapi's request-start message; perceived cost to the caller is zero.

High-level explanation ​

The objection ​

The argument runs: the LLM is stateless. The in-call agent's "context" is fully expressed in the message history sent at each turn. The post-call extractor can read the same transcript and reason to the same conclusion. Therefore the commit tool just slows down the call to memorialize a decision that's already in the transcript.

The argument's premise is correct (the LLM is stateless; no hidden persistent memory exists). The conclusion does not follow, because the post-call extractor is not given the same message history.

What the in-call agent sees ​

Every turn, Vapi assembles a message history for the in-call LLM. That history contains:

  • The compiled system prompt
  • User messages (transcribed customer speech)
  • Assistant messages (what the agent has said)
  • Structured tool-call invocations ({ name: "autoOpsGetAvailability", arguments: {...} })
  • Structured tool-call results (full JSON return values, including arrays of slots with slotTimeUtc, shopClientId, per-shop metadata blocks, etc.)
  • variableValues injected into the prompt at call setup

The agent's "I have 5pm at Chatfield or 6pm at South Park" utterance is generated from a structured JSON object in its context. The shop identity is metadata it actually saw.

What the post-call extractor sees ​

The post-call extraction's chat history is constructed by buildExtractionChatHistory at apps/web/lib/auto-service/extraction.ts:49:

ts
export function buildExtractionChatHistory(
  extractionPrompt: string,
  transcript: string
): ChatMessage[] {
  return [
    { role: 'system', content: extractionPrompt },
    { role: 'user', content: transcript },
  ];
}

System message + a single transcript string. That's the extractor's entire context window for the call.

Upstream, the transcript itself is filtered at auto-service-post-call.ts:116:

ts
const transcriptMessages = ctx.report.structuredMessages
  .filter((m) => m.role === 'user' || m.role === 'bot')

Only user and bot roles survive into the rendered transcript. Tool-call invocations and tool-call results are stripped before the extractor ever sees them.

Where the gap shows up in practice ​

For a single-shop booking flow the gap is usually invisible: the transcript is unambiguous ("the 5pm Tuesday slot") and the extractor reconstructs the booking target reliably. The gap is dormant.

For the cross-shop case the gap becomes load-bearing. autoOpsGetAvailability returns slots tagged with shopClientId. The agent's spoken disambiguation ("5pm at Chatfield") may or may not preserve shop attribution clearly in the transcript. The extractor has to correlate transcript wording back to one of N structured tool-result entries that it can no longer see. Reconstruction sometimes fails. The commit tool closes this by capturing the agent's choice as { shopClientId, serviceId, slotTimeUtc } at the moment the choice was made.

Why the commit tool is not just a latency cost ​

The commit tool's value is not adding information that the post-call LLM doesn't have access to in principle. Its value is removing the post-call LLM as a decision-maker for one specific step (booking target identification), replacing inference with structured certainty.

Latency: ~200ms per round trip, fully masked by Vapi's existing request-start message ("Got it, submitting your request") that plays in parallel with the tool execution. The caller perceives no added delay. This is documented in the cross-shop solution doc.

Detailed analysis ​

Vapi's message model ​

The in-call LLM is invoked per turn with an OpenAI-format messages[] array that includes structured tool calls and results inline, exactly as the OpenAI Chat Completions API expects. The same data is preserved in Vapi's call record at artifact.messages after the call ends. The data exists at the platform level.

The constraint is purely an Avoca implementation choice: today's post-call extractor consumes a rendered text transcript instead of the raw structured messages. That choice has consequences:

  • A long sales call's tool-call results can be megabytes of JSON; converting them to a transcript string drops that detail by design.
  • Multiple CRM verticals share the same extractor shape (AutoOps, AlbiWare, Window Nation), each with its own extraction prompt. Switching to a structured-messages consumer would require touching each.
  • Today's extractor prompt is built around natural-language reasoning ("identify the booking target from this conversation"); rewriting it to consume structured tool calls is a larger change than swapping in a string.

So the gap is real for the post-call extractor as currently built, even though Vapi has the data.

Three architectural options ​

Three ways to align the post-call extractor's context with the in-call agent's context:

OptionApproachCostReliability
(a) Status quoPost-call extractor receives a text transcript. Tool-call results are stripped.Zero — already shipped.LLM reconstructs the booking target from natural language. Works for single-shop. Fragile for multi-shop attribution.
(b) Structured messagesRewrite post-call extractor to consume artifact.messages directly, including tool-call invocations and results.Higher — every CRM's extraction prompt and pipeline. Per-CRM regression risk.LLM still has to correlate transcript wording to one of N structured tool results. Better than text-only but still inference-based.
(c) Commit toolIn-call agent invokes a no-op function tool at the decision moment, with { shopClientId, serviceId, slotTimeUtc } as structured args. The args are saved to toolCalls[] on the call record. Post-call workflow reads those args directly.Lower — one new tool handler + one workflow read path.No LLM inference at the booking-target step. The args ARE the decision.

Option (c) is what the cross-shop PR proposes. It is strictly cheaper than (b) AND strictly more reliable, because it removes the LLM from one inference step entirely instead of giving the LLM more data to reason from. Option (a) is what we have today; it works adequately for single-shop and is the active risk for multi-shop.

Why (c) is more reliable than (b) ​

Even with full structured messages (option b), the post-call LLM still has to determine which of the structured tool results matched the customer's eventual choice. The transcript says "5pm" and the structured results include a 5pm Chatfield slot and a 5pm South Park slot; the extractor still has to correlate the customer's words back to one entry. This correlation is the failure mode.

The commit tool removes that correlation step entirely. The in-call agent, which made the choice in conversation with the customer, attaches its choice as structured arguments. The post-call workflow reads toolCalls[], finds the autoOpsConfirmAppointment entry, and uses its args. No correlation required.

Stateless does not mean equally-informed ​

The architectural claim "the LLM is stateless, so the post-call LLM has the same context" conflates two separate questions:

  1. Does the LLM have hidden state? No. The LLM is stateless. State is fully captured in the prompt + message history + variableValues at each turn.
  2. Does the post-call LLM receive the same prompt + message history + variableValues as the in-call agent received? No. Today it receives a flat transcript with tool-related entries stripped.

Question 1's answer is true and important. Question 2's answer is what the commit tool is designed to address. The objection conflates the two.

A subtler reasoning gap ​

Even if question 2's answer were "yes" (post-call LLM receives the full structured messages), there is still a difference in how the two LLMs reason:

  • The in-call agent makes decisions one turn at a time. At turn N, it sees the prompt + messages 1..N, makes one decision, returns one response.
  • The post-call extractor makes one decision after the call, with the prompt + messages 1..N+M visible all at once.

These are not the same cognitive frame. In-call decisions are local; post-call extraction is global. For a conversation where the customer wavered ("the 5pm... actually let's see what else... OK the 5pm"), the in-call agent's decision at the moment of confirmation has structural priority over any single sentence in the transcript. The post-call extractor reading the whole conversation has to weigh all the sentences against each other and pick a winner.

This is not the load-bearing argument for the commit tool, but it's a real second-order benefit. The commit tool preserves the in-call agent's "decision moment" rather than asking another LLM to reconstruct which moment was decisive.

Implications ​

  • For cross-shop specifically: the commit tool is necessary for reliable shop attribution. Without it, transcript correlation will sometimes mis-attribute the chosen slot to the wrong shop. This is the load-bearing case.
  • For single-shop: the commit tool is defense in depth. The transcript usually suffices; the commit tool makes "usually" into "always."
  • For other workflows that re-derive in-call decisions post-call: the same pattern applies. Anywhere we currently rely on a post-call LLM to identify what the in-call agent committed to, capturing that commitment as structured toolCalls[] args is strictly more reliable than transcript inference.
  • For the long-term post-call architecture: option (b) (structured-messages consumer) is a worthwhile follow-up regardless. It is not a substitute for the commit tool, but it would tighten the extractor's natural-language reasoning for fields the commit tool doesn't cover (customer name, vehicle, callback phone, etc.).

Out of scope ​

  • Performance characterization of option (b) vs option (c). The commit tool is cheaper to implement; that is sufficient for V1.
  • Whether the commit-tool pattern generalizes to other Avoca verticals (ServiceTitan, HCP). It does in principle. Each vertical adopts when it benefits from structured certainty at a decision point. The cross-shop PR documents this as a follow-up consideration.
  • LLM-quality measurement of the post-call extractor's failure rate on multi-shop attribution. Worth measuring once the commit tool is live as a counterfactual baseline.

Source references ​