Prod Validation Runbook

The after-merge checklist for any post-call workflow PR. Originally written for PR #9877 (AutoOps reschedule + cancel), generalized so future PRs follow the same shape.

Unsolicited Opinion

"Merge and test in prod" is not optional. Every post-call workflow change needs a real call walked through every layer before we trust it. This runbook makes that walkthrough cheap and reproducible, not improvised.

When to use this

Run this ritual after merging any PR that:

Adds a new branch to the post-call dispatcher (runAutoOpsBookingWorkflow or sibling)
Changes how AutoOps mutations are wired
Touches the AutoOps client (lib/autoops/autoops.ts)
Modifies the LLM extraction → workflow handoff
Or any time Kareem says "test it in prod"

Pre-conditions

Merged PR is deployed to prod (avoca-next Vercel main deploy points at the merged commit).
You have AutoOps API access (AUTOOPS_API_KEY in your env).
You have a phone you can call EAS Ponderosa from.
You can read Datadog logs for service:avoca-next-prod.

Open these tabs before placing the test call

Stack them across two monitors if you have them — you'll be reading all four during the call:

Tab	What to watch	Refresh?
Vapi dashboard → Calls → (live call)	Transcript appears turn-by-turn; tool calls appear with expandable args + results	Auto-refreshes
Vapi dashboard → Assistants → EAS assistant	System prompt + tool definitions for reference (cross-reference what the agent SHOULD do vs what it does)	Static
Datadog logs with query `service:avoca-next-prod` time range "last 15 min"	Each `/api/vapi/tools/dispatch` POST shows up; tool handler errors surface here first	Auto-tail (or refresh every few sec)
Inngest production dashboard → Functions → `EndOfCallReportInngestFunction`	After call ends, the run materializes within seconds; expand step graph	Refresh after call ends

If you have access to the Datadog MCP (post-Claude Code restart), ask the agent to query for recent /api/vapi/tools/dispatch POSTs while the call is happening — the agent can stream-explain what it sees.

For background on what each surface represents in the in-call loop, see the in-call sequence page — it has the turn-by-turn diagram and a bug-pointing matrix ("here's the symptom, here's where to look").

The 6-step ritual

1. Confirm the deploy

Check that prod is on the merged commit, not stale.

bash

# in your avoca-next checkout
git fetch origin main
git log origin/main -1 --oneline

Compare that commit hash to Vercel's most recent prod deploy:

https://vercel.com/avoca/avoca-next/deployments?environments=production

Both should match. If they don't, wait for the deploy or investigate before proceeding.

2. Create a test job in AutoOps

Never test against real customer jobs. Create a fresh job with the EAS shop's customer record so the test stays in your sandbox.

bash

# from the toolkit (paths assume EAS Ponderosa)
AUTOOPS_API_KEY=${key} ./scripts/observe/autoops-job.sh create-test-job

TIP

If you don't have a create-test-job helper yet, create one inline via Avoca's apps/web/scripts/list-test-jobs.ts pattern. Future TODO: extract a reusable create-test-job.ts script.

Record the returned jobId — you'll reference it in the call transcript and during verification.

3. Place the test call

Call EAS Ponderosa's number from your phone. Drive the conversation to exercise the path the PR changed:

Path	Conversational trigger
Booking	"I'd like to schedule a service appointment"
Reschedule	"I need to reschedule job `${jobId}` to next Tuesday at 9am"
Cancel	"I need to cancel job `${jobId}`"

Speak the jobId clearly — the LLM extracts it from the transcript and validates against AutoOps. If the LLM misses it, the workflow returns a graceful failure with reason: 'jobid-not-found'.

After the call ends, note the time + your phone number. You'll need both to find the call record.

4. Trace the call through every layer

Each scripts/observe/ script gives you a deep link or query for one layer.

Layer 1 — Vapi (call exists):

bash

./scripts/observe/last-call.sh
# Open the Vapi dashboard link, find your call by time, copy the call_id

Layer 2 — Avoca call record (post-call processing started): Run the SQL from last-call.sh's output in Supabase Studio. Find your call by caller_id (your phone) + recent created_at. Note the avoca-internal id (uuid) — that's your callId.

Layer 3 — Inngest workflow run (post-call dispatch fired):

bash

./scripts/observe/inngest-run.sh ${callId}
# Open the Inngest production dashboard link, find the EndOfCallReportInngestFunction run

For your run, expand the step graph and confirm the right step ran:

runAutoOpsBookingWorkflow — should always run for AUTO_SERVICE
The step's input shows extraction.extractedData.avoca.callReason
The step's output shows the result (success or graceful failure)

Layer 4 — AutoOps state (mutation landed):

bash

./scripts/observe/autoops-job.sh ${jobId}

What you expect depends on the path:

Path	Expected AutoOps state
Booking	New job exists with the booked time
Reschedule	Job's `currentStartAt` updated to the new time
Cancel (SMS-typed customer)	Job state UNCHANGED, AutoOps API returned 400 "Client SMS does not support cancels" — captured as graceful failure

Layer 5 — Email path (shop notified): Check Resend's dashboard or your inbox if you're configured as a team recipient:

https://resend.com/emails

For success cases: subject does not contain [AUTOOPS ERROR]. For graceful failures (e.g., SMS-cancel rejection): subject is prefixed [AUTOOPS ERROR] and the body's autoops_error field carries the rejection reason.

Layer 6 — Datadog (no unexpected errors): Search Datadog for the call's window:

service:avoca-next-prod @callId:${callId}

Look for:

Any level:error entries — should be zero unless the test deliberately exercised an error path
The expected info-level breadcrumbs (extraction, leaf invocation, AutoOps API call, email send)

5. Verify outcomes match expectations

Walk through the matrix for the path you exercised:

Reschedule

[ ] Inngest run shows runAutoOpsBookingWorkflow step ran with callReason: 'Rescheduling'
[ ] AutoOps job's currentStartAt matches the requested time (in the team's IANA tz, converted to UTC)
[ ] Email sent (no [AUTOOPS ERROR] prefix)

Cancel (SMS-typed customer, e.g., EAS Ponderosa)

[ ] Inngest run shows runAutoOpsBookingWorkflow step ran with callReason: 'Cancellation'
[ ] Inngest step output shows success: false, reason: 'autoops-rejected', status: 400
[ ] AutoOps job state UNCHANGED
[ ] Email sent with [AUTOOPS ERROR] prefix and autoops_error field populated

Booking

[ ] Inngest run shows runAutoOpsBookingWorkflow step ran with callReason: 'Booking' and appointmentBooked: true
[ ] AutoOps shows a new job with the booked time
[ ] Email sent (no error prefix)

If any row is unmet, file a Linear issue or message Kareem with the call's callId + the row that failed.

6. Clean up

Cancel the test job manually via AutoOps's UI (or leave it if the test was a cancel — the email already flagged it for the shop).

Document the test in internal/clients/avoca/change-log.md (internal-only, not deployed) under a "Prod validation" entry: which PR, which paths exercised, which callIds, any anomalies.

Common failures

Inngest run is missing entirely. The Vapi end-of-call webhook didn't reach Avoca. Check scripts/observe/last-call.sh's Datadog query for /api/vapi/tools/dispatch — should be a POST during the call's window. If absent, the assistant config might still be pointing at staging or an unconfigured webhook URL.

Inngest run shows callReason: '' or wrong reason. The LLM extraction didn't classify correctly. Inspect the step's input — the transcript should contain the explicit ask ("reschedule", "cancel"). If it does and the LLM still mis-classified, file an extraction-prompt issue against the auto-service prompt.

reason: 'jobid-not-found' for a reschedule/cancel test. The LLM didn't pick the jobId out of the transcript. Reread your call: did you say the jobId clearly? If yes, this is a real extraction bug — file it. If no, redo the call with the jobId spoken in full.

AutoOps updatedAt doesn't reflect the reschedule. Known upstream bug: AutoOps's reschedule endpoint returns 200 and currentStartAt updates, but updatedAt stays at job creation time. Use currentStartAt (or scheduledStartAt depending on the schema version) to verify the mutation, not updatedAt. See PR #9877's follow-ups for the future-jobs-sync workaround.

Cancel returns 400 "Client SMS does not support cancels" — is that a bug? No. AutoOps disallows API-initiated cancels for SMS-typed customers (permanent product policy). The 400 + graceful-failure-with-[AUTOOPS ERROR]-email path is the steady-state design for SMS customers, not a bug to fix on Avoca's side.

Future improvements

Tagged test-call filtering: a header or env-data flag that marks "Sandy is testing" so we can filter Datadog/Inngest/calls noise.
A create-test-job.ts reusable helper.
A Datadog dashboard for graceful-failure paths (currently silent; only the email surfaces them).
Hamming-on-demand for non-prod regression coverage of these flows (deferred to Plan H).

Prod Validation Runbook ​

When to use this ​

Pre-conditions ​

Open these tabs before placing the test call ​

The 6-step ritual ​

1. Confirm the deploy ​

2. Create a test job in AutoOps ​

3. Place the test call ​

4. Trace the call through every layer ​

5. Verify outcomes match expectations ​

Reschedule ​

Cancel (SMS-typed customer, e.g., EAS Ponderosa) ​

Booking ​

6. Clean up ​

Common failures ​

Future improvements ​

Prod Validation Runbook

When to use this

Pre-conditions

Open these tabs before placing the test call

The 6-step ritual

1. Confirm the deploy

2. Create a test job in AutoOps

3. Place the test call

4. Trace the call through every layer

5. Verify outcomes match expectations

Reschedule

Cancel (SMS-typed customer, e.g., EAS Ponderosa)

Booking

6. Clean up

Common failures

Future improvements