Avoca — Internal

Appearance

Voice-agent feature workflow (Hamming-driven) ​

The end-to-end pattern for shipping a voice-agent behavior change — prompt edit, new tool, intent flow, etc. Goes from local code change to production regression test in one ritual.

Status

v1 — proposed pattern. Not yet validated end-to-end. Plan P's env-var unblocks must merge before phase 4 fully works; Plan K (the phone-not-found prompt edit) is the first plan that will run through this playbook. Refinements will land after Plan K's first pass.

Why Hamming-driven ​

Per feedback_hamming_for_all_voice_testing: Hamming is the testing surface for Avoca voice work. Don't split with Vapi Test Suites. One tool, build fluency, accumulate coverage.

The win of doing this as a workflow: the same Hamming test artifact that gates a PR becomes a permanent production regression test. Voice-agent tests accumulate as the team's coverage instead of being throwaway smoke at PR-time.

The six phases ​

#PhaseSurfaceOutput
1Local buildWorktree on FDE laptopWorking code
2Manual smokeDial test phone"It does the thing on my code" — eyeball confidence
3Author Hamming testHamming dashboardTest scenario (script + rubric) capturing the verified behavior
4Local Hamming runHamming dials FDE's test phone → routes to FDE tunnelGreen Hamming run against branch code
5PRGitHubPR with description linking the Hamming test
6Post-merge prod validationHamming run repointed at prodProduction-validated test, now part of regression suite

Phase detail ​

1. Local build ​

Set up local dev per local-dev-env.md:

  • Worktree of avoca-next off main
  • Tunnel running (Cloudflare / ngrok / Tailscale Funnel)
  • .env.local has NGROK_BASE_URL + TEST_PHONE_POST_CALL set (per Plan P)
  • Test phone synced (Sync test phone action in admin UI bakes your tunnel into both URL surfaces)

Then write your code. Prompt edit lives in assistant_configs.multi_agent_config JSON or via Avoca admin's prompt-builder, depending on the assistant's mode (see vapi-squads.md and agent-architecture-legacy-vs-current.md).

2. Manual smoke ​

Dial the test phone from your phone. Drive the conversation through the scenario you're changing. Watch the dev server logs:

  • POST /api/responder/common/workflow lands locally (assistant-level events reach your tunnel)
  • POST /api/vapi/tools/dispatch lands locally (tool dispatches reach your tunnel — this is what NGROK_BASE_URL enables)

Verify the agent behaves the way you intended. The smoke isn't proof — it's "I'm not building against a hallucinated mental model."

3. Author Hamming test ​

In the Hamming dashboard, create a scenario that captures the behavior you just verified manually.

  • Script — what the test caller says. Multi-step instructions; the AI tester improvises within them.
  • Rubric — questions the LLM uses to judge success. "Did the agent ask for a different phone number before asking for first/last name?" — yes/no, plus reasoning.
  • Target — your FDE-owned test phone. (Per Plan Q's multi-test-phone schema, each FDE has their own.)

Author the test on the behavior you just shipped manually, not on the broader feature. One test, one behavior. Multiple tests if you fixed multiple behaviors in one PR.

Authoring caveat

Hamming tests are dashboard-authored and live in Hamming's cloud — not in your repo. There's no "git commit a Hamming test" workflow today. The test artifact persists in Hamming permanently; reference it from your PR description by URL, not by file path.

4. Local Hamming run ​

Trigger the Hamming test. Hamming dials your test phone via SIP. Twilio routes the call per the test phone's voice URL (set to Avoca's routing webhook). The voice URL hands off to Vapi, which uses the squad you synced in phase 1. Sub-agent assistant.server.url is your tunnel; per-tool server.url (built via buildUrl(NGROK_BASE_URL)) is your tunnel. The entire dispatch chain reaches your localhost.

When the test passes against your branch:

  • The agent did the right thing under the rubric
  • Your local code path executed (verify in dev server logs)
  • You have evidence beyond "it worked when I dialed manually"

If it fails: iterate (back to phase 1 or 3 depending on whether the prompt edit or the test rubric was wrong).

5. PR ​

Open the PR with:

  • Tight description per feedback_dry_terse_artifacts (don't over-justify)
  • A line referencing the Hamming test URL (so reviewer can re-run it independently if they want)
  • Note that the test passes against the branch's code

Reviewer can request changes; iterate phase 1-4 as needed. The Hamming test serves as a shared falsification target — "your prompt edit must keep this test green."

6. Post-merge prod validation ​

After the PR merges and the change deploys to production:

  • Repoint the Hamming test from your FDE-owned test phone to the production test phone (or the agreed shared regression target)
  • Run the test against prod
  • If green, the test joins the persistent regression suite

The test now lives indefinitely as production coverage for the behavior you shipped. Future PRs that touch the same area run the same test as part of CI. The behavior is now permanently falsifiable.

Open: how exactly does the repoint happen

The mechanic for "repoint Hamming test from FDE phone to prod phone" is TBD. Could be:

  • Edit the test target in the Hamming dashboard
  • Duplicate the test with a new target (one for dev, one for prod)
  • Some workspace-level setting

Will be resolved when Plan K runs through this playbook and we hit this step for real.

What's NOT in this playbook ​

  • One-off / throwaway tests. If you genuinely don't want the test to persist, don't author it — phase 2's manual smoke is enough.
  • Code-only changes that don't affect agent behavior. Refactors, type cleanups, structural changes that preserve behavior — these don't need Hamming coverage. Use unit/integration tests via existing patterns.
  • Pre-PR exploration. If you're not sure what behavior you want yet, iterate at phase 1-2 freely. Only commit to a Hamming test when you've decided what "correct" looks like.
  • Multi-agent / squad structural changes (adding a sub-agent, changing hand-off rules). The playbook covers prompt edits + tool behavior changes within an existing squad shape. Structural changes need a different review path.

Open verification gaps (v1) ​

This playbook hasn't been run end-to-end yet. Things to validate when Plan K is the first feature through:

  1. Phase 4 actually works. Plan P's env-var unblocks haven't fully merged; once they do, verify that Hamming dialing the FDE's test phone reaches localhost as expected.
  2. Phase 6 repointing mechanic. See callout above — TBD.
  3. Hamming workspace conventions at Avoca. Are tests per-FDE, per-team, or shared? Affects step 3 authoring.
  4. PR description format. Establish a convention for "this PR is gated by Hamming test X" — link, expected status, etc.

Each gap becomes a tightening of the playbook after Plan K runs.