The Weekly AI Call Review SOP

Purpose

An AI phone answering service sounds fine on every call you happen to hear, because it produces fluent, polite speech whether it routed the call correctly or not. The failures sit in the calls nobody listens to: the flooded basement booked as a routine leak for Thursday, the caller in the next county given a Tuesday slot. This procedure guarantees that someone hears a fixed, deliberately chosen slice of calls every week, scores them the same way, logs every misroute, fixes the script one pattern at a time, and takes the system off the phones when the stop rule says to. Without it, the first sign of a broken script is a complaint or a damaged house.

Scope

Covers: the weekly review of calls already handled by an AI answering or booking system, from pulling the sample through scoring, logging, the script fix and the decision to keep the system on or pull it.

Does not cover:

• Initial setup and pre-launch testing. See related: Setting Up an AI Phone Answering Service and Testing It, which owns the test-call method this SOP reuses in step 8.
• Writing the system's FAQ content. See related: Building a Phone Agent FAQ From Real Calls.
• Approval and rollback of a configuration change. See related: The Automation Change Control SOP, which step 7 hands off to.
• Real-time emergency handling. This review lags; the live safety net is the system's rule that emergency words transfer to a person at once. This SOP verifies that rule worked, it does not replace it.

Assumes lawful recording: the federal floor is one-party consent (18 U.S.C. 2511(2)(d)), but about a dozen states, California and Florida among them, require every party's, so the greeting announces recording. See related: AI Meeting Notes, Call Recording and Consent.

Roles and responsibilities

One person may hold two roles, except that whoever wrote a change never scores its step 8 test: the author hears the call they expected.

Procedure

Step 1: Pull the week's calls and fix the sample before listening

Export the list of every call the system handled from Monday 00:00 to Sunday 23:59. Build a sample of 25 calls in this order: every call the system tagged as an emergency or whose transcript contains one of the shop's emergency words (gas, smell, carbon monoxide or CO alarm, smoke, sparking, burning, flooding, water near a light, outlet or panel, no heat or no cooling with an infant, an elderly or a medically fragile person in the house); every call transferred to a person; every call under 30 seconds or abandoned; then random calls to make 25. Acceptance: a numbered list of exactly 25 call IDs (or every call, if the week had fewer than 25), with the stratum written beside each, saved before the first recording is played. Wrong looks like a sample chosen after listening, or an all-random one, which almost never lands on the rare emergency call that matters. If the three mandatory strata alone exceed 25, review every emergency call and cap the rest at 25 total. Hazard: none at this step, it is a desk task.

Step 2: Listen to each call in full, with the transcript open

Play the recording from greeting to hang-up while reading the transcript. Where they differ, the audio wins and the difference gets a note, because the system acted on its own transcript and a mishearing ("Elm Court" becoming "Elm Port") is a cause, not a footnote. Acceptance: all 25 calls marked as listened, with durations written down. Wrong looks like scoring from the system's own call summary; the summary is the model describing what it believes happened, in the same confident voice whether or not it was right. If a recording is missing or truncated, fail that call on the capture line and log the missing recording as its own finding. Hazard: none physical; use headphones in a shared office because callers state addresses and access details.

Step 3: Score each call on the five-line rubric

Mark each line pass or fail. (1) Intent: the system identified what the caller needed. (2) Route: the call ended in the right place: booked, transferred, message taken, or politely declined as out of area. (3) Capture: name, callback number, full service address including unit or lot number, and the problem in the caller's own words. (4) Promises: no price, arrival time, warranty or diagnosis beyond what the approved script allows. (5) Close: the caller was told they were speaking with an automated assistant where shop policy requires it, and told what happens next. Acceptance: five marks on all 25 calls, no blanks. Wrong looks like passing a call because it "sounded good", which is why tone is not on the rubric. Hazard: none at this step.

Step 4: Classify every failure and write it into the misroute log

A failure is critical if an emergency was handled as routine, a caller who asked for a person was not transferred, a price or commitment outside the script was given, or an out-of-area job was booked. Everything else is non-critical. Each failed call gets one log row: call ID, date and time, rubric line, what the caller said, what the system did, and critical or not. Acceptance: the count of log rows equals the count of calls with at least one failed line, and every critical row carries a callback owner's name. Wrong looks like a critical failure softened to non-critical because the customer "seemed fine"; classification follows what the system did, not how lucky the outcome was. Hazard: none at this step.

Step 5: Act on every critical failure the same day

The dispatcher calls back every customer behind a critical row before end of business and fixes the booking. Acceptance: each critical row shows the callback time and its outcome. Stop rule: any emergency handled as routine, or two or more critical failures of any kind in one sample, takes the system off the phones now: calls forward to the office during hours and to a human answering service or voicemail with a stated callback window after hours, and it stays off until step 8 passes. A single non-emergency critical failure is fixed this week with the system left on. Hazard: if the callback finds a live emergency, instruct first. For a gas smell or a CO alarm, everyone leaves the house now, touches no switches, and calls the gas utility or 911 from outside. For water near a fixture, outlet or panel, nobody touches the switch or stands in the water, and the water is shut off at the main only if it can be reached without crossing the wet area. Then dispatch, and if that does not hold because the caller cannot get out or cannot reach the main, the dispatcher stays on the line and calls 911.

Step 6: Compute the week's rates against the gates

Count calls passing all five lines, non-critical failures and critical failures, and check that the three counts add to the sample size. Non-critical failure rate is non-critical calls divided by sampled calls. Acceptance: the week passes with zero critical failures and non-critical failures on at most 3 of 25 sampled calls, 12 percent. Wrong looks like comparing against all calls handled rather than calls sampled, which divides a sample count by a weekly total and produces a reassuringly tiny number. Stop rule: above 12 percent for one week means a script change is required in step 7; above it for two consecutive weeks means the owner decides whether to narrow the system's job, for example to message-taking only. Hazard: none at this step.

Step 7: Make one change per failure pattern and set its recheck

Group the log rows by cause, not by rubric line, and write one script change per cause, routed through change control rather than edited live. Acceptance: every critical row and every non-critical pattern with two or more rows maps to a logged change, each naming the 10 calls of that intent next week's sample adds to recheck it. Wrong looks like five fixes in one edit, after which nobody can tell which change helped and which broke something new. Hazard: none at this step.

Step 8: Test before restoring, then restore

Before a pulled system goes back on, the system admin places 10 scripted test calls of the failed intent, phrased the ways real callers phrased it, and a reviewer who did not write the change scores them. Acceptance: 10 of 10 handled correctly, emergency calls reaching a person within 60 seconds of the emergency words. Stop rule: any miss keeps it off, back to step 7. Hazard: restoring is the step where the next real emergency caller meets the change, so keep after-hours forwarding live until the owner initials the restore and the first real emergency call afterwards has been heard.

The record this produces

• The review log: one row per sampled call (ID, stratum, duration, five marks), initialled by the owner. It answers "were we checking?" a year later.
• The misroute log: one row per failed call with classification, callback owner and outcome. Read monthly for patterns that recur after a fix.
• The change record: cause, change, date live, recheck calls, result, filed through change control.
• The pull-and-restore record: time pulled, who decided, forwarding arrangement, test results, time restored, initials. This shows a reasonable shop caught its own fault.

At about 3 minutes of listening plus 3 of scoring per call, 25 calls cost roughly 150 minutes a week.

Worked pass

Week of a Monday in October. The system handled 212 calls.

Step 1. Sample: 6 emergency-tagged or keyword calls, 4 transfers, 3 short calls, 12 random, which is 6 + 4 + 3 + 12 = 25. Saved before listening. Pass.

Step 2. All 25 played. One transcript read "Elm Port" for "Elm Court"; noted. Pass.

Step 3. Results: 19 calls passed all five lines, 5 failed one non-critical line, 1 failed critically. 19 + 5 + 1 = 25. Pass.

Step 4. Six log rows. Non-critical: 3 capture failures (unit number never asked), 2 close failures (caller not told what happens next). Critical: Saturday 21:40, "water is dripping out of the kitchen ceiling light", booked as a routine leak for Tuesday 13:00. It came from the emergency stratum; the 12 random calls would not have found it. Pass.

Step 5. FAILS, stop rule taken. Callback Monday 09:10. The customer had shut the main Saturday and left the light off. The dispatcher told her not to use that switch or enter the wet area, and moved the job to same day. An emergency handled as routine pulls the system: off at 09:25, office forwarding by day, answering service by night.

Step 6. Non-critical: 5 of 25 sampled calls, 20 percent, above the 12 percent gate. Critical: 1. Fails; step 7 required.

Step 7. Two causes, two changes. The emergency intent now asks "Is any water near a light, an outlet or the electrical panel?" and transfers on yes. The address flow now asks for a unit or lot number on every booking. The two close failures shared no cause and stay on watch. Next week's sample adds 10 water-intent calls.

Step 8. First run fails. 9 of 10 test calls transferred. "The light's flickering and there's water on the floor" was booked as routine, because the new question only fired on "dripping". Stays off; back to step 7, where the trigger was widened to water mentioned with a light, outlet, switch or panel. Second run passes, 10 of 10, each reaching a person in under 60 seconds. Restore initialled Tuesday 16:00, after-hours forwarding kept until the first real emergency call had been heard.

References

• 18 U.S.C. 2511(2)(d), the federal one-party consent provision; check your state's recording statute, which may require all-party consent.
• See related: Setting Up an AI Phone Answering Service and Testing It
• See related: The Automation Change Control SOP
• See related: The AI Booking Agent Booked a Week of Jobs Wrong
• See related: Telling Customers When AI Is Involved