Building a Phone Agent FAQ From Real Calls

Why this matters

An AI phone agent answers from whatever it was given. Most shops give it a list of questions somebody imagined customers ask, written in the language of a website, and then wonder why the agent stumbles on the calls that actually come in. Real callers do not ask "What is your service area?" They ask "Do you come out to the lake side of the county?" They do not ask about "diagnostic fees". They ask "What's it cost just to have somebody look at it?"

The fix is to build the agent's FAQ from your own calls: count what people really ask, write the answers the way your best office person already says them, and test the result against the calls it came from. This card is that build. Setting up the service itself and testing its basic behavior is covered elsewhere. See related: Setting Up an AI Phone Answering Service and Testing It.

Before you start: the recordings, and the right to use them

You need a month of calls, as recordings, transcripts or at minimum a call log with a line on what each caller wanted. Before you mine recordings, confirm they were made lawfully: federal law sets a one-party consent floor, and a number of states, California (Penal Code 632) and Florida (Statutes 934.03) among them, require every party on the call to consent, which is why most shops play a recording notice at the start of every call. If your recordings were made without that notice in a state that requires it, do not build on them; use the call log instead. See related: AI Meeting Notes, Call Recording and Consent.

A month is the right window for most residential shops because it catches a full billing cycle and enough calls to rank, while staying close enough to the present that the questions reflect current prices and service area. If your work is strongly seasonal, pull a month from each season you serve and build a seasonal section; a furnace shop's October calls and July calls barely overlap.

Step 1: Sort every call into a reason

Go through the month and give each call one reason code. Keep the codes coarse at first: booking, status of an existing job, price question, do-you-do-this, hours and emergencies, billing, and not-a-customer (vendors, sales, spam). Coarse codes are fast to assign and show where the volume is; you split them into specific questions in the next step.

Here is what one month looked like at a seven-truck plumbing shop:

Drop the 32 non-customer calls. That leaves 380 customer calls to work from.

What breaks if you skip it: you write answers for the questions you remember, which are the memorable ones, and the agent fails on the boring high-volume ones that make up most of the day.

Step 2: Split the reasons into the questions people actually ask

Now take each reason and list the distinct questions inside it, written in the caller's words. "Price questions" at the plumbing shop split into eight: what it costs for someone to come look, whether that fee comes off the repair, what a water heater runs, whether there is an after-hours charge, and so on. Count each.

At this shop, 66 distinct questions came out of the 380 calls. The twenty the shop finally chose, picked as described next, covered 301 calls, about 79 percent of the 380. The remaining 79 calls spread across 46 questions, most asked once or twice.

Choose the twenty by frequency and by cost of a wrong answer, not frequency alone. Two questions made the list here despite being asked only three and four times in the month: "I smell gas near the water heater, what do I do?" and "Sewage is coming up in the basement tub, is it safe to run the washer?" A wrong answer to either costs far more than a wrong answer to the most common question. They go in. So does anything touching price, warranty or a promise about timing, because those are the answers that turn into obligations. See related: The Chatbot Promised a Price We Do Not Charge.

What breaks if you skip it: frequency alone produces an FAQ that handles the routine and has nothing for the one call where the agent's words matter most.

Step 3: Write each entry with five parts

A good FAQ entry for a phone agent is not a question and an answer. It is five parts, and the agent needs all of them:

1. The question, plus the ways callers say it. Pull three or four phrasings verbatim from the calls. "Do you come out to Millbrook?", "Is Millbrook too far?", "You service out past the river?" all mean the same thing.
2. The answer, in the shop's words. Find the call where your best office person answered it well and use their phrasing, trimmed. Customers trust a familiar tone, and your staff will recognize the agent as speaking for the shop.
3. The boundary: what the agent must not say. No diagnosis, no price beyond the published ones, no promised arrival time, no warranty interpretation.
4. The handoff rule. When the agent stops answering and transfers, takes a message, or gives an emergency instruction.
5. The source calls. The dates of the calls the entry came from, so anyone revising it later can go back to the real thing.

Here is the price entry the plumbing shop wrote, built from fourteen calls:

• Question variants: "What's it cost just to have someone look at it?" / "Do you charge to come out?" / "Is there a fee for the estimate?"
• Answer: "There's a diagnostic fee for the visit, and if you go ahead with the repair that day, it comes off the repair price. The tech will give you the price before doing any work, so there are no surprises." (The fee itself is stated from the shop's published price list, and the agent reads the current figure from there rather than having it typed into the entry, so a price change happens in one place.)
• Boundary: never estimate the cost of a repair, never say what the problem "probably" is.
• Handoff: if the caller asks for a repair price, say the tech prices it on site and offer to book; if they push, offer a callback from the office.
• Source calls: fourteen, all from the first three weeks of the month (the fourth week is held back for testing, below).

And the safety entry, which is built differently because its job is to get the caller out, not to inform them:

• Question variants: "I smell gas", "smells like rotten eggs by the heater", "is that gas I'm smelling?"
• Answer, given first and before anything else: "Please leave the house now with everyone in it. Don't flip any switches or use anything that could spark on the way out. Once you're outside, call your gas company's emergency line or 911. We'll be here when you're safe."
• Boundary: never troubleshoot, never ask the caller to check the pilot or the valve.
• Handoff: after the instruction, alert the on-call person immediately.

What breaks if you skip parts 3 and 4: the agent answers everything it can generate an answer for, which is everything. The boundary and handoff are what keep a fluent tool from improvising.

Step 4: Test against calls the FAQ was not built from

The most common mistake in testing is replaying the same calls you built from. Of course the agent handles them; you wrote the entries from their wording. Rank the questions on the whole month, but write the entries' wording from the first three weeks only and hold back the fourth for testing. That is the honest test, because week four's callers phrased things their own way.

Feed each held-back caller's opening words to the agent in its test mode and score the result into one of four bins: answered correctly, handed off correctly, answered when it should have handed off, or answered wrongly.

The plumbing shop's week four had 95 calls in the top-twenty questions. First result:

Set the bar before you look. A workable default for a residential shop: zero wrong answers on anything tagged safety, price or warranty, and wrong-plus-should-have-handed-off under 3 in 100 overall. Here 5 wrong plus 3 that should have handed off is 8 of 95, about 8 percent, so it fails.

Unnecessary handoffs (the 6) count against convenience, not against the bar. A handoff costs a callback; a wrong answer costs a customer.

What breaks if you skip the hold-out: a pass that means nothing, and a first live week that does your testing for you on real customers.

Step 5: Fix the entries the failures point at, then retest

Read every failure and trace it to an entry. At the plumbing shop:

• The 5 wrong answers. Four were service area: callers named a neighborhood, and the agent matched it to a same-named street inside the service area. The fix was listing out-of-area places by name in the boundary part of the entry. The fifth said the shop installs tankless units in mobile homes, which it does not; the "do you do this" entry gained an explicit exclusions list.
• The 3 that should have handed off. All three were warranty questions ("is this covered since you were just here?"), and the agent answered from general language. The entry was rewritten to hand off on any mention of a previous visit.

Then retest on fresh calls, not the same 95, since you have now tuned to those. The shop used the next two weeks of live calls with the agent shadowing (listening and drafting, not answering). On 88 calls it got 1 wrong answer, on a price question, and 1 that should have handed off: 2 of 88, about 2 percent, under the 3 in 100 bar overall. But the rule says zero wrong answers on price, so it still did not pass. The price entry was found to be reading an old fee from a cached list; once that was pointed at the live list, the next week was clean, and it went live.

Step 6: Keep it current

The FAQ goes stale the moment prices, service area or hours change, and it also goes stale slowly as caller language drifts. Two habits keep it honest. Any change to a published price, the service area, hours or a policy updates the matching entry the same day, by name, as part of the change. And each month, pull the calls the agent handed off or got wrong and ask whether a new question has entered the top twenty. The weekly review of live calls is where you find them. See related: The Weekly AI Call Review SOP.

References

• Federal Wiretap Act, 18 U.S.C. 2511(2)(d), one-party consent; state recording statutes as enacted (California Penal Code 632 and Florida Statutes 934.03 require all-party consent)
• See related: Setting Up an AI Phone Answering Service and Testing It
• See related: The Weekly AI Call Review SOP
• See related: AI Meeting Notes, Call Recording and Consent
• See related: The Chatbot Promised a Price We Do Not Charge