Setting Up an Assistant That Answers From Your Own SOPs

Why this matters

A shop that has written its procedures down still has a retrieval problem: the new tech in a crawlspace will not scroll through forty documents to find the step for re-lighting a pilot, so he calls the office or guesses. An assistant loaded with the shop's own SOPs can answer that question in seconds, in the shop's words. Set up carelessly, it answers in a blend of the shop's words and the internet's, with no way for the tech to tell which parts are which. The difference between the two setups is not the tool. It is which documents go in, what the assistant is told to do when they do not cover a question, and whether anyone tests it on questions whose right answers are already known.

What this kind of assistant is, in plain terms

Most assistants that "answer from your documents" work by retrieval: when a question comes in, the system searches your loaded documents for the passages most related to it, hands those passages to the language model along with the question, and the model writes an answer from them. The model still knows everything it learned in general training. Nothing stops it from mixing that general knowledge into the answer, or from answering entirely from general knowledge when the search finds nothing relevant, unless it is instructed not to and that instruction is tested.

That one fact shapes every step below. The documents decide what the right answer is; the instruction decides whether the assistant stays inside them; the test tells you whether it does. How the model generates text at all, and why it sounds equally certain inside or outside your documents, is its own card (see related: What a Language Model Is Actually Doing When It Answers).

Step 1: Load only current, approved procedures, one version of each

Collect the SOPs the shop actually runs today. Leave out drafts, superseded versions, old training handouts, and anything nobody has approved. Put the version date in each document's title.

Why it matters: retrieval finds the passage most similar to the question, not the most current one. If last year's gas leak procedure and this year's are both loaded, the assistant may quote either, and an older version that told the tech to do something your current one forbids is exactly the passage that looks most relevant.

What breaks if skipped: the assistant confidently quotes a procedure you retired, with a correct-looking citation.

What may go in at all is set by the shop's AI use policy (see related: The Shop AI Use Policy SOP). Procedures usually qualify; customer lists, pricing files and personnel records usually do not, and an SOP that has a customer's name in a worked example should be cleaned first.

Step 2: Make each document easy to retrieve correctly

One procedure per document. A clear title that says what it is for ("Gas odor on arrival: response procedure," not "Safety 3"). Numbered steps. Text, not a scanned photo of a printout, since many systems cannot read images reliably. Section headings inside long documents.

Why it matters: retrieval works on chunks of text. A forty-page "Field Manual" with every procedure in it gets chopped into pieces, and a question about lockout may pull a chunk from the lockout section and a chunk from the confined-space section that happens to use the same words. The model then writes one answer from both.

What breaks if skipped: answers that blend two procedures into a sequence neither one contains.

Step 3: Write the instruction, and make "not in our procedures" an answer

Most of these systems let you set a standing instruction that applies to every question. The instruction that works has four parts:

• Answer only from the loaded shop documents.
• Cite the document title and step number for every instruction.
• For any safety step (anything involving gas, electrical isolation, lockout, confined spaces, chemicals, heights, or carbon monoxide), quote the procedure's wording exactly rather than paraphrasing.
• If the documents do not cover the question, say "This is not in our procedures. Ask your lead," and do not answer from general knowledge.

Why it matters: without the last line, the assistant fills the gap with the typical answer, which is the exact behavior you set this up to avoid. Without the citation line, a tech cannot tell an answer from your documents from an answer the model made up.

What breaks if skipped: the tech gets general internet advice formatted to look like shop procedure.

Step 4: Build a test set before anyone uses it

Write 25 questions whose correct answers you already know:

• 20 in scope, answered somewhere in the loaded documents. Include at least 4 safety questions whose answers must be quoted exactly.
• 5 out of scope, deliberately not covered by any document, but plausible questions a tech might ask ("what's the torque spec on this brand's burner orifice," "can I use this sealant on a gas thread").

For each, write down the correct answer, the document and step it comes from, and for the out-of-scope ones, that the correct answer is "not in our procedures."

Set the pass gate before you run anything, so the result cannot talk you into lowering it: at least 18 of the 20 in-scope questions answered correctly with the right citation (90 percent), all 4 safety questions quoted exactly, and all 5 out-of-scope questions declined. The out-of-scope gate is 5 of 5 because an assistant that answers outside its documents even occasionally will be trusted every time.

What breaks if skipped: you discover the assistant's failure modes from a tech in the field.

Worked example: two test runs

A ten-person plumbing and heating shop loads 34 SOPs and runs the 25-question test.

Run 1.

16 of 20 is 80 percent against the 90 percent gate; 3 of 4 safety questions against a 4 of 4 gate; 3 of 5 declined against a 5 of 5 gate. Run 1 fails all three gates.

Reading the failures, one at a time.

• The outdated citation came from two versions of the water heater replacement SOP, both loaded. The fix was step 1: remove the old one.
• The blended answer was a safety question about locking out a circulator pump. The assistant combined step 3 of the lockout SOP with a step from the boiler annual-service SOP, which mentions closing isolation valves. Both documents used similar language. The fix was step 2: the boiler SOP had its isolation steps split into their own clearly titled document, and the lockout SOP was retitled to make its scope obvious.
• The two uncited answers were correct but gave the tech nothing to check. The fix was step 3: the instruction was reworded to require a citation "for every instruction, without exception."
• The two out-of-scope questions it answered were the sealant question and a question about a refrigerant pressure. It answered both from general knowledge, fluently. The fix was step 3 again: the "not in our procedures" line was moved to the top of the instruction and made explicit that general knowledge is not allowed.

Run 2, after the fixes, same 25 questions:

19 of 20 is 95 percent, which clears the 90 percent gate. 4 of 4 and 5 of 5 clear theirs. The one uncited answer stays on the watch list and gets checked again in the next run.

What would have changed the decision. If the safety questions had still failed after the document cleanup, the shop would not have turned the assistant on for field use at all; it would have kept it for office staff answering admin questions, where a wrong answer costs time rather than a hazard. If the out-of-scope gate had kept failing after the instruction rewrite, the system itself would have been the problem, not the setup, and the shop would have looked for one that can be restricted more tightly.

Step 5: Tell techs exactly what an answer is and is not

Before rollout, say three things at a huddle. The assistant answers from shop procedures and cites them. For safety steps, the posted procedure is still the procedure; the assistant is a way to find it faster, so open the cited document when the step matters. And "not in our procedures" means call your lead, not search the internet.

What breaks if skipped: techs treat every answer as equally authoritative, including the rare one that slips.

Step 6: Keep documents and assistant in step

Every SOP has an owner. When an SOP changes, the owner replaces the loaded version the same day and deletes the old one, then re-runs the test questions that touch that SOP. Every quarter, re-run all 25 and add 5 new questions drawn from real questions techs asked. The underlying model also changes when the vendor updates it, and behavior changes with it, which is a second reason the quarterly run is not optional.

What breaks if skipped: the assistant drifts out of date silently, and it does not get less confident as it does.

Step 7: Treat every answer outside the documents as a defect

When a tech reports an answer that did not come from a shop document, or an answer with no citation, log it with the question asked and the answer given. Then decide which of two things it was:

• A gap in the procedures. Techs keep asking a question you never wrote down. Write the SOP, load it, add the question to the test set. Drafting procedures from how techs actually do the work is its own method (see related: Drafting an SOP From a Recorded Tech Walkthrough).
• A gap in the setup. The answer was in the documents and the assistant ignored or blended them. Fix the document structure or the instruction, and re-run the test.

If the same kind of failure keeps coming back after fixes, the shop's question is no longer how to tune the assistant but whether to keep it on (see related: When to Turn an AI Feature Off).

References

• See related: The Shop AI Use Policy SOP
• See related: What a Language Model Is Actually Doing When It Answers
• See related: Drafting an SOP From a Recorded Tech Walkthrough
• See related: Turning a Trusted Procedure Into a Checklist With AI
• See related: When to Turn an AI Feature Off