When to Turn an AI Feature Off

Why this matters

Shops are good at turning AI features on. A vendor adds a button, someone tries it, it works on the first ten jobs, and it becomes part of the day. What shops almost never do is decide to turn one off. Instead the feature degrades, the office starts working around it, and six months later it is still running on one path nobody watches, still billing, still holding customer data, and still occasionally sending something nobody reviewed. That is the expensive way for a tool to die.

This card covers the signals that a feature has stopped earning its place, the two cases where it comes off the same day, and how to retire one as a planned event rather than a quiet abandonment. How to measure the time a tool saves is its own discipline. See related: Measuring Whether an AI Tool Actually Saved Time.

Four signals, and what each one is really telling you

1. The correction rate is climbing. Count how often a person has to fix the tool's output before it is used: per 100 drafts, per 100 calls, per 100 bookings. Compare against the feature's own first month, not against a vendor claim. A rising rate has three usual causes, and they need different responses. The vendor updated the underlying model and its behavior shifted. Your inputs changed (a new price list, a new service area, a new job type the tool has never seen). Or, the most uncomfortable one, the rate was always this high and review is finally catching what used to slip through. Find out which before deciding anything.

2. Customers are complaining about it, by name or by effect. A customer who writes "your text said Tuesday" is complaining about the feature even if they never say "bot". Tag every complaint that traces back to an automated output. A residential shop hears from a small fraction of the customers who were bothered, so treat each one as a sample, not an isolated event.

3. The review burden has caught up with the savings. Every AI feature has a cost that the demo never shows: the time a person spends checking and fixing what it produced. When the checking plus the fixing equals or exceeds the time the tool saves, the tool is running at a loss even though it feels busy and modern. This is the signal owners most often miss, because the savings are visible in one person's day and the review cost is spread across three.

4. Staff are routing around it. The quietest signal and often the most honest. The dispatcher retypes the AI's schedule from scratch. The office manager switches the after-hours replies off by hand on busy nights. A tech stops using the voice-to-notes feature and goes back to typing. Usage logs show it: the feature's use falls while the workload it serves does not. People rarely announce that they have stopped trusting a tool. They just stop using it, and they usually have a reason that a meeting would surface in five minutes.

The defaults that trigger a review

Signals only help if somebody looks at them on a schedule and knows what number means "now". These are a reasonable starting point for a small shop; tune them to your volume, but set them before the feature goes live, not after it disappoints you.

Any one trigger opens a review, not an automatic shutdown. The review is a short meeting with whoever uses the feature, the person who owns it, and the numbers, and it has three possible outcomes: fix it (a setting, a prompt, an input), restrict it (keep it on the paths where it works, take it off the ones where it does not), or retire it. Write the outcome down with the date.

The two cases that come off the same day

Two situations skip the review and go straight to off, then get investigated:

• The feature made a commitment the shop does not make. A price, a warranty term, an appointment time, a promise about a refund. A booking agent that confirms appointments in a slot nobody can reach, or a chat assistant that quotes a price list you retired, is creating obligations in your name with every conversation. Stopping it costs a few hours of manual work; leaving it on while you investigate multiplies the number of customers you have to call back. See related: The Chatbot Promised a Price We Do Not Charge.
• The feature exposed data it should not have. A summary emailed to the wrong customer, one customer's address appearing in another customer's text, a recording shared outside the shop. Turn it off, preserve what happened, and treat it as a possible breach until you know otherwise, because state breach-notification duties can start running from discovery.

In both cases, turning it off means the actual off switch, on every path. A chat widget removed from the home page but still live on the booking page is still on.

One set of triggers, two features: worked example

A nine-person HVAC shop runs two AI features and applies the triggers above to both at its monthly check.

Feature A: invoice description cleanup. Techs' rough notes are rewritten into customer-readable invoice lines, and the office reviews each before sending.

• Correction rate: 8 per 100 invoices in the first month, 11 per 100 in month four. 11 against 8 is up about 37 percent, below the 50 percent trigger.
• Complaints: none traced to it in 30 days.
• Net time: about 600 invoices a month. Each saves roughly 1.5 minutes of writing (900 minutes, or 15 hours) and takes about 0.5 minutes of review (300 minutes, or 5 hours). Net, about 10 hours a month saved.
• Usage: flat with invoice volume.

No trigger fires. But the rising correction rate is worth a look before it crosses: the office finds most of the new corrections are on a heat-pump line the shop added in month three, which the tool keeps describing in furnace language. The fix is a short instruction added to the prompt template and two example descriptions. Outcome recorded: keep, fixed.

Feature B: automatic replies to after-hours texts. The tool answers customer texts overnight and offers a callback or a booking.

• Correction rate: 5 replies per 100 needed a morning follow-up in month one, 12 per 100 in each of the last two weekly checks. 12 against 5 is up 140 percent, over the trigger on two consecutive checks.
• Complaints: 3 in 30 days, two about a callback promised for a time the office was closed.
• Usage: the office manager switched it off by hand on 9 of 20 working nights, the busy ones, which is exactly when it was supposed to help.
• Net time: on the 11 nights it ran, it saved about 20 minutes of on-call answering each night, 220 minutes in all, and generated follow-up corrections averaging 25 minutes each next morning, 275 minutes in all. Net, 55 minutes lost for the month.

Three of four triggers fire, and the complaints point at a commitment problem (callback times). The review finds the vendor changed how the tool reads business hours, and the tool now offers callbacks on any hour of the next day. The shop could restrict it to "message received, we will reply by 9 tomorrow" with no booking or times at all. It tries that for two weeks; the correction rate falls back to 4 per 100 and complaints stop. Outcome recorded: restricted, not retired.

What would have changed the answer. If the restricted version had still drawn complaints, or if the vendor could not stop the tool from offering times, Feature B would have been retired. And if one of those complaints had been a customer who skipped calling an emergency line because the text promised a morning callback during a no-heat call in a freeze, that is a commitment the shop cannot afford, and the feature comes off the same day under the rule above.

The sunset, run as a planned event

When the answer is retire, do it in this order, with one named owner and a date:

1. Pick the off date and announce it internally. Everyone who touched the feature knows the date and what replaces it.
2. Stand up the replacement first. A human callback list, a different greeting, a plain form. Turning a feature off before its replacement is ready moves its failures onto customers.
3. Change everything customer-facing that mentions it. Phone greeting, website text, the privacy notice if it named the tool. See related: The Customer Privacy Promise When You Use AI.
4. Get your data back and ask for deletion. Export what you need (call logs, transcripts that belong in job records), then ask the vendor in writing to delete the rest and keep their reply.
5. Cancel the billing and remove the access. Revoke accounts and any connection to your field software. A dormant integration still holding a key to your customer list is a liability with no benefit.
6. Write down why, and what would bring it back. One paragraph: the triggers that fired, what was tried, and the condition under which the shop would try again (a vendor change, a different price, a new use).

That last paragraph is what saves the shop from relearning the same lesson in a year when a new person finds the button and turns it back on.

What a quiet abandonment leaves behind

The difference between retiring a feature and abandoning it shows up later, in specific places: a monthly charge for a tool nobody uses; customer data sitting with a vendor the shop no longer has any relationship with; a website path still answering in the shop's name; customers who learned to expect an instant reply and now get silence; and, worst, no record of why it was dropped, so the next decision about the same kind of tool starts from nothing. Every one of those is avoided by the six steps above, and none is avoided by simply not using the feature anymore. Changes to automated features should run through the shop's change control. See related: The Automation Change Control SOP.

References

• See related: Measuring Whether an AI Tool Actually Saved Time
• See related: The Automation Change Control SOP
• See related: The Chatbot Promised a Price We Do Not Charge
• See related: The Customer Privacy Promise When You Use AI