How to Check Whether Training Actually Stuck

Why this matters

The end of a training session is the moment you have the least information you will ever have about whether it worked. Everyone nods, the demonstration went well, the teach-back came back clean, and all of that is consistent with the skill evaporating within three weeks.

That gap costs real money in a quiet way. The shop believes a skill is covered, dispatch routes on that belief, and the first evidence to the contrary is a callback on a job that was assigned to someone who was signed off on paper. Checking retention is not an academic exercise or a quality-department ritual, it is how you find out whether your dispatch assumptions are true, and it takes far less time than the training did.

Step 1: Decide what "stuck" means for this skill before you teach it

Write the check before you write the lesson. If you cannot state what you will observe in four weeks that will convince you, you have not defined the skill tightly enough to teach it either.

Three questions, answered in one line each:

• What behavior would I see? Not "understands," but "performs the isolation before touching anything, without prompting, every time."
• What number would move? Callbacks on that work type, average duration, documentation completeness, parts returned unused. Pick one that already exists in your records so you do not create a new tracking chore.
• What would prove it did not stick? Name the specific failure now, while you are unattached to the outcome. Deciding after the fact what counts as failure is how everything gets scored as a success.

Step 2: Check on the same day, and then do not trust that check

An immediate check is still worth doing, because it catches outright misunderstanding while it is cheap to fix. Have them explain the reasoning back in their own words and watch for the paraphrase that reveals a wrong model.

But treat a same-day pass as evidence of one thing only: they were paying attention. It is not evidence of retention, and the two feel identical to the person doing the checking. Recall decays fastest in the first days after learning, and what is left after a few weeks is a fraction of what was demonstrable on the day. That is the normal shape of learning, not a sign anyone did anything wrong. It is only a problem when the same-day pass is treated as the final answer and no later check is scheduled.

Schedule the real check before you leave the session. On the calendar, with a date.

Step 3: Run the cold check at two to six weeks

Cold means unannounced in content, though not in fact: tell them a check is coming, do not tell them which skill or when. Ambush checks produce accurate data and a crew that stops trusting you, which is a bad trade.

The cold check has one rule that makes it useful: no props, no notes, no prompting. Ask them to walk through the task on a real job or, if the job type has not come up, on the equipment in the yard. The moment you say "and then what about the isolation step" you have supplied the answer and learned nothing.

Two to six weeks is the window where a real gap has emerged and a fix is still cheap. Earlier and you are re-measuring short-term recall. Later and the habit is set, correctly or otherwise, and unwinding a wrong habit takes several times the effort of correcting it early.

Step 4: Read the field evidence instead of relying on checks

Checks are self-report and observation, and both are distorted by your presence. The stronger evidence is already sitting in your job records, and it costs nothing to read:

That last row deserves attention. The questions a tech asks change shape as a skill lands. Early questions are procedural ("what do I do next"). Later questions are conditional ("this reads high but the equipment is oversized, does that change what I should conclude"). A tech still asking procedural questions six weeks on has memorized steps and not built a model, and no amount of repeating the steps will fix that.

Step 5: Test transfer, not repetition

The strongest check is the one that changes a variable. Have them perform the skill on different equipment, in a different configuration, or with one condition altered, and ask what changes about their approach.

A tech who learned the procedure will run the same steps and get the wrong answer confidently. A tech who learned the skill will pause at the changed variable and say why it matters. That pause is the thing you are checking for, and it is worth more than a flawless repetition of the original case.

Where the transfer test touches a hazard - gas, water in combination with electricity, stored energy, height - the check begins with the safety step and stops there if it fails. If the learner does not isolate and verify before proceeding on the unfamiliar configuration, the check is over and the answer is no, regardless of how well the rest would have gone. Safety behavior that only appears on familiar equipment has not stuck at all.

Step 6: Diagnose which layer failed before you re-teach

When something did not stick, resist repeating the original session. Repeating a lesson that failed usually fails the same way. Find the layer:

• Never landed. They could not do it at the end of the session either, and it was recorded as a pass out of politeness. Fix the sign-off bar, not the lesson.
• Landed and decayed from disuse. They were competent and have not touched the work in weeks. Fix dispatch, not training. This is the most common cause and the most misdiagnosed.
• Landed and was overwritten. They saw a different tech do it differently and adopted that. Fix the inconsistency between your seniors, which is a bigger problem than the one learner.
• Landed as steps, not reasoning. They can do the exact case and nothing adjacent. Re-teach at the level of why, using cases rather than repetition.
• Blocked by conditions. They know it and the job does not allow it: no time, no tool, a customer standing over them. Fix the condition. Training cannot beat a missing tool.

A worked example: three techs, one maintenance sequence, ninety days

A shop teaches a standardized maintenance sequence to three techs, replacing the version where each of them did roughly their own thing.

Same-day check: all three explain the sequence back correctly. On the strength of that, everyone would have called this a success and moved on.

Week four cold check: two of three walk the full sequence unprompted. The third skips two steps and, when asked, describes the steps as optional. Note what this means: the same-day check had no discriminating power at all. It passed the person who did not retain it.

Field evidence at ninety days, compared to the ninety days before the training:

• Callbacks on that work type: 4 out of 60 calls before, which is 6.7%, against 2 out of 66 calls after, which is 3.0%. That is a relative reduction of about 55%.
• Average duration: 1.20 hours before, 1.35 hours after, about 12.5% longer.
• Readings recorded on the ticket: 41% of jobs before, 88% after.

Read the duration number carefully, because it is the one most likely to be misinterpreted by whoever reviews it. Jobs got 12.5% longer and that is evidence the training worked, not evidence it cost you. The techs are now performing steps they previously skipped, and the callback rate roughly halved as a result. A manager reading duration alone would conclude the training made the crew slower and would be exactly wrong. Always read a productivity metric next to a quality metric or you will optimize your way back into the callbacks.

The distribution matters more than the average. Both remaining callbacks came from the third tech, the one who failed the week-four check. So the crew-level number, a 55% reduction, is not the story. Two of three techs went to zero callbacks on that work type, and one did not change at all.

The correct action is therefore not to re-run the training for everyone, which would waste two techs' time and imply their improvement was not noticed. It is one targeted session with the third tech, starting from the diagnosis in step 6. In this case the tech had described the skipped steps as optional, which points at "landed as steps, not reasoning": nobody had explained what those two steps prevent, so they read as padding. The re-teach is one call spent on what those steps catch and what the failure looks like when they are skipped, not a repeat of the sequence.

What changes the answer

• Low-volume work types. If the skill only comes up twice a month, ninety days of field evidence is a handful of jobs and the callback rate is statistical noise. Lean on the cold check and the transfer test instead, and extend the evidence window to six months.
• A skill taught to one person. No distribution to read. The cold check carries the whole load, so make it a transfer test rather than a repetition.
• Safety-critical behavior. Do not wait two to six weeks for the first real check, and do not rely on outcome numbers, because the outcome you are trying to avoid is one you cannot afford to observe even once. Check by direct observation, early and repeatedly.
• A crew-wide process change rather than an individual skill. Read adoption before you read outcomes. If only half the crew is doing it, the outcome numbers are measuring the mix, not the method.

How to keep the checking honest

Check that your checks discriminate. If every check comes back a pass, the check is not measuring anything. A check that has never once produced a fail should be redesigned.

Do not teach to the check. If the crew knows the cold check is always the same three questions, you are measuring whether they remember the three questions. Vary the case, keep the standard fixed.

Separate the roles when you can. The person who taught it has an interest in it having worked. Where you have a second qualified person, let them run the cold check. In a shop where that is not possible, at minimum write down the pass criteria before the check so you cannot quietly move the bar.

Recheck at six months. Retention is not permanent. A skill signed off and then unused is a skill you no longer have, and the recheck is one supervised instance against the alternative of finding out during the emergency.

References

• See related: The Teach-Back Method: Confirming They Actually Got It, The Competency Sign-Off SOP
• See related: The Ride-Along Debrief SOP, How to Teach a Skill in the Field Without Losing the Day
• OSHA guidance on retraining when an employee's performance indicates that required knowledge or skill has not been retained
• Trade-standard quality practice: reading callback and rework rates as a lagging indicator of field competence