What a Language Model Is Actually Doing When It Answers

Why this matters

Somebody in your office has already asked a chat assistant a trade question and pasted the answer into a customer email. The answer read well. Whether it was right depended on something nobody in the room could see: what the tool was actually doing when it produced those sentences. Once you know the mechanism, you stop asking "is this tool smart?" and start asking the useful question, which is "is this the kind of answer it can produce reliably, or the kind it produces fluently?"

This card owns that mechanism for the AI shelf. Sibling cards build on it: how to check output, what data may go in, how to write a prompt. They assume what is explained here.

The core move: predict the next piece of text

A language model is a program trained on an enormous amount of written text to do one thing: given some text, predict what text most plausibly comes next. It does not look anything up by default. It does not have a filing cabinet of facts it consults. It has a very large set of learned statistical patterns about which words tend to follow which other words, in which contexts.

When you type a question, the model produces an answer one small chunk at a time. Each chunk is called a token, which is roughly a short word or a piece of a longer one. After each token it asks, in effect, "given everything so far, what is a likely next token?", picks one, adds it, and repeats. A paragraph of answer is a few hundred of those choices in a row.

That is the whole trick, and it is far more capable than it sounds. Predicting the next word well, across billions of examples, forces a model to absorb grammar, tone, the structure of an estimate, how a polite apology is shaped, and a great deal of real factual content, because facts are part of what makes text predictable. "Water boils at 212 degrees Fahrenheit at sea level" is a very predictable sentence.

The catch is in the same sentence. The model learned what text is likely, not what text is true. Most of the time those overlap heavily. Where they do not, the model has no internal alarm that goes off.

Vocabulary you will hear

• Prompt. Everything you send the model: your question, any instructions, any text you pasted in.
• Context or context window. The total amount of text the model can see at once while answering, which includes your prompt, the earlier turns of the conversation, and its own answer so far. Anything outside that window does not exist for it. Window sizes have grown fast and differ by tool; as of 2026 many tools hold the equivalent of a long manual, but older parts of a long conversation can still drop out or get less attention.
• Training data. The text the model learned from, frozen at some date. The model does not know what happened after that date unless a tool feeds it newer text.
• Hallucination. The common name for a confident, fluent answer that is false: an invented part number, a code section that does not exist, a quote nobody said. The word makes it sound like a malfunction. It is not. It is the normal mechanism producing a plausible sentence where no true one was available.
• Grounding or retrieval. A setup where the tool first finds real documents (your SOPs, a manual, a web page) and pastes them into the context before the model answers, so the prediction leans on actual text rather than memory.

Why invention comes out sounding certain

Ask a model "what code section covers dryer vent length?" and look at the job it is doing. It has seen thousands of documents in which a question like that is followed by a section number, a short paraphrase, and a confident tone. The shape of a correct answer is extremely predictable. The specific digits of the section number are much less predictable, because they appear far less often, differ by code edition, and differ by which code a jurisdiction adopted.

So the model produces the predictable shape with total fluency and fills the unpredictable slot with something plausible. If it happened to see the real section many times, the slot fills correctly. If not, the slot fills with digits that look right: the right chapter prefix, a plausible subsection, a clean decimal. Nothing in the output marks which case you got. The confident tone belongs to the shape, not to the digits.

This is why the dangerous outputs cluster in a predictable place: specific, rare, checkable-looking details inside a familiar shape. Part numbers, model numbers, code sections, statute citations, phone numbers, addresses, prices, torque values, dates, and named people. The prose around them is usually fine. The specifics are where the risk concentrates. The full catalog of trade-flavored inventions is a sibling card's job; see related: The Invention Patterns to Expect in Trade Questions.

Two more consequences follow from the mechanism, and both catch people out:

• Asking "are you sure?" does not check anything. The model does not go and re-verify. It predicts what text follows "are you sure?", which is often an apology and a different answer, or a confident restatement. Either can be wrong. A changed answer under pressure tells you the first one was weakly held, not which one is right.
• It does not know what it does not know. A person who is unsure usually sounds unsure. A model's tone is set by the kind of text it is producing, so an invented answer and a correct one arrive in the same voice.

Worked example: one question, followed through

Say your office manager asks a general-purpose chat assistant:

PROMPT:
What is the replacement igniter part number for a
furnace with model number on the label ending in -045?

ANSWER:
For that series, the OEM hot surface igniter is
part 41-2257-03. Universal alternatives include a
round 80V silicon nitride igniter. Always confirm
the model number on the rating plate before ordering.

Walk through what produced each piece.

• "For that series" is a generic opener. The model has no idea what series a model number ending in -045 belongs to; a fragment that short matches dozens of manufacturers.
• "part 41-2257-03" is the slot. It has the format of a real part number: a two-digit prefix, a four-digit body, a dash suffix. That format is highly predictable. The digits are not. Treat this number as invented until a parts catalog shows it.
• "round 80V silicon nitride igniter" blends real knowledge (silicon nitride igniters exist, and some run on lower voltage than line voltage) with a detail that may not match this furnace at all. Ordering a lower-voltage igniter for a unit that expects line voltage is a wrong part at best.
• "Always confirm the model number" is the safety-sounding tail the model has learned to attach. It is good advice and it does nothing to fix the invented number above it.

Now change one input. The manager pastes the full model number and the relevant page of the installation manual into the prompt, then asks the same question. The manual page now sits in the context window, and the most predictable continuation is the part number printed on that page. The answer will very likely be right, because the model is now mostly copying from text it can see.

What changed was not the model getting smarter. What changed was where the predictable answer came from: memory of patterns, versus a document in the window. That is the single most useful thing to understand about these tools.

What would still go wrong even with the page pasted in: if the page covers several model variants and the model picks the wrong row, or the manual is for a different revision of the furnace, the answer is copied faithfully from the wrong place. Grounding moves the risk; it does not remove it.

What the context window does to a long conversation

A model has no memory between conversations unless a tool deliberately saves one. Inside a single conversation, it sees the whole transcript up to the window limit. Two practical effects:

• Early instructions fade. If you said "never mention prices" at the top of a long thread and then pasted in three long documents, that instruction is still technically in view but competes with a lot more text. Restate the important rules near the question.
• It will happily use what you pasted, including what you did not mean it to. Paste a customer's whole email thread to get a reply drafted and the model can see every phone number, address and gate code in it. That is the reason the shop's data rules exist; see related: The Shop AI Use Policy SOP.

Where the mechanism is strong and where it is weak

The prediction engine is genuinely good at tasks where the predictable answer and the correct answer are the same thing, or where you can see at a glance whether it is right:

• Strong: rewording, shortening, changing tone, turning notes into sentences, drafting a structure you then fill, summarizing a document you also pasted in, generating options you will choose between.
• Weak: recalling a specific rare fact from memory, anything that changed recently, arithmetic across many numbers (it predicts digits, and long sums are where predicted digits drift), anything about your specific customer, job site or local code adoption, which it has never seen.

A useful rule of thumb follows: the model is a strong writer and an unreliable witness. Use it to write things you could verify. Do not use it to tell you things you cannot.

What this means for a shop

Four working conclusions, each of which a sibling card develops:

1. Sort every output into "I can see if it is wrong" and "I cannot." A reworded email you can judge by reading. A code section you cannot judge without the code book. The verification standard by output type lives in: The Verify-Before-You-Send Rule for AI Output.
2. Give it the source instead of asking it to remember. Pasting the manual page, the job notes or your own price sheet turns a recall task into a copying task, which is the kind it does well.
3. Treat specifics inside fluent prose as the danger zone. When you review, read the numbers, names and citations first, before the prose lulls you.
4. Never let tone stand in for confidence. "Certainly!" and a crisp section number mean the model is producing an answer-shaped answer, nothing more.

How to tell you understood this

Test yourself on a real output from your own office. Take one answer someone actually used last month and circle every specific: each number, name, citation, date and promise. For each circle, say where the model would have got it: from text that was in the prompt, or from memory of patterns. Everything that came from memory is unverified until a real source confirms it.

If most of the circles came from memory and nobody checked them, the tool was being used as a witness. That is the habit to change.

References

• See related: The Verify-Before-You-Send Rule for AI Output
• See related: The Invention Patterns to Expect in Trade Questions
• See related: The Shop AI Use Policy SOP
• See related: AI Tools for a Small Shop: Hype vs Useful
• The equipment manufacturer's installation and parts documentation for any part number or specification