How LLMs Work
Tokens, training, and inference — the three-part mechanism behind ChatGPT, Copilot, and every AI tool that writes text.
Educational content for accountants learning AI — not technology advice or a substitute for professional judgment. Always verify AI outputs against primary sources before relying on them in client or firm work.
Why This Matters
You already know that generative AI creates new content by predicting "what word comes next." That explanation is correct — but it's also the kind of half-answer that leaves smart, skeptical accountants unsatisfied. How does a system predict the next word well enough to draft a client memo, explain a GAAP standard, or summarize a 40-page lease? And why does that same system sometimes confidently invent a citation that doesn't exist?
This lesson opens the hood. By the end, you'll understand the actual mechanics behind every large language model (LLM) you'll use in this course and in your career: how text becomes numbers, how those numbers get trained into a model, and how that trained model generates a response one piece at a time. This isn't optional technical trivia — it's the foundation for knowing when to trust an AI tool's output and when to double-check it yourself.
Three stages — tokenization, training, and inference — explain fluency, knowledge cutoffs, pricing, and hallucination.
The Big Picture: Three Stages, One Mechanism
Every interaction you have with an LLM involves three distinct stages, only one of which happens while you're actually using it.
STAGE 1: TOKENIZATION
Converting text into numbers the model can process. Happens every single time — both when you type a prompt and when the model generates a response.
STAGE 2: TRAINING
The (very expensive, one-time-per-model) process of teaching the model statistical patterns in language by showing it enormous amounts of text. Happens once, before you ever use the model.
STAGE 3: INFERENCE
The process of actually generating a response to your specific prompt, using the already-trained model. Happens every single time you use the tool.
Training happens ONCE. Inference happens EVERY TIME. This distinction explains a lot of confusing AI behavior — including why an LLM's "knowledge" has a cutoff date, but its ability to respond to your specific question is instant.
Stage 1: Tokenization — How Text Becomes Numbers
Computers cannot process the letters "GAAP" or the word "depreciation" directly. Before an LLM can do anything with your text, it must first be broken into pieces called tokens and converted into numbers.
What Is a Token?
A token is a chunk of text — sometimes a whole word, sometimes a piece of a word, sometimes a single punctuation mark or space.
- Common tokens are entire words: "the" → 1 token; "cat" → 1 token
- Uncommon or complex words get split into sub-word pieces: "depreciation" → might become ["dep", "reci", "ation"]
- A simple rule of thumb: 1 token ≈ 0.75 English words — so a 1,000-word memo is roughly 1,300 tokens
Tokenization Demo
Pick a phrase to see a conceptual split into tokens. Real tokenizers vary by model — this is the idea, not a live GPT tokenizer.
Input
"GAAP"
Conceptual tokens (1)
A common acronym often stays as one token.
Rule of thumb: ~1 token ≈ 0.75 English words — so a 1,000-word memo is roughly 1,300 tokens.
The Algorithm Behind It: Byte-Pair Encoding (BPE)
Most modern LLMs — including the GPT family — use a specific tokenization method called Byte-Pair Encoding.
HOW BYTE-PAIR ENCODING WORKS
Step 1: Start with individual characters. "hello" → ['h', 'e', 'l', 'l', 'o']
Step 2: Find the most frequently occurring pair across the training dataset and merge it. 'l' + 'l' → 'll' → ['h', 'e', 'll', 'o']
Step 3: Repeat thousands of times. Common words become single tokens; rare words break into statistically common pieces.
This is why the same sentence can produce a different number of tokens depending on which model's tokenizer processes it — each model builds its own vocabulary from its own training data.
From Tokens to Numbers: Embeddings
Once text is split into tokens, each token is assigned a unique ID number. But a raw ID alone doesn't capture meaning — the model needs to understand that "invoice" and "bill" are related. That's the job of embeddings.
Each token is converted into a long list of numbers (a "vector") — often hundreds or thousands of numbers long — learned during training so that tokens used in similar contexts end up with similar number patterns.
That's why an LLM can recognize that "accounts receivable" and "money owed by customers" are related — their embeddings end up numerically close, even though they share no tokens in common.
Stage 2: Training — Teaching the Model Language Patterns
Training is where an LLM actually "learns." It happens once, requires enormous computing resources, and produces the reusable model that powers every future conversation. Modern LLM training happens in distinct phases.
Phase 1: Pretraining
The model is shown a staggering amount of text — a significant fraction of the publicly available internet, books, articles, and code — and given one simple, repeated task: predict the next token, given everything that came before it.
Example training instance:
Input: "The current ratio is calculated by dividing current assets by current"
Correct next token: "liabilities"
If the guess is wrong, backpropagation adjusts millions (or billions) of internal parameters so the model is a tiny bit more likely to guess correctly next time. Repeated trillions of times — there is no separate "understanding module." Everything emerges from getting extremely good at this one narrow task.
Phase 2: Supervised Fine-Tuning (SFT)
After pretraining, the raw model is good at completing text, but not necessarily good at being a helpful assistant. Fine-tuning trains it further on a smaller, curated dataset of example conversations — showing it what a well-formed, helpful answer looks like, rather than just "the statistically likely next words on the internet."
Phase 3: Reinforcement Learning from Human Feedback (RLHF)
Human reviewers rate multiple candidate responses to the same prompt from best to worst. The model is further adjusted to produce more responses like the highly-rated ones and fewer like the poorly-rated ones.
This phase is largely responsible for making modern chatbots feel polite, cautious about harmful requests, and inclined to admit uncertainty — behaviors that emerge from human preference, not from the base pretraining data alone.
Why this matters for accountants: The model's "knowledge" is frozen at the point pretraining data was collected — often called the "knowledge cutoff." An LLM trained on data through a certain date has no built-in awareness of a new accounting standard issued after that date, unless it's explicitly told about it in your prompt or given access to a live search tool.
Stage 3: Inference — Generating a Response, One Token at a Time
Inference is what happens every time you actually use an LLM — type a prompt into ChatGPT, and inference is the process that produces the reply. Unlike training, inference happens in a fraction of a second and uses the model exactly as it was left after training, without changing it.
THE INFERENCE LOOP (AUTOREGRESSIVE GENERATION)
- Your prompt is tokenized into numbers.
- The model calculates a probability score for every possible next token in its entire vocabulary — tens of thousands of candidates — given everything in the conversation so far.
- The model selects one token, usually the highest-probability one (with some controlled randomness).
- That selected token is appended to the sequence, and the entire process repeats — predicting the next token given the original prompt plus what it just generated.
- This continues until the model generates a special "stop" token or reaches a length limit.
Every single word you see appear during a ChatGPT response is a separate inference step, each one building on everything generated so far.
The Attention Mechanism: How the Model Decides What Matters
The technical breakthrough that makes modern LLMs possible is called self-attention, introduced in a landmark 2017 paper. At each step of generation, the model doesn't treat every previous word equally — it calculates which earlier words in the conversation are most relevant to predicting the current next word.
Consider generating the next word in: "ABC Coffee Shop's revenue increased because the new espresso machine allowed the barista to serve customers faster, which…"
To correctly predict the next word, the model needs to pay close attention to "revenue increased" and "faster" — and can mostly ignore less relevant words like "the" or "which." Self-attention lets the model dynamically weigh every previous token — which is what allows coherent responses across long documents rather than losing track after a few sentences.
Temperature and Randomness
If the model always picked the single highest-probability token, every response to the same prompt would be word-for-word identical. In practice, most LLM tools introduce controlled randomness governed by a setting called temperature.
Low temperature (closer to 0)
Almost always picks the highest-probability token. Output is more focused, predictable, and repetitive across attempts.
High temperature (closer to 1+)
More often selects lower-probability tokens. Output is more varied and creative — but also more prone to going off-track.
Professional implication: For precision and consistency — numerical explanations, contract summaries — lower temperature (where available) produces more reliable output. For brainstorming, higher temperature can help. Most consumer chat interfaces use a moderate, fixed temperature you don't directly control.
Putting It All Together: A Complete Walkthrough
Let's trace a real accounting prompt through all three stages.
YOUR PROMPT
"Explain the matching principle in one sentence."
STAGE 1 — TOKENIZATION
Text ≈ ["Explain", " the", " matching", " principle", " in", " one", " sentence", "."] → numeric IDs like [4531, 279, 12847, 8621, 304, 832, 11914, 13]
STAGE 2 — ALREADY COMPLETE
Training happened months or years ago. Internal parameters already encode statistical relationships from millions of examples discussing the matching principle, revenue recognition, and expense timing.
STAGE 3 — INFERENCE (NOW)
Probabilities for the first response token might look like: "The" 34%, "Matching" 22%, "This" 9%. It selects "The," then predicts the next token given the prompt plus "The," and repeats until a complete sentence is generated.
"The matching principle requires expenses to be recorded in the same period as the revenue they helped generate."
Why This Explains Common LLM Behavior
Understanding tokens, training, and inference resolves a lot of AI behavior that otherwise seems mysterious or inconsistent.
Why LLMs struggle with math and letter-counting
The model doesn't "calculate" the way a calculator does — it predicts tokens based on patterns in training text. It has seen "2 + 2 = 4" thousands of times, so it reliably predicts "4." For an unusual calculation without similar examples, it may confidently predict an incorrect sequence.
Why LLMs have a "knowledge cutoff"
Training happens once, using data collected up to a certain date. Nothing after that date exists in the model's learned parameters unless it's provided in your prompt or through a connected search tool.
Why longer conversations cost more and can lose coherence
Every token — in your prompt and in the growing conversation history — must be reprocessed by the attention mechanism at every generation step. Longer chats need more computation and can exceed a model's maximum "context window," at which point earlier parts get dropped.
Why the same question can get slightly different answers
Controlled randomness (temperature) in token selection means the model doesn't always pick the exact same highest-probability path every time.
Why hallucination happens at all
At every step, the model selects the statistically most probable next token — not verifying a fact against a database. If a plausible-sounding but false accounting standard number is statistically likely to follow, the model can generate it with the same fluent confidence as a true statement. There is no internal "truth checker" separate from token prediction.
Tokens Have Real Business Consequences
For any accountant evaluating or budgeting for AI tools at their firm, tokens aren't just a technical detail — they're the literal unit of billing.
Token-Based Pricing
- Input tokens — the text you send (prompt + documents or context)
- Output tokens — the text the model generates back (often priced higher)
Pasting an entire 60-page financial statement to ask one simple question costs significantly more than asking about a targeted excerpt.
Context Window Limits
Every model has a maximum combined token limit (input + output) — the context window. Exceed it and the tool truncates content or rejects the request.
Before pasting an entire general ledger export or lengthy client contract, check the tool's context window. Large documents may need to be summarized or split — and you should verify nothing important got silently cut off.
Key Takeaway
Every large language model operates through three distinct stages. Tokenization breaks your text into sub-word chunks (tokens) and converts them into numbers, using algorithms like Byte-Pair Encoding, because representing every whole word individually would require an impossibly large vocabulary. Training — pretraining, supervised fine-tuning, and reinforcement learning from human feedback — happens once, teaching the model statistical language patterns by having it repeatedly predict the next token across trillions of examples. Inference happens every time you use the tool: the model generates a response one token at a time, using the self-attention mechanism to weigh which earlier words matter most, with controlled randomness (temperature) determining how predictable or varied the output is. This single mechanism — predicting the next most probable token, over and over — is the entire engine behind both the impressive fluency of LLMs and their tendency to hallucinate, since the model has no separate fact-checking process distinct from token prediction itself.
Test Your Understanding
Knowledge cutoffs, sub-word tokens, and token-based pricing — check your answers below.
Question 1: An LLM is asked a question about an accounting standard issued last month. It confidently gives an answer that turns out to be about an older, different standard. The most likely explanation is:
Question 2: Why do LLMs break text into sub-word tokens (like "token" + "ization") rather than treating every whole word as a single unit?
Question 3: A firm notices that pasting an entire 80-page contract into an AI chat tool to ask a single question is significantly more expensive than asking about a short, relevant excerpt. This is best explained by:
Ready to Practice?
Apply accounting fundamentals in the Practice Lab while you build the AI fluency this course develops — judgment first, tools second.
Try the Practice LabWhat's Next?
Neural Networks Explained — Now that you understand the token-training-inference pipeline at the system level, the next lesson goes one layer deeper into the actual computational structure — neural networks — that makes all of this pattern-learning possible in the first place.
Neural Networks Explained
The computational structure behind pattern learning
AI in Accounting Hub
Browse all AI pillar topics