Neural Networks Explained
Neurons, weights, and the "guess, check, adjust" loop that powers every modern AI system
Educational content for accountants learning AI — not technology advice or a substitute for professional judgment. Always verify AI outputs against primary sources before relying on them in client or firm work.
Why This Matters
In the last lesson, you learned that LLMs learn by repeatedly predicting the next token and adjusting internal parameters when they're wrong. This lesson answers the obvious follow-up: adjusting what, exactly? What is the actual computational structure underneath every neural network, LLM, fraud detector, and image classifier — and how does "adjusting parameters" really work, mechanically?
ABC Coffee Shop: The bookkeeper watches her expense-categorization AI mislabel a $340 charge from "Blue Bottle Roasters" as "Travel" instead of "Cost of Goods Sold." A week later, after she corrects it a few times, the system starts getting it right. Something inside that system changed. This lesson explains exactly what changed, and how.
This is the deepest, most mathematical lesson in the AI foundations series — and also the one that makes everything else click into place. The math stays conceptual: enough to supervise AI tools, not enough to become a data scientist.
The Building Block: A Single Artificial Neuron
Every neural network — no matter how large or sophisticated — is built from a huge number of extremely simple units called neurons (or "nodes"), loosely inspired by how biological brain cells connect and fire.
WHAT A SINGLE NEURON DOES:
STEP 1: Receive one or more numeric inputs.
Example: a spam-detecting neuron might receive "how many times does 'free' appear" and "how many exclamation points are in the subject."
STEP 2: Multiply each input by a WEIGHT — how important that input is.
STEP 3: Add a BIAS — a constant that shifts the result up or down.
STEP 4: Pass the total through an ACTIVATION FUNCTION, which decides how strongly the neuron "fires."
The formula (conceptually)
Output = Activation( (Input₁ × Weight₁) + (Input₂ × Weight₂) + … + Bias )
Weights and biases are the ONLY things a neural network actually "learns." Everything else — the structure, the math, the activation function — is fixed by the people who designed the network. Learning means finding the right numbers for every weight and bias in the entire network.
The key insight for accountants: A neural network with millions or billions of parameters is really just millions or billions of these simple weight-and-bias calculations, wired together. There's no hidden "understanding module" — just an enormous number of very simple arithmetic operations, tuned through training to produce useful outputs.
Stacking Neurons Into Layers
A single neuron can only capture a very simple pattern. Real capability comes from stacking many neurons into layers, and stacking many layers into a full network.
Input Layer
Receives the raw data — pixel values in an image, numeric features of a transaction, or (for an LLM) the numeric embeddings of each token. No calculation happens here — it's just the entry point.
Hidden Layer(s)
Where the actual pattern-recognition happens. Each neuron takes the outputs of every neuron in the previous layer as its inputs, applies its own weights, bias, and activation function, and passes its result to the next layer.
A network with many hidden layers is called a "deep" neural network — this is where the term "deep learning" comes from.
Output Layer
Produces the final result — a probability that a transaction is fraudulent, a predicted category, or (for an LLM) a probability score for every possible next token.
Visualizing a Simple Fraud-Detection Network
INPUT → HIDDEN → OUTPUT
Input layer
- Transaction amount
- Time of day
- Merchant category
- Distance from home
- Account age
- Frequency pattern
Each weighs inputs differently
Fraud probability
(0% to 100%)
Each arrow is a weighted connection. One hidden neuron might learn to weight "distance from home" heavily; another weights "time of day." Together they capture combinations of signals that predict fraud far better than any single rule could.
Why Activation Functions Are Non-Negotiable
It's tempting to think a neural network is "just" a big pile of weighted sums. Without one critical ingredient, it actually would be — and it would be nearly useless. That ingredient is the activation function.
The Linear Trap
If you stack layer after layer of neurons that only do weighted sums (no activation function), the entire stack — no matter how many layers — mathematically collapses into the equivalent of a single layer doing simple linear regression. All that depth would be wasted. The network could only ever learn straight-line relationships — hopeless for messy, non-linear patterns in fraud, language, or images.
Activation functions introduce non-linearity — a deliberate "bend" in the math — after each neuron's weighted sum. That bend is what lets a deep network model genuinely complex relationships, not just straight lines.
Sigmoid
Squeezes any input into a smooth range between 0 and 1 — useful for outputs that represent probabilities (like "38% chance this is fraud").
Historically important, but suffers from the "vanishing gradient" problem in very deep networks.
ReLU (Rectified Linear Unit)
Simple rule: if the input is positive, pass it through; if negative, output zero. Extremely fast — and a major reason the deep learning boom of the last decade became possible.
Think of it as a "switch": fire or stay silent.
GELU / Swish
More refined, smoother variants used in modern transformer-based LLMs like GPT — designed to handle billions of parameters more gracefully.
You'll see these names in model papers; the idea is the same bend, refined.
How a Network "Learns": The Complete Training Loop
This is the heart of the lesson — the exact mechanism by which a network improves from a random guesser into something genuinely useful. The process has four repeating steps, and it applies identically whether you're training a fraud detector or a large language model.
The Training Loop — Visual Map
Guess → measure → trace blame → adjust. Repeat thousands or millions of times. That's deep learning at its core.
Forward Pass
Make a guess
Data flows input → hidden → output using current weights (initially random).
Loss Function
Measure the miss
Compare the guess to the known correct answer. Low loss = good; high loss = bad.
Backpropagation
Trace the blame
Work backward: calculate how much each weight contributed to the error.
Gradient Descent
Nudge the weights
Take a small step downhill on the error surface. Learning rate sets step size.
Forward Pass — Make a Guess
Data flows through the network from input layer, through every hidden layer, to the output layer — using whatever weights and biases the network currently has (initially, these are random numbers). Example: a spam detector receives "the word 'free' appears 2 times" and, using its current weight and bias, outputs a 73.1% probability that the email is spam.
Loss Function — Measure How Wrong
The network's guess is compared to the actual correct answer (known during training). A loss function converts the size of that error into a single number. Low loss = good prediction; high loss = bad. The entire goal of training is to make this number as small as possible.
Backpropagation — Figure Out Who's to Blame
The single most important algorithm in deep learning. Backpropagation works backward from the output layer toward the input, calculating exactly how much each individual weight and bias contributed to the total error. It uses the calculus chain rule to trace responsibility layer by layer — producing a precise number (a "gradient") for each weight, representing how much and in which direction that weight should change.
Gradient Descent — Take a Small Step Toward Better
Once every weight's gradient is known, gradient descent updates each weight — nudging it slightly in the direction that reduces the loss.
The mountain analogy
Imagine standing blindfolded on a hillside, trying to reach the lowest point in the valley. You can't see the whole landscape — but you can feel the slope under your feet. Gradient descent takes a small step downhill, checks the new slope, takes another step, and repeats — gradually working toward the bottom (lowest possible error).
The loop repeats: Forward pass → Loss → Backpropagation → Gradient descent → Forward pass again with slightly better weights — repeated thousands or millions of times across many passes through the training data (each full pass is an "epoch"), until the loss shrinks to an acceptably small level. Forward pass is the guess; backward pass is the learning.
A Fully Worked Numerical Example
Let's trace one single neuron through exactly one round of this loop, using real numbers — so the abstract steps become concrete. (Keep the spirit: this is a walkthrough, not homework.)
Goal: Train a single-neuron model to detect spam based on how many times the word "free" appears.
Step 1 — Forward pass
Weighted sum = (0.5 × 2.0) + 0.0 = 1.0. Apply sigmoid → predicted probability ≈ 0.731 (73.1% chance of spam).
Step 2 — Loss function
True answer: 100% spam. Predicted: 73.1%. Using cross-entropy loss ≈ 0.313. Not terrible — room to improve.
Step 3 — Backpropagation
Error signal = predicted − actual = 0.731 − 1 = −0.269. Gradient for weight = −0.269 × 2.0 = −0.538. Gradient for bias = −0.269.
Step 4 — Gradient descent
w_new = 0.5 − (0.1 × −0.538) = 0.5538. b_new = 0.0 − (0.1 × −0.269) = 0.0269.
Result — run the forward pass again
New weighted sum ≈ 1.135 → new predicted probability ≈ 0.757 (75.7%).
The prediction improved from 73.1% to 75.7% — a small step closer to 100%. Repeat across thousands of labeled emails, and the weight and bias gradually converge on values that make accurate predictions across the board.
The Vanishing Gradient Problem: Why This Isn't Always Easy
Backpropagation sounds clean in theory, but for a long time it ran into a serious practical obstacle in very deep networks — one worth knowing by name, because it explains a real turning point in AI history.
The vanishing gradient problem
With sigmoid, the "blame signal" calculated during backpropagation gets multiplied by a very small number at each layer. Multiply many small numbers together, and the signal shrinks toward zero by the time it reaches the earliest layers. Those early layers essentially stop learning — their weights barely update at all.
The ReLU breakthrough (~2012)
Because ReLU passes positive values through unchanged, it doesn't shrink the gradient the same way. Swapping sigmoid for ReLU in hidden layers is a major reason the deep learning revolution of the 2010s became possible — enabling networks with dozens or hundreds of layers to train successfully where they previously couldn't.
Connecting This Back to LLMs
Everything in this lesson is the literal foundation underneath the LLM training process from the previous lesson.
THE FULL PICTURE:
An LLM is a neural network — an enormous one, with billions of neurons organized into specialized layers (including attention layers), each with its own learned weights and biases.
When an LLM is trained by predicting the next token across trillions of examples, the mechanism underneath is exactly this four-step loop: forward pass (predict next token) → loss (how wrong?) → backpropagation (trace blame through billions of parameters) → gradient descent (nudge every parameter slightly toward better predictions).
There is no separate reasoning engine, no rule book, no built-in accounting knowledge. There is only this loop, run an almost unimaginable number of times, producing weights that happen to encode extraordinarily useful statistical patterns.
Why This Matters for How You Evaluate AI Tools
Understanding the neuron-level mechanics gives you real, practical judgment about AI claims and behavior.
"More data generally means better predictions"
Each additional labeled example is another forward pass, loss calculation, and weight adjustment — refining the network's learned patterns further.
"The model can't explain itself the way a rule can"
The "explanation" for any prediction is distributed across millions or billions of individual weights, not a readable if-then statement. That's exactly why neural networks are often called "black boxes," in contrast to transparent rule-based systems.
"Training requires representative data"
The network only learns patterns that exist somewhere in its training examples. A fraud model trained only on wire transfer fraud will not automatically detect an entirely different fraud pattern it never saw labeled examples of.
"A model can be overtrained (overfit)"
If training continues too long on too narrow a dataset, the network can start memorizing specific training examples rather than learning generalizable patterns — performing beautifully on data it's seen before and poorly on new, real-world data.
Key Takeaway
Every neural network is built from simple neurons that multiply inputs by learned weights, add a bias, and pass the result through a non-linear activation function — stacked into input, hidden, and output layers. Training is a repeating four-step loop: a forward pass makes a prediction using the network's current weights, a loss function measures exactly how wrong that prediction was, backpropagation uses the calculus chain rule to calculate how much each individual weight contributed to the error, and gradient descent nudges every weight slightly in the direction that reduces future error. This loop — guess, measure, trace blame, adjust — repeats millions or trillions of times during training, and it is the entire mechanism underneath every neural network in existence, from a simple fraud detector to the largest large language models. There is no separate understanding module; there is only this loop, run an extraordinary number of times, converging on weights that encode useful patterns.
Test Your Understanding
Activation functions, backpropagation, and learning rates — check your answers below.
Question 1: A neural network is built using only weighted sums, with no activation function applied at any layer. What happens to its ability to model complex, non-linear patterns?
Question 2: During training, a loss function reports a very high error score after a forward pass. What does backpropagation do next?
Question 3: A network is trained with an extremely large learning rate. What is the most likely result?
Ready to Practice?
Apply accounting fundamentals in the Practice Lab while you build the AI fluency this course develops — judgment first, tools second.
Try the Practice LabWhat's Next?
AI Limitations & Hallucinations — Now that you understand the core neuron-and-training mechanism, the next lesson covers why fluent AI output can still be confidently wrong — and what professional habits you need when supervising these systems on client work.
AI Limitations & Hallucinations
Why fluent output can still be wrong
AI in Accounting Hub
Browse all AI pillar topics