SBU · MSc · Exam Deck · 14 chapters

Artificial Neural
Networks

The whole course, compressed: every formula, architecture and trade-off you need to walk into the exam with. Arrow keys to move, T for contents, R to hide formulas and quiz yourself.

Part I · Chapters 1–4

Foundations

1 — Introduction: the neuron, brain vs machine, history
2 — Origins of deep learning: MCP, XOR, depth, convolution
3 — Neuron models & network architectures
4 — The perceptron: boundary, learning rule, convergence

CH 01 · Introduction · neuron model

One neuron — weighted sum, then squash

net input — soma $$ z=\sum_{i=1}^{n} w_i x_i + b $$

weights = synaptic strength: $w\gt0$ excitatory, $w\lt0$ inhibitory. Bias shifts the firing threshold.

output — axon $$ y=f(z)=f\!\Big(\sum_i w_i x_i+b\Big),\qquad f(z)=\frac{1}{1+e^{-z}} $$

$z \ne y$. Net input in, activation out — classic exam trap.

bio → math mapping

dendrite → input soma → Σ axon hillock → f synapse → w

The three defining components (سه مؤلفه)

The reference frames every artificial neuron around three ingredients that the slide's single diagram compresses together:

the set of input signals the neuron receives from other neurons;
the rule by which its internal state changes once those inputs arrive — its activation (فعال‌سازی);
the single output signal the activated neuron then broadcasts to a set of downstream neurons.

Crucially, that downstream set may include the neuron's own input neurons — or even the firing neuron itself. When outputs loop back like this the network is recurrent (شبکه بازگشتی) rather than strictly feed-forward; the strict one-way $x\to z\to y$ picture is only the simplest case.

Biology → mathematics, part by part

The bio→math pills on the slide come from the biological neuron schematic. Filling in the parts:

Dendrites gather signals — the inputs $x_i$.
The soma (cell body, with its nucleus) integrates them — the summation $\Sigma$.
The axon hillock decides whether to fire — the activation $f$.
The axon carries the output; information flow is strictly directional. Its myelin sheath is insulation (عایق) and the nodes of Ranvier act as relays that boost the travelling pulse.
The axon terminal meets the next cell's dendrite at a synapse, whose strength is the weight $w_i$.

Net input versus activation

The boxed rule on the slide, $z=\sum_i w_i x_i+b$, is the analogue of the membrane potential; $f$ then squashes it into an output. Sign matters: $w\gt0$ is excitatory, $w\lt0$ is inhibitory, and $b$ slides the firing threshold. The common exam error is treating $z$ and $y$ as the same quantity — $z$ is the pre-activation, $y=f(z)$ is what leaves the neuron.

Simple unit, complex behaviour

A single unit is only a linear-threshold device; the PDF defines a neural network as a collection of simple computational units communicating over connections, and stresses that complex computation is realised by the network, not the unit. Four enablers make this possible: a high neuron count, parallel processing, asynchronous processing, and flexible connections — links can be created, strengthened (تقویت) or inhibited (مهار). That last property, connection plasticity, is where learning lives.

The same neuron is therefore studied at three engineering levels — neuron → layer → network — a hierarchy the later chapters build on, and the reason a device this simple can end up powering applications from autopilots and fault detection to speech recognition and medical-image analysis.

from: مقدمه.pdf · ch 1

CH 01 · Introduction · why parallelism wins

Slow parts, smart whole

Intelligence is emergent — many simple units, massively parallel, asynchronous, with plastic connectivity (= learning).

	Brain	Computer
Switching time	~1 ms (10⁻³ s)	~1 ns (10⁻⁹ s)
Signal speed	~100 m/s	~3×10⁸ m/s
Speed gap	machine ≈ 10⁶× faster per unit
Units	~10¹¹ neurons · 10¹³ synapses	~10¹⁰ transistors
Wins at	perception, robustness	arithmetic, precision

course scope

In: engineering view — neuron → layer → network. Out: neurobiology, VLSI hardware.

paradigm 1 · adaptive weight learning

Local, direct weight updates — MCP · Hebb · Perceptron · Adaline · RBF · SOM · Hopfield

paradigm 2 · error backpropagation

Chain rule pushes error backward — MLP · Deep nets · CNN · RNN · Autoencoders (+ SGD, dropout, batch-norm)

big numbers

10¹¹ neurons10⁴ synapses / neuron cortex 2×10¹⁰ n10¹³ connections

The paradox the numbers hide

Per unit the machine is roughly a million times faster than the brain (nanosecond vs millisecond switching; signals near light speed vs about 100 m/s down an axon), yet the brain still wins at perception and robustness. The resolution is architectural, not clock-speed: the brain trades slow, cheap parts for extreme parallelism and adaptivity.

~100 billion neurons at birth ($10^{11}$).
Each neuron connects on average to 10–15 thousand others.
~20 billion neurons in the cortex, and about $10^{13}$ connections in total.

Why parallelism beats raw speed

Because millions of neurons can fire simultaneously and asynchronously, the effective throughput dwarfs a sequential processor even though every individual neuron is sluggish. The extra decisive ingredient is plasticity (شکل‌پذیری): connections can be created, strengthened, or inhibited — this reconfigurability is learning, and it has no direct analogue in fixed silicon. In the deck's own framing, intelligence is emergent: it comes from many simple, slow units wired densely rather than from any single fast component. The comparison also flips on unit count — the brain's ~$10^{11}$ neurons and ~$10^{13}$ synapses far exceed a chip's ~$10^{10}$ transistors, and each synapse is adaptive.

Two reasons to study ANNs

Engineering: bio-inspired models appear to give high power on hard AI problems — pattern recognition, text and language, signal processing.
Scientific: analysing the properties of artificial networks can, in reverse, illuminate how information is processed in biological networks such as the brain.

Levels of study — two viewpoints

The course fixes a scope by choosing one of two ways to slice the subject:

Engineering / mathematics: neuron, then layer, then network — the view this deck takes.
Neuroscience: synapse, neuron, circuits (with a defined function), maps, and subsystems.

Everything neurobiological below the "circuit" line, and all VLSI/electronic implementation, is explicitly out of scope.

The two learning paradigms

The two cards on the slide organise the whole syllabus. Adaptive-weight learning covers MCP, Hebb, Perceptron, Adaline, radial-basis, Kohonen/SOM–LVQ and Hopfield. Error backpropagation covers the MLP, deep nets, CNNs, RNNs and autoencoders — together with the practical machinery of stochastic gradient descent, regularisation, dropout, batch normalisation and loss-function design.

from: مقدمه.pdf · ch 1

CH 01 · Introduction · 70 years in one line

Two springs, one winter

the winter

1969: single-layer perceptron can't do XOR — wrongly generalized to all neural nets → funding collapse.

the thaw

1986: backprop trains hidden layers → non-separable problems solved, critique voided.

the boom

2012: AlexNet = depth + GPUs + data. Deep learning era begins.

Before the timeline: 1938

The deck's line starts at 1943, but the PDF opens earlier: in 1938 Rashevsky described neural activation dynamics with differential equations — the first attempt to put neural behaviour in mathematical form, before the discrete-logic model arrived.

The founding decades (1943–1964)

1943 McCulloch & Pitts — the first mathematical model of a biological neuron (threshold logic).
1949 Hebb — the learning rule: repeated activation of one neuron by another strengthens their connection.
1958 Rosenblatt — the perceptron, essentially the MCP model plus a learning algorithm.
1960 Widrow & Hoff — the Adaline, a simple network trained by gradient descent.
1962 Hubel & Wiesel — visual-cortex properties that motivate self-organising models (and later CNNs).
1964 Taylor — the first winner-take-all circuit, using inhibition among output units.

The winter — and a widespread misconception

In 1969 Minsky & Papert proved that a single-layer perceptron is not computationally universal — famously, it cannot compute XOR. The damage came from over-generalisation: the limitation of one layer was wrongly read as a limitation of all neural networks, interest collapsed, and funding dried up. The correct statement is narrow — add a hidden layer and XOR is solvable.

The thaw and the boom

1982 — Hopfield's auto-association network; Kohonen's self-organising map.
1986 Rumelhart, Hinton & Williams — backpropagation in its modern form, finally training hidden layers and voiding the 1969 critique.
1989 LeCun et al. — convolutional networks applied to handwritten zip-code recognition.
1997 Hochreiter & Schmidhuber — Long Short-Term Memory, the most successful recurrent unit.
2012 AlexNet — won ImageNet by combining depth, GPUs and large data, opening the deep-learning era.

The pattern is the deck's "two springs, one winter": a first spring ending in 1969, a revival from the mid-1980s, and the modern boom from 2012 — each recovery driven by removing a specific limitation (depth, then compute + data). A useful way to remember it: each thaw fixes exactly what the previous winter exposed — backpropagation answers the XOR/depth objection of 1969, and GPU-trained deep CNNs answer the scaling objection that had kept multi-layer nets impractical.

from: مقدمه.pdf · ch 1

CH 02 · Origins · McCulloch–Pitts 1943

Logic from a threshold

mcp rule $$ y=\begin{cases}1 & \sum_i x_i \ge \theta\\[2pt] 0 & \text{otherwise}\end{cases} $$

Unweighted binary inputs; the only free parameter is θ. Inhibitory input = absolute veto, not a negative weight.

gate	weights	θ	note
AND	+1, +1	2	line $x_1{+}x_2{=}2$
OR	+1, +1	1	line $x_1{+}x_2{=}1$
NOT	inhibitory	0	$y=1-x_1$ · veto
XOR	impossible in one layer

4 limits → 4 upgrades

binary inputs	→ real-valued $x_i$
hand-set θ	→ learnable bias
equal weights	→ free weights $w_i$
separable only	→ hidden layers

threshold ↔ bias $$ b=-\theta \qquad\text{(bias = weight on a constant input } x_0{=}1) $$

Same summing hardware, but now the threshold is learnable like any weight. Watch the sign flip.

Why start with logic at all

The chapter's motivation runs deeper than the gate table. The claim being tested is that an artificial neural network is a universal model for general AI — i.e. it "can learn any computable formula" — a statement that has been proved (Scarselli & Tsoi, 1998; Schäfer & Zimmermann, 2006), but which fails without depth. To reach that goal the course deliberately "mimics nature", and the smallest natural unit is the McCulloch–Pitts neuron.

Being intelligent, the PDF argues, means understanding logical compounds, arithmetic/algebra, and the association between words and meanings. Two routes are offered: organising IF–THEN rules (a decision tree, درخت تصمیم) or finding the right way to "multiply and sum" — a linear model $ax_1+bx_2+c=0$ (دسته‌بند خطی). The MCP neuron is the second route in its purest form.

The MCP formalism

Inputs are binary, $x_i\in\{0,1\}$, and there are only excitatory inputs. The neuron first sums (جمع), then applies a threshold activation (فعال‌سازی):

$$ g(\mathbf{x})=\sum_{i=1}^{n}x_i,\qquad y=f(g(\mathbf{x}))=\begin{cases}1 & g(\mathbf{x})\ge\theta\\ 0 & g(\mathbf{x})\lt\theta\end{cases} $$

Geometrically the equation $\sum_i x_i=\theta$ is a separating hyperplane; the region where $y=1$ is the half-space on the far side of it. This is exactly why the boxed rule on the slide works gate by gate.

Reading the gate table geometrically

AND: $\theta=2$. Only $(1,1)$ lies on or beyond the line $x_1+x_2=2$.
OR: $\theta=1$. Every point except $(0,0)$ satisfies $x_1+x_2\ge1$.
3-input OR: $\theta=1$ gives the plane $x_1+x_2+x_3=1$; only the origin is excluded.
NOT: handled by an inhibitory input acting as an absolute veto, giving $y=1-x_1$.

Four limitations that force the upgrades

The four-row "limits → upgrades" card summarises what the pure MCP cannot do, phrased in the PDF as four questions: What about non-boolean (real-valued) inputs? Must the threshold always be hand-coded? Are all inputs equal, or should some carry more importance? And what about functions that are not linearly separable? Each answer is an upgrade — real inputs, a learnable bias, free weights $w_i$, and hidden layers respectively. XOR is the standard counter-example that survives none of these fixes until weights, a learnable bias, and hidden layers are all present at once.

from: خاستگاه یادگیری ژرف.pdf · ch 2

CH 02 · Origins · XOR · depth · UAT

XOR breaks the line — depth fixes it

no single line separates ● from ○ — band needs two lines

minimal fix · 2 hidden units $$ y=\mathrm{OR}(x_1,x_2)-\mathrm{AND}(x_1,x_2) $$

why nonlinearity — collapse proof $$ (\mathbf{x}W_1{+}\mathbf{b}_1)W_2{+}\mathbf{b}_2=\mathbf{x}(W_1W_2)+(\mathbf{b}_1W_2{+}\mathbf{b}_2) $$

Stacked linear layers = one linear layer. Without $f$, depth buys nothing. (ReLU's corner at 0 is its nonlinearity.)

hidden layers	boundary you can draw
0	straight line only
≥ 1	closed / connected curves
≥ 2	disjoint, arbitrary regions

universal approximation · 1989 $$ |g(\mathbf{x})-f(\mathbf{x})|\lt\varepsilon \quad\text{(one hidden layer suffices)} $$

Existence only — says nothing about width needed or learnability. Params grow $(L{-}1)n^2$: linear in depth, quadratic in width → deep beats wide.

From threshold to learnable bias

Adding weights turns the sum into $z=f\big(\sum_i x_i w_i\big)$, and folding the threshold into a bias gives $z=f\big(b+\sum_i x_i w_i\big)$, where $b$ is simply the weight on a constant input $x_0=1$. The two formulations are equivalent: classifying with "$z\ge\theta\Rightarrow C_1$" on $f(\mathbf{x}\cdot\mathbf{w})$ matches "$z\ge0\Rightarrow C_1$" on $f(b+\mathbf{x}\cdot\mathbf{w})$ provided (for increasing $f$)

$$ b=f^{-1}(0)-f^{-1}(\theta). $$

The payoff: the threshold is now a trainable parameter like any weight — watch the sign, since $b=-\theta$ in the simple linear case.

XOR needs a hidden layer

XOR is not linearly separable, so no single MCP line works. The minimal fix on the slide, $y=\mathrm{OR}(x_1,x_2)-\mathrm{AND}(x_1,x_2)$, is realised by a small hidden layer feeding an output unit (e.g. hidden-to-output weights $1,-2,1$). This is the concrete reason depth was introduced historically.

Why the activation must be nonlinear

Stacking layers as a composition $y=f\big(f\big(f(\mathbf{x}\cdot W_1)\cdot W_2\big)\cdot W_3\big)$ only buys power if $f$ is nonlinear. With a linear $f$ the whole stack collapses to a single linear map — the deck's collapse proof — so "repeating linear layers yields nothing but the linear model." The usable choices:

Sign / step — non-differentiable; the classic Rosenblatt (1957) perceptron.
Sigmoid — $\sigma(a)=\dfrac{1}{1+e^{-a}}$, its argument $a=\sum_j w_j x_j-b$ being the logit; used in the MLP.
Tanh — behaves like a shifted-scaled sigmoid, $\sigma(z)=\tfrac{1}{2}\big(1+\tanh(z/2)\big)$.
ReLU — $\max(a,0)$; derivative $0$ for $a\lt0$ and $1$ for $a\gt0$, non-differentiable at the origin, dominant in deep and convolutional nets.

How much depth buys — and its cost

The universal-approximation guarantee is precise about existence but silent about size: for any $f:\mathbb{R}^n\to\mathbb{R}^m$ there is a network with hidden layer(s) whose output $g$ satisfies $|g(\mathbf{x})-f(\mathbf{x})|\lt\Theta$. In classification terms:

0 hidden layers — only linearly separable data.
≥1 hidden layer — any nonlinear problem with a continuous decision boundary.
≥2 hidden layers — even discontinuous / disjoint regions.

Intuitively a hidden layer applies a nonlinear transform $f(\mathbf{x}\cdot W)$ that projects the data into a space where it becomes linearly separable. The catch, stressed at the end of the chapter, is that the parameter count grows explosively with depth, so in practice a plain deep MLP failed to yield a general model — the problem that convolutional networks were introduced to solve.

from: خاستگاه یادگیری ژرف.pdf · ch 2

CH 02 · Origins · parameter sharing

Convolution — one kernel, every position

discrete convolution $$ [f*g](i)=\sum_a f(a)\,g(i-a) \qquad V(x,y)=\sum_m\sum_n I(x{+}m,y{+}n)\,K(m,n) $$

The image version is really cross-correlation (no kernel flip) — identical for symmetric kernels.

kernel 3×3	effect
sharpen	0,−1,0 / −1,5,−1 / 0,−1,0 — boost contrast
blur	(1/16)·[1,2,1 / 2,4,2 / 1,2,1] — smooth
outline	−1 ring around 8 — all edges
Sobel	1,2,1 / 0,0,0 / −1,−2,−1 — horizontal edges

fully connected vs conv

FC	one weight per input×output — params scale with image size
Conv	9 weights + 1 bias (3×3), any image size

biological root · hubel & wiesel

Cat visual cortex (1959, Nobel 1981): simple → complex → hypercomplex cells = kernel → stacked conv layers. Local receptive fields, hierarchy of features.

The problem convolution solves

A fully connected deep net's parameter count explodes with depth, and in practice that blocked a general model. The chapter's answer is parameter sharing within a layer: reuse one small set of weights at every position. This keeps the advantage of cheap linear operations while cutting parameters drastically — and, like the neuron itself, it is nature-inspired.

The biological root: Hubel & Wiesel

The 1959 cat experiment (Nobel 1981) recorded from the visual cortex while showing oriented bars. A cell fires maximally at its preferred orientation and falls off for others — an orientation tuning curve peaking near 0° of misalignment. Cells are organised as a hierarchy: simple → complex → hypercomplex, with a topographical mapping from low-level to high-level features. That hierarchy of local detectors is exactly what stacked convolutional layers imitate: a kernel is a "simple cell", deeper layers build "complex" features.

Convolution, precisely

The operator is defined continuously and discretely:

$$ [f*g](x)=\int_{\mathbb{R}^d} f(z)\,g(x-z)\,dz,\qquad [f*g](i)=\sum_a f(a)\,g(i-a). $$

Here $g$ is the kernel, and the operation acts as a local feature detector. Note the $-a$ (or $-z$): true convolution flips the kernel. The image-processing form the slide uses,

$$ V(x,y)=\sum_m\sum_n I(x{+}m,\,y{+}n)\,K(m,n), $$

has a $+m,+n$ with no flip — so it is really cross-correlation, which coincides with convolution only for symmetric kernels. Each output pixel is the element-wise product of the kernel and the underlying image patch, summed.

Worked kernels and their outputs

On the PDF's demonstration image the vertical-edge kernel $K=\begin{bmatrix}1&0&-1\\1&0&-1\\1&0&-1\end{bmatrix}$ returns $-7$ at the sampled position; the 1-D version $K=[\,1\;\;-1\,]$ produces $\pm1$ exactly at vertical edges and $0$ on flat regions — hence "vertical-line detection". The same setup gives characteristic single-pixel outputs:

sharpen ($0,-1,0/-1,5,-1/0,-1,0$) → $226$;
blur (normalised Gaussian) → $173$;
outline ($-1$ ring around $8$) → $208$;
top Sobel ($1,2,1/0,0,0/-1,-2,-1$) → $-14$.

The lesson carried into CNNs: a handful of shared weights per kernel can extract a meaningful local feature at every location, independent of image size. A classifier can then learn these kernels instead of hand-designing them, so early layers discover edge- and blur-like detectors on their own — the engineered filters above are just the interpretable special cases of what a trained convolutional layer converges to.

from: خاستگاه یادگیری ژرف.pdf · ch 2

CH 03 · Neuron models · hagan notation

The general neuron and its dimensions

neuron → layer → network $$ a=f(\mathbf{Wp}+b) \qquad\Rightarrow\qquad \mathbf{a}^{l}=f^{l}\!\big(\mathbf{W}^{l}\mathbf{a}^{l-1}+\mathbf{b}^{l}\big),\;\; \mathbf{a}^0=\mathbf{p} $$

Whole feedforward net = composition: $\mathbf{a}^3=f^3(\mathbf{W}^3 f^2(\mathbf{W}^2 f^1(\mathbf{W}^1\mathbf{p}+\mathbf{b}^1)+\mathbf{b}^2)+\mathbf{b}^3)$

dimension check — free marks

$\mathbf p$ R×1$\mathbf W$ S×R $\mathbf b$ S×1$\mathbf n,\mathbf a$ S×1 S neurons · R inputs

depth — off-by-one trap

Depth = number of weight layers. Input layer computes nothing and is not counted. First $L{-}1$ layers hidden, last is output.

most realistic — hodgkin–huxley 1952 $$ I=C_m\frac{dV_m}{dt}+g_K(V_m{-}V_K)+g_{Na}(V_m{-}V_{Na})+g_l(V_m{-}V_l) $$

Membrane as RC circuit; Nobel 1963. Model spectrum: HH → … → integrate-and-fire → our neuron (only the I/O relation survives).

	MCP	general
inputs	{0,1}	ℝ
weights	fixed ±1	learnable
threshold	hand-set θ	bias b = −θ
activation	hardlim	any f

From biology to a computable unit

The biological cell contributes three parts to the abstraction: dendrites collect signals (the inputs $p_i$), the cell body/soma integrates them (the summer $\Sigma$), and the axon with its terminals broadcasts one output to many downstream cells. The artificial neuron keeps only this input–output relation and throws away membrane chemistry. The PDF lists a whole ladder of biological-fidelity models, from most detailed to most abstract: integrate-and-fire, Hodgkin–Huxley, leaky integrate-and-fire, FitzHugh–Nagumo, Morris–Lecar, Hindmarsh–Rose, cable theory, and compartmental models. The slide's Hodgkin–Huxley RC circuit sits near the detailed end; our neuron $a=f(\mathbf{Wp}+b)$ is the far, purely functional end.

McCulloch–Pitts (1943) — the first computational neuron

Before the general model, MCP proposed a binary threshold unit: inputs and output live in $\{0,1\}$, weights are fixed at $+1$ for an excitatory synapse (سیناپس تحریک‌کننده) and $-1$ for an inhibitory synapse (سیناپس مهارکننده), and firing is a step at threshold $T$:

$$ y=\begin{cases}0 & \sum_i w_i x_i \lt T\\[2pt] 1 & \sum_i w_i x_i \ge T\end{cases} $$

Worked example — a NAND gate: two inputs with weights $w_1=w_2=-1$ and $T=-1.5$. Check $x=(1,1)$: $\sum w_i x_i=-2 \lt -1.5 \Rightarrow y=0$; any other input gives $\sum \ge -1 \ge -1.5 \Rightarrow y=1$. That reproduces NAND, and since NAND is functionally complete, networks of MCP units can compute any Boolean function.

Why bias replaces threshold

The general neuron on the slide upgrades MCP on three axes at once (real-valued inputs, learnable weights, any activation), but the subtle one is the threshold. Rewrite the MCP condition $\sum w_i x_i \ge T$ as $\sum w_i x_i - T \ge 0$, i.e. $\mathbf{Wp}+b\ge 0$ with $b=-T$. So the bias is minus the threshold: moving $T$ is exactly translating the boundary, and making $b$ learnable means the firing threshold itself is learned rather than hand-set.

Dimension and depth traps

Keep the shapes straight: with $R$ inputs and $S$ neurons, $\mathbf p$ is $R\times1$, $\mathbf W$ is $S\times R$, and $\mathbf b,\mathbf n,\mathbf a$ are $S\times1$. A common exam slip is counting layers: depth = number of weight layers, so the input vector is not a layer (it computes nothing). A "3-layer" net has weight matrices $\mathbf W^1,\mathbf W^2,\mathbf W^3$, with the first two layers hidden and the last the output — giving the composition $\mathbf a^3=f^3(\mathbf W^3 f^2(\mathbf W^2 f^1(\mathbf W^1\mathbf p+\mathbf b^1)+\mathbf b^2)+\mathbf b^3)$.

from: مدل‌های نورونی و معماری شبکه.pdf · ch 3

CH 03 · Neuron models · interactive

Activation functions — click to explore

solid = f(n) · dashed = f′(n)

name	range	use
hardlim / hardlims	{0,1} / {−1,1}	perceptron
purelin	ℝ	regression, Adaline
satlin / satlins	[0,1] / [−1,1]	clipped linear
logsig	(0,1)	output prob.
tansig	(−1,1)	hidden (zero-centered)
poslin = ReLU	[0,∞)	deep nets
compet	one-hot	winner-take-all (layer-wise)

gradient rule of thumb

Corners (hardlim, satlin) break differentiability; sigmoids saturate; ReLU keeps gradient = 1 for n>0.

Threshold (hard) functions

The Hard Limit (hardlim) is $a=0$ for $n\lt 0$ and $a=1$ for $n\ge 0$; its symmetric twin hardlims outputs $-1$ / $+1$ on the same split. Both are flat everywhere except a jump at $n=0$, so their derivative is $0$ almost everywhere and undefined at the corner. That is exactly why perceptrons are trained by the perceptron/Hebb rules and not by gradient descent — there is no usable slope to follow.

Linear family

purelin: $a=n$, unbounded range $\mathbb{R}$, constant derivative $1$; used for regression outputs and Adaline. Saturating linear (satlin) clips the line into a ramp: $a=0$ for $n\lt 0$, $a=n$ for $0\le n\le 1$, $a=1$ for $n\gt 1$. The symmetric version satlins ramps between $-1$ and $1$: $a=-1$ for $n\lt -1$, $a=n$ for $-1\le n\le 1$, $a=1$ for $n\gt 1$. These are piecewise-linear, so their derivative is $1$ in the active band and $0$ in the saturated tails.

Sigmoid family

Log-Sigmoid (logsig): $a=\dfrac{1}{1+e^{-n}}$, squashes $\mathbb{R}$ into $(0,1)$ — natural for probability-like outputs; its derivative has the tidy form $a(1-a)$. Hyperbolic Tangent Sigmoid (tansig): $a=\dfrac{e^{n}-e^{-n}}{e^{n}+e^{-n}}$, range $(-1,1)$, derivative $1-a^2$. Because tansig is zero-centred it is the classic hidden-layer choice; both sigmoids saturate for large $|n|$, driving the gradient toward $0$ (the vanishing-gradient issue).

ReLU and the layer-wise competitive unit

Positive Linear (poslin), better known as ReLU: $a=0$ for $n\lt 0$, $a=n$ for $n\ge 0$. Its derivative is exactly $1$ on the positive side, so it neither saturates nor shrinks the gradient there — the reason it dominates deep nets. Finally the Competitive layer (compet) is different in kind: it is not a pointwise function but acts across a whole layer, setting $a=1$ for the single neuron with the largest net input $n$ and $a=0$ for all others (winner-take-all). Remember this when a question mixes compet with the others — you cannot plot it as a single $f(n)$ curve.

Choosing one

Binary decision at the output → hardlim / hardlims.
Continuous target → purelin.
Bounded probability output → logsig; zero-centred hidden units → tansig.
Deep hidden layers → ReLU (poslin); one-hot / clustering output → compet.

from: مدل‌های نورونی و معماری شبکه.pdf · ch 3

CH 03 · Neuron models · architectures

Static map vs dynamical system

recurrent dynamics + fixed point $$ \mathbf{a}(t{+}1)=f(\mathbf{W}\mathbf{a}(t)+\mathbf{b}),\quad \mathbf{a}(0)=\mathbf{p} \qquad\; \mathbf{a}^{*}=\mathbf{W}\mathbf{a}^{*}+\mathbf{b} $$

	feedforward	recurrent
computes	static $a{=}F(p)$	trajectory $a(t)$
feedback	none	delay block D
needs	—	initial state $a(0)$
behavior	one pass	converge · oscillate · diverge

recurrent ≠ deep

Depth counts forward weight layers; recurrence is feedback to an already-visited point. A 1-layer net can be fully recurrent.

layer receptive field

Fully connected → sees entire input. Convolutional → local subset. Same math $f(\mathbf{Wp}+\mathbf{b})$, different sparsity of W.

What counts as a "layer"

A neuron layer is a group of neurons that share a common receptive field (میدان ادراکی) — the set of inputs each of them can potentially draw from. Stack $S$ such neurons over $R$ inputs and the layer is summarised by one weight matrix $\mathbf W$ of size $S\times R$ with $a=f(\mathbf{Wp}+\mathbf b)$. Defining one layer cleanly is the first step toward more elaborate architectures, because a multilayer net is just layers chained so that each layer's output vector becomes the next layer's input.

Feedforward = one open pass

In a feedforward net information moves along an open path: each layer sends its output only forward to later layers, and there is no feedback. The map is static — for a fixed $\mathbf p$ you compute $\mathbf a^1=f^1(\mathbf W^1\mathbf p+\mathbf b^1)$, then $\mathbf a^2=f^2(\mathbf W^2\mathbf a^1+\mathbf b^2)$, then $\mathbf a^3$, in a single sweep. The whole network is therefore a fixed function composition $\mathbf a=F(\mathbf p)$; run it twice on the same input and you get the same answer.

Recurrence via the delay block

A recurrent net removes the "forward-only" restriction: information can travel through closed loops and be reused. The standard building block is the delay block (بلوک تأخیر) $\mathbf D$, defined by $\mathbf a(t)=\mathbf u(t-1)$ — it simply hands this step's output back as next step's input. Because the block needs a value to start from, a recurrent net requires an initial condition $\mathbf a(0)=\mathbf p$; a feedforward net needs none. The PDF's concrete example uses a symmetric-saturating-linear layer:

$$ \mathbf a(0)=\mathbf p,\qquad \mathbf a(t{+}1)=\operatorname{satlin}(\mathbf W\mathbf a(t)+\mathbf b). $$

Iterating this produces a trajectory $\mathbf a(t)$ that can converge to a steady state, oscillate, or diverge — behaviours a static feedforward map cannot show.

Fixed points and a warning

A steady state $\mathbf a^{*}$ satisfies $\mathbf a^{*}=\mathbf W\mathbf a^{*}+\mathbf b$ (before the nonlinearity); it is where the dynamics stop moving. Two frequent confusions: (1) recurrent is not the same as deep — depth counts distinct forward weight layers, whereas recurrence is feedback to an already-visited point, so even a single-layer net can be fully recurrent; (2) full vs local connectivity is orthogonal to recurrence — a fully connected layer's neuron sees the entire input, a convolutional one sees only a local subset, but both use the same $f(\mathbf{Wp}+\mathbf b)$ with a different sparsity pattern in $\mathbf W$.

from: مدل‌های نورونی و معماری شبکه.pdf · ch 3

CH 04 · Perceptron · rosenblatt 1958

The boundary is a hyperplane — and w is normal to it

hand-design recipe

1. pick $\mathbf w$ ⊥ desired boundary, pointing at class 1 · 2. plug one boundary point into $\mathbf w^T\mathbf p+b=0$, solve $b$. Any positive scaling works — solutions not unique.

perceptron $$ \mathbf a=\mathrm{hardlim}(\mathbf W\mathbf p+\mathbf b),\qquad \mathrm{hardlim}(n)=\begin{cases}1 & n\ge 0\\ 0 & n\lt 0\end{cases} $$

Convention: $n=0$ outputs 1 — easy to miss under pressure.

why w ⊥ boundary $$ \mathbf w^T\mathbf p_a+b=0,\;\; \mathbf w^T\mathbf p_b+b=0 \;\Rightarrow\; \mathbf w^T(\mathbf p_a-\mathbf p_b)=0 $$

capacity

$S$ neurons → $S$-bit code → up to $2^S$ classes (upper bound, not all realizable). OR: $\mathbf w=[0.5,0.5]^T, b=-0.25$ · AND: $[1,1]^T, b=-1.5$

Single-layer perceptron, indexed

Stacking $S$ hard-limit neurons gives $\mathbf a=\operatorname{hardlim}(\mathbf W\mathbf p+\mathbf b)$, where row $i$ of $\mathbf W$ is neuron $i$'s weight vector, written ${}_i\mathbf w^{T}$. Then each neuron acts on its own line, $a_i=\operatorname{hardlim}({}_i\mathbf w^{T}\mathbf p+b_i)$, so the network is $S$ independent linear classifiers sharing one input. Watch the tie convention: $\operatorname{hardlim}(0)=1$, so a point exactly on a boundary is assigned to class $1$.

A concrete two-input neuron

Take $w_{1,1}=1,\ w_{1,2}=1,\ b=-1$. The output is $a=\operatorname{hardlim}(w_{1,1}p_1+w_{1,2}p_2+b)=\operatorname{hardlim}(p_1+p_2-1)$, and the boundary line is $p_1+p_2-1=0$, i.e. the segment from $(1,0)$ to $(0,1)$. Everything on or above that line fires ($a=1$); everything below is $a=0$. The weight vector $[1,1]^T$ points into the $a=1$ region and is perpendicular to the line — the two defining geometric facts.

Why the boundary is a hyperplane normal to w

The decision boundary is the locus of inputs whose inner product with the weight vector equals the constant $-b$: $\ {}_1\mathbf w^{T}\mathbf p=-b$. Take any two boundary points $\mathbf p_a,\mathbf p_b$; subtracting their defining equations gives ${}_1\mathbf w^{T}(\mathbf p_a-\mathbf p_b)=0$. Since $\mathbf p_a-\mathbf p_b$ is an arbitrary direction along the boundary, $\mathbf w$ is orthogonal to all of them — hence normal to the hyperplane. The orientation is fixed by $\mathbf w$; $b$ only slides the plane along that normal.

Hand-designing the OR neuron

Data: $\mathbf p_1=[0,0]^T\,(t{=}0)$, $\mathbf p_2=[0,1]^T\,(t{=}1)$, $\mathbf p_3=[1,0]^T\,(t{=}1)$, $\mathbf p_4=[1,1]^T\,(t{=}1)$. Only the origin is class $0$, so the boundary must separate it from the other three. Step 1: pick $\mathbf w$ perpendicular to a good separating line and pointing at class $1$, e.g. ${}_1\mathbf w=[0.5,\,0.5]^T$. Step 2: force one point on the boundary to satisfy ${}_1\mathbf w^{T}\mathbf p+b=0$. Using the midpoint $[0,\,0.5]^T$: $[0.5\ 0.5][0,\,0.5]^T+b=0.25+b=0\Rightarrow b=-0.25$. The same recipe gives AND as $[1,1]^T,\ b=-1.5$. Solutions are not unique: any positive rescaling of $(\mathbf w,b)$ works, since it leaves the sign of $\mathbf w^{T}\mathbf p+b$ unchanged.

Capacity of many neurons

Each neuron carves its own line ${}_i\mathbf w^{T}\mathbf p+b_i=0$; one neuron separates the plane into $2$ classes. With $S$ neurons the outputs form an $S$-bit code, so up to $2^{S}$ regions/classes can be named — an upper bound, not a guarantee that every desired partition is realisable.

from: perceptron slides (04_ArtificialNeuralNetworks) · ch 4

CH 04 · Perceptron · learning rule + convergence

Add the sample, subtract the sample

the rule — one line, three cases $$ e=t-a\in\{-1,0,1\}\qquad \mathbf w^{new}=\mathbf w^{old}+e\,\mathbf p,\quad b^{new}=b^{old}+e $$ $$ \text{matrix form: }\; \mathbf W^{new}=\mathbf W^{old}+\mathbf e\,\mathbf p^{T},\quad \mathbf b^{new}=\mathbf b^{old}+\mathbf e $$

Don't forget the bias update — top exam error. Trace protocol: show $n, a, e$ for every sample before updating.

error	meaning	action
e = +1	said 0, truth 1	w += p (pull toward)
e = −1	said 1, truth 0	w −= p (push away)
e = 0	correct	no change

novikoff convergence bound $$ \#\text{updates}\;\le\;\frac{R^{2}}{\gamma_g^{2}} \qquad (\|\mathbf p_i\|\le R,\ \text{margin }\gamma_g) $$

Proof skeleton: alignment grows linearly $\mathbf u^T\mathbf w^{(k)}\ge k\gamma_g$, norm grows as √k $\|\mathbf w^{(k)}\|\le\sqrt{k}R$; Cauchy–Schwarz squeezes $k\le R^2/\gamma_g^2$.

xor impossibility — contradiction

Need $b\lt0$, $w_1\ge-b$, $w_2\ge-b$ → $w_1{+}w_2{+}b\ge-b\gt0$, but XOR(1,1)=0 needs it $\lt0$. ∎ Converges iff linearly separable — else updates forever.

worked numbers

R=3, γ=0.5 → ≤36 updatesEx 4.2: 3 corrections

Where the rule comes from — a full trace

The rule on the slide is discovered, not postulated. Use a no-bias neuron on $\mathbf p_1=[1,2]^T\,(t{=}1)$, $\mathbf p_2=[-1,2]^T\,(t{=}0)$, $\mathbf p_3=[0,-1]^T\,(t{=}0)$, starting from a random ${}_1\mathbf w=[1.0,-0.8]^T$.

Present $\mathbf p_1$: $n=[1.0\ {-}0.8][1,2]^T=-0.6$, so $a=\operatorname{hardlim}(-0.6)=0$, but $t=1$ — wrong. Add the pattern: ${}_1\mathbf w=[1.0,-0.8]^T+[1,2]^T=[2.0,\,1.2]^T$.
Present $\mathbf p_2$: $n=[2.0\ 1.2][-1,2]^T=0.4$, $a=1$, but $t=0$ — wrong. Subtract: ${}_1\mathbf w=[2.0,1.2]^T-[-1,2]^T=[3.0,-0.8]^T$.
Present $\mathbf p_3$: $n=[3.0\ {-}0.8][0,-1]^T=0.8$, $a=1$, but $t=0$ — wrong. Subtract: ${}_1\mathbf w=[3.0,-0.8]^T-[0,-1]^T=[3.0,\,0.2]^T$. This now classifies all three correctly.

A rejected first idea was ${}_1\mathbf w^{new}=\mathbf p$ (copy the misclassified point) — it is unstable, thrashing between patterns. Adding/subtracting the pattern instead rotates $\mathbf w$ toward or away from it more gently.

Collapsing three cases into one line

The three observed moves — $t{=}1,a{=}0\Rightarrow \mathbf w{+}\mathbf p$; $t{=}0,a{=}1\Rightarrow \mathbf w{-}\mathbf p$; $t{=}a\Rightarrow$ no change — collapse using $e=t-a\in\{-1,0,1\}$ into ${}_1\mathbf w^{new}={}_1\mathbf w^{old}+e\,\mathbf p$. Treating the bias as a weight on a constant input of $1$ gives the companion update $b^{new}=b^{old}+e$, which is the most-forgotten step in exams.

Learning in context

This is supervised learning: input/target pairs $\{\mathbf p_q,\mathbf t_q\}$ are given and the goal is minimising prediction error. It contrasts with unsupervised learning (inputs only, cluster similar samples) and reinforcement learning (learn from environment reward/penalty). All fit the generic template ${}_1\mathbf w^{new}={}_1\mathbf w^{old}\oplus H(\text{sample})$; the perceptron, Hebb, and Kohonen rules are three instances.

Convergence, margin, and separability

The algorithm (initialise $\mathbf w=0$; for each example compute $e=t-\operatorname{hardlim}(\mathbf w^{T}\mathbf p_i)$ and, on a mistake, $\mathbf w\mathrel{+}= e\,\mathbf p_i$) cycles through the data, possibly reusing each sample many times, until a convergence test lets it break. Novikoff guarantees termination after at most $R^2/\gamma_g^2$ updates, where $\|\mathbf p_i\|\le R$ and $\gamma_g$ is the geometric margin — the width of the gap between classes. A wide margin (good separation) means few mistakes; a thin one means many. The whole guarantee holds iff the data is linearly separable; for XOR no line exists, $\gamma_g$ is undefined, and the updates never stop.

from: perceptron slides (04_ArtificialNeuralNetworks) · ch 4

Part II · Chapters 5–8

Training

5 — MLP & error backpropagation: the δ recurrence
6 — SGD, learning-rate decay, momentum, Nesterov
7 — Adaptive rates: AdaGrad → RMSProp → Adam
8 — Regularization: L1/L2, dropout, batch norm

CH 05 · MLP · why hidden layers

Make the basis functions trainable

forward pass — index 0 is the bias unit $$ a_j=\sum_{i=0}^{D}w^{(1)}_{ji}x_i,\quad z_j=h(a_j) \qquad a_k=\sum_{j=0}^{M}w^{(2)}_{kj}z_j,\quad y_k=f(a_k) $$

Index convention: first = destination, second = source. Params: $M(D{+}1)+K(M{+}1)$.

glm vs mlp $$ y=f\Big(\textstyle\sum_i w_i\,\phi_i(\mathbf x)\Big)\;\longrightarrow\; y=f\Big(\textstyle\sum_i w_i\,\phi_i(\mathbf x,\mathbf w^{i})\Big) $$

GLM: features $\phi_i$ fixed by hand (engineering doesn't scale — $\binom{D+p}{p}$ monomials). MLP: hidden units are learned features.

output activation	loss (from ML)	task
identity	SSE (Gaussian noise)	regression
sigmoid	binary CE (Bernoulli)	2-class
softmax	multi-class CE	K-class

three notes, three marks

1. all-linear layers collapse: $\sum_j w^{(2)}_{kj}w^{(1)}_{ji}=W_{ki}$ · 2. output $f$ only interprets range · 3. power lives in hidden nonlinearity.

two hidden families

Ridge (σ, tanh — global directions) vs RBF $e^{-\|\mathbf x-\boldsymbol\mu\|^2/2s^2}$ (local regions). tanh $=2\sigma(2a)-1$ — same model class.

From fixed features to learned features — the full motivation

The chapter opens where GLMs stop. Data that is hopeless in raw $\mathbf x$-space (a central class ringed by two outer clusters) becomes linearly separable after mapping through two hand-placed Gaussian bumps $\phi_1,\phi_2$: in $(\phi_1,\phi_2)$-space one straight line splits the classes. The course's worked examples push this further: logistic regression trained on a 6-D quadratic feature space and a perceptron on a 10-D cubic feature space both recover elliptical boundaries in the original plane. Recall the logistic-regression machinery, because backprop reuses it: $p(\mathcal C_1\mid\phi)=\sigma(\mathbf w^{T}\phi)$, Bernoulli likelihood $\prod_n y_n^{t_n}(1-y_n)^{1-t_n}$, and — thanks to $y'_n=y_n(1-y_n)$ cancelling both denominators — the gradient collapses to $\nabla E=\sum_n (y_n-t_n)\,\phi_n$. The catch: feature engineering does not scale, and the number of monomials explodes with dimension. So replace $\phi_i(\mathbf x)$ by $\phi_i(\mathbf x,\mathbf w^{i})$ — each basis function gets its own adjustable weight vector, i.e. it becomes a perceptron-like unit. The hidden layer is the learned feature map; the MLP is "two stacked levels of linear regression/classification".

What hidden units actually build

Three PDF demonstrations to be able to sketch:

Regression: just 3 tanh hidden units plus a linear output reproduce $y=x^2$, $y=\sin x$ and even $y=|x|$ almost exactly on an interval. The dashed curves are the individual $z_i=\tanh(w^{(1)}_{1i}x+w^{(1)}_{0i})$; the output is the plain weighted sum $y=w^{(2)}_0+w^{(2)}_1z_1+w^{(2)}_2z_2+w^{(2)}_3z_3$. Every smooth target is assembled from shifted, scaled sigmoid ramps — the universal-approximation intuition.
Classification (Bishop's synthetic example): 2 inputs, 2 tanh hidden units, one logistic output. Each hidden unit's $z=0.5$ contour is a straight dashed line; the network's $y=0.5$ boundary is a curve that closely tracks the Bayes-optimal boundary computed from the generating distributions. Curved boundaries emerge purely from combining ridge units.

The three activation notes, unpacked

Linearity collapses: nesting identity (or any linear map) gives nothing beyond a linear model — the product of weight matrices is one weight matrix, exactly the contraction shown on the slide. Depth without nonlinearity is cosmetic.
Output activation only interprets: since $f$ is monotone, $y=\text{const}$ iff $a=\text{const}$, so $f$ never changes the intrinsic boundary shape; choose it to make the output readable as the desired quantity (probability, real value).
Hidden activation is the power: sigmoid-like functions (logit, tanh, erf, sign) suit classification and regression; locally supported functions (Gaussian, multi-quadric RBFs) suit regression and clustering. Logit and tanh are linearly equivalent — $\tanh(a)=2\sigma(2a)-1$ — so they span the same model class, only weights and biases rescale.

Why the loss table is forced, not chosen

The pairings on the slide come from maximum likelihood, not taste. Regression: assume Gaussian observation noise, $p(t\mid\mathbf x,\mathbf w)=\mathcal N(t\mid y(\mathbf x,\mathbf w),\sigma^2)$; then $\hat{\mathbf w}_{\text{ML}}=\arg\max p=\arg\min E$ with $E(\mathbf w)=\tfrac12\sum_n\{y(\mathbf x_n,\mathbf w)-t_n\}^2$, because $-\ln\mathcal N$ is the squared error over $2\sigma^2$ plus constants, and $\sigma^2$ does not move the argmin. Two-class: Bernoulli $p(t\mid\mathbf x,\mathbf w)=y^t(1-y)^{1-t}$ with $y=\sigma(a)$ yields $E(\mathbf w)=-\sum_n\{t_n\ln y_n+(1-t_n)\ln(1-y_n)\}$. Same architecture, different noise model, different loss. Either way $E$ has no closed-form minimizer and multiple minima — training is numerical optimization from the start.

from: ann-mlp-backpropagation.pdf · ch 5

CH 05 · MLP · the exam derivation

Backprop in four boxed steps

1 · local rule — gradient = δ × source activation $$ \frac{\partial E_n}{\partial w_{ki}}=\delta_k\,z_i \qquad \delta_k\equiv\frac{\partial E_n}{\partial a_k} $$

δ is sensitivity to the net input, not the weight gradient itself.

2 · output δ — all canonical pairs collapse $$ \delta_k = y_k - t_k $$

identity+SSE, sigmoid+BCE, softmax+CE — all give the same line (softmax: $\sum_k t_k{=}1$ one-hot).

3 · hidden δ — the backward recurrence $$ \boxed{\;\delta_i = h'(a_i)\sum_k w_{ki}\,\delta_k\;} $$

Sum over downstream units; reuses forward weights in reverse. Deep nets: apply per layer, $\delta^{(l)}_i=h'(a^{(l)}_i)\sum_k w^{(l+1)}_{ki}\delta^{(l+1)}_k$.

4 · first layer + update $$ \frac{\partial E_n}{\partial w^{(1)}_{ji}}=\delta_j\,x_i \qquad \mathbf w\leftarrow\mathbf w-\eta\,\nabla E $$

tanh: $h'(a)=1-z^2$. Cost: one multiply per weight — vs finite differences' full pass per weight.

Setup: per-pattern errors, and why $\delta$ lives on the net input

For iid data the error decomposes, $E(\mathbf w)=\sum_n E_n(\mathbf w)$ and $\nabla E=\sum_n\nabla E_n$, so backprop only ever needs one pattern's gradient $\partial E_n/\partial w$; batch training sums them, sequential training updates per pattern ("iterate ad lib"). The course stresses that backprop efficiently computes both the error value and its gradient in one forward+backward sweep. The pivotal observation behind step 1 on the slide: $w_{ki}$ influences $E_n$ only through the net input $a_k=\sum_i w_{ki}z_i$. Hence one clean chain-rule factorization, $\partial E_n/\partial w_{ki}=(\partial E_n/\partial a_k)(\partial a_k/\partial w_{ki})$, and because $a_k$ is linear in the weights, $\partial a_k/\partial w_{ki}=z_i$ exactly. Defining $\delta_k$ on the net input means one stored scalar per unit instead of one computation per weight — that is the whole efficiency trick. Bias weights ride along free: their "source activation" is $z_0=1$, so their gradient is $\delta_k$ itself.

Output deltas, derived case by case

Identity + SSE: $y_k=f(a_k)=a_k$, $E_n=\tfrac12\sum_k(y_{kn}-t_{kn})^2$, so $\delta_k=(y_{kn}-t_{kn})\cdot 1$ — the PDF writes the trailing 1 to show $f'=1$ explicitly.
Softmax + multiclass CE: $y_k=\exp(a_k)/\sum_l\exp(a_l)$, $E_n=-\sum_k \mathbf 1[t_n{=}k]\,\ln y_{kn}$ with one-hot targets $t_{kn}$. Using the softmax Jacobian $\partial y_k/\partial a_l=y_k(\mathbf 1[k{=}l]-y_l)$ and $\sum_k t_{kn}=1$: $\delta_l=y_{ln}\sum_k t_{kn}-t_{ln}=y_{ln}-t_{ln}$.
Sigmoid + BCE: $\partial E_n/\partial y=-t/y+(1-t)/(1-y)$ multiplied by $y'=y(1-y)$ gives $y-t$. Numeric check: $a=0.5$, $y=\sigma(0.5)=0.6225$, $t=1$: the long route gives $(-1/0.6225)\cdot 0.6225\cdot 0.3775=-0.3775=y-t$.

The collapse is engineered: each output activation is the canonical link of its likelihood, so the $f'(a)$ factor cancels. A mismatched pair, e.g. sigmoid with SSE, leaves $\delta_k=(y_k-t_k)\,y_k(1-y_k)$ — near zero when the unit is saturated and wrong, hence painfully slow learning. Common exam trap: assuming $\delta=y-t$ holds for every activation/loss combination.

Hidden deltas: the chain rule through downstream net inputs

A hidden $a_i$ reaches $E_n$ only via every downstream net input $a_k$, so the total derivative sums over them: $\delta_i=\sum_k(\partial E_n/\partial a_k)(\partial a_k/\partial a_i)=\sum_k\delta_k\,\partial a_k/\partial a_i$. Then split the inner factor along the only path, $a_i\to z_i\to a_k$: $\partial a_k/\partial a_i=(\partial a_k/\partial z_i)(\partial z_i/\partial a_i)=w_{ki}\,h'(a_i)$. Since $h'(a_i)$ is independent of $k$ it factors out, giving the boxed recurrence on the slide. Three things to say about it under exam pressure: forward fan-out becomes backward fan-in; the very same weights $w_{ki}$ from the forward pass are reused in reverse; $h'$ is evaluated at the unit's own cached net input — for tanh $h'(a_i)=1-z_i^2$, for logistic $z_i(1-z_i)$, both free from stored forward values. The three-step algorithm: feed forward (get all $z_i,y_k$ and $E_n$), compute output $\delta$s and propagate back to get every gradient, iterate.

Vector form, cost, and the vanishing-gradient corollary

Layer/Jacobian form: $\boldsymbol\delta^{(l)}=h'(\mathbf a^{(l)})\odot\big(W^{(l+1)T}\boldsymbol\delta^{(l+1)}\big)$, gradient matrix $\nabla_{W^{(l)}}E_n=\boldsymbol\delta^{(l)}\,\mathbf z^{(l-1)T}$ — the outer product of "delta in" and "activation below".
Cost: one multiply–accumulate per weight per direction, so $O(W)$ total; central finite differences need a full forward pass per weight, $O(W^2)$, and survive only as a gradient-checking tool.
Vanishing gradients: each backward layer multiplies by $h'(a)$ with $|h'(a)|\le 1$ for tanh and $\le 1/4$ for the logistic; through many (especially saturated) layers the recurrence shrinks $\delta$ geometrically — read it straight off the boxed formula.

from: ann-mlp-backpropagation.pdf · ch 5

CH 05 · MLP · symmetry · init · early stop

Non-convex, symmetric, overfittable

weight-space symmetry $$ 2^M\,M! \;\text{ equivalent minima} $$

permute hidden units (M!) × sign-flip odd activations ($2^M$). M=5 → 3840 copies of every minimum.

never zero-init

Equal weights → equal δ → identical updates forever → one effective hidden unit. Random init breaks the symmetry.

early stopping = capacity control by epochs

rules of the loss

$E(\mathbf w)$ non-convex → only local minima guaranteed; use random restarts. Validation set touches no gradients.

worked backprop numbers

x=(1, .5), tanh hidden: $z_1{=}.2913$, y=.1222, t=1 → $\delta_y{=}-.8778$, $\delta_1=h'(a_1)w^{(2)}_{11}\delta_y=-.4017$ — know how each number is produced.

The loss surface, honestly described

In high-dimensional weight space even simply-written functions ripple with local minima — the PDF's surface plot makes the point visually. Analytic minimization is impossible, so training is literally $\hat{\mathbf w}=\text{minimize}(E,\mathbf w_0)$: the course explicitly delegates the optimizer to numerical libraries. What you control is the error function, the architecture, and the starting point $\mathbf w_0$ — which is why initialization and restarts carry real weight in this chapter.

Deriving the $2^M M!$ count

Permutation: relabel hidden units together with their weight vectors — swap rows of $W^{(1)}$ and, matching, the columns of $W^{(2)}$. The sum over $j$ is reordered, $y(\mathbf x)$ is untouched: $M!$ equivalent weight settings.
Sign flip: tanh is odd, $h(-a)=-h(a)$. Negate all in-weights of hidden unit $j$: $z_j$ flips sign. Negate its out-weights $w^{(2)}_{kj}$ too: every product $w^{(2)}_{kj}z_j$ is restored. One binary choice per hidden unit gives $2^M$ configurations; combined with permutations, $2^M M!$.

The PDF frames this as good news: a huge fraction of the "many local minima" are identical in function space, so they never need to be distinguished — landing in any one of the $2^M M!$ mirror copies is exactly as good as any other. With $M=10$ that is $2^{10}\cdot 10!\approx 3.7\times 10^9$ copies of every genuine minimum.

Initialization: the zero-init failure, step by step

Equal weights into every hidden unit $\Rightarrow$ all $a_j$ equal $\Rightarrow$ all $z_j$ equal.
Backward: $\delta_j=h'(a_j)\sum_k w_{kj}\delta_k$ is the same number for every $j$ (same $h'(a_j)$, same outgoing weights).
So every row of the gradient $\partial E_n/\partial w^{(1)}_{ji}=\delta_j x_i$ is identical, the update preserves equality, and by induction the network remains one effective hidden unit forever.

Random initialization breaks the tie (and merely selects which symmetric basin you enter — irrelevant by the argument above). Scale matters too: large initial weights push $|a_j|$ into tanh saturation where $h'(a)=1-z^2\approx 0$, so $\delta_j\approx 0$ and the unit stops learning before it starts — hence small random values that keep net inputs in the active region.

Capacity, overfitting, and the early-stopping protocol

Model complexity is set by the hidden layer. The PDF's sinusoid experiment: $M=1$ underfits (a single ramp), $M=3$ fits cleanly, $M=10$ threads every noisy point. Training error is monotone in $M$, so $M$ must be chosen on held-out data. The early-stopping plots show the exact protocol: training error decreases essentially monotonically with epochs, validation error falls, flattens, then turns upward; the dashed stop line sits at the validation minimum (near epoch 28 in the PDF's run). Interpretation for the exam: the effective complexity of the network grows with training time, so stopping early is capacity control without touching the architecture. The validation set steers only the stopping decision — never the gradients — and a final untouched test set estimates generalization.

Finishing the slide's worked numbers

The slide stops at $\delta_1=-.4017$; an exam answer should finish with every gradient. With $z_1=.2913$ (so $a_1=0.3$), $h'(a_1)=1-z_1^2=.9151$, and indeed $w^{(2)}_{11}=\delta_1/(h'(a_1)\,\delta_y)=-.4017/(.9151\times(-.8778))=0.5$. Then: $\partial E/\partial w^{(2)}_{11}=\delta_y z_1=-.8778\times.2913=-.2557$; output bias gradient $=\delta_y=-.8778$; $\partial E/\partial w^{(1)}_{11}=\delta_1x_1=-.4017$; $\partial E/\partial w^{(1)}_{12}=\delta_1x_2=-.2009$; hidden bias gradient $=\delta_1$. Update with $\eta=0.1$: $w^{(2)}_{11}\leftarrow 0.5-0.1\times(-.2557)=.5256$. Sanity check the signs: $t=1$ but $y=.1222$, the net under-predicts, $\delta_y\lt 0$, so gradient descent raises weights fed by positive activations — exactly what the numbers do.

from: ann-mlp-backpropagation.pdf · ch 5

CH 06 · SGD · three batch regimes

Noisy gradients are a feature

Training loss is a surrogate — the true target is unseen-data loss, so cheap noisy gradient estimates are legitimate.

	batch	stochastic	mini-batch
m	N	1	32–512
gradient	exact	unbiased, very noisy	noise ∝ 1/√m
cost/update	O(N)	O(1)	O(m)
near minimum	step → 0	noise ball — needs η decay

sgd update + linear decay $$ \hat g\leftarrow\tfrac1m\nabla_\theta\textstyle\sum_i L_i,\quad \theta\leftarrow\theta-\epsilon_k\hat g \qquad \epsilon_k=(1{-}\alpha)\epsilon_0+\alpha\epsilon_\tau,\ \alpha=\tfrac{k}{\tau} $$

robbins–monro — when does SGD converge $$ \lim_{k\to\infty}\epsilon_k=0,\qquad \sum_k\epsilon_k=\infty,\qquad \sum_k\epsilon_k^2\lt\infty $$

Steps shrink, total travel unbounded, total noise variance bounded.

why mini-batch — both reasons for full credit

1. vectorized hardware (matrix–matrix ≫ matrix–vector) · 2. 1/√m noise reduction with diminishing returns (100× data → only 10× less noise). Stability: $\epsilon\lt 2/H$.

Learning is not optimization

The course frames this chapter (گرادیان کاهشی تصادفی — stochastic gradient descent, stabilized with تکانه, momentum) as "techniques for improving learning efficiency based on gradient-descent optimization". Behind the slide's surrogate callout: the quantity we care about is expected loss under the data distribution — generalization — and the training set is only a sample of it. Paying $O(N)$ per step for an exact gradient of a proxy objective buys precision we don't need; a cheap unbiased estimate of the same expectation moves us just as reliably, and makes thousands of updates in the time one full pass takes. Per minibatch the loss (تابع زیان) is literally additive: loss(mbatch) $=\sum_{\text{sample}\in\text{mbatch}}$ loss(sample).

Unbiasedness and the variance calculation

Why "unbiased" in the slide's table: draw index $i$ uniformly from $\{1,\dots,N\}$; then $\mathbb E[\nabla L_i]=\frac1N\sum_j\nabla L_j$, exactly the full-batch gradient. Even $m=1$ SGD points the right way on average. Averaging $m$ i.i.d. draws keeps the mean and divides the variance by $m$:

$\mathrm{Var}[\hat g]=\sigma^2/m$ — variance shrinks like $1/m$; standard error $\sigma/\sqrt m$ is the slide's $1/\sqrt m$ noise;
hence sub-linear returns: $100\times$ the per-step cost buys only $10\times$ less noise — the statistical half of the mini-batch argument;
the i.i.d. argument assumes sampling with replacement; shuffled epochs are the benign practical approximation.

Choosing ε: the three-picture story

The reference slides give a full page to the qualitative regimes: too small a rate "takes forever"; too large, and the next point perpetually bounces haphazardly across the bottom of the well, never settling; and if the gradient is small you can safely try a larger rate, which compensates for the small gradient and restores a useful step. This is why $\epsilon$ is the crucial parameter of the loop, indexed per iteration as $\epsilon_k$.

Why decay — and which schedules qualify

Batch GD needs no decay: the true gradient of the total cost becomes small and then $0$ at a minimum, so a fixed rate self-anneals. SGD has a permanent noise source — random sampling of the $m$ examples — so the gradient estimate does not vanish at the minimum and the iterate rattles in a noise ball unless $\epsilon_k$ itself shrinks. Test schedules against the slide's Robbins–Monro conditions: $\epsilon_k\propto 1/k$ satisfies all three; a constant rate fails $\epsilon_k\to 0$; $\epsilon_k\propto 1/k^2$ dies too fast — its sum converges, total travel is bounded, and a distant minimum may be unreachable. The linear schedule in the slide's box runs only until iteration $\tau$; afterwards the rate is held constant at the small positive floor $\epsilon_\tau$. Keras equivalent, recomputed at each update (end of each mini-batch): lrate = initial_lrate · 1/(1 + decay · iteration), e.g. SGD(lr=0.01, momentum=0.9, decay=0.01); the bare default is lr = 0.01 with no momentum.

Exam checkpoints

Unbiased ≠ accurate: bias is zero for every $m$; only variance changes with batch size.
Quote scales precisely: standard deviation $\propto 1/\sqrt m$, variance $\propto 1/m$.
The convergence conditions are sufficient, not necessary.
The cap $\epsilon\lt 2/H$ is a separate deterministic constraint from curvature: decay handles noise, the cap handles overshoot — do not merge them in an answer.

from: SGD & momentum slides · ch 6

CH 06 · SGD · heavy ball 1964 · nesterov 1983

Velocity remembers ≈ 1/(1−α) gradients

classical momentum $$ v\leftarrow\alpha v-\epsilon\,\nabla_\theta J(\theta),\qquad \theta\leftarrow\theta+v $$

Sign trap: $v$ already contains $-\epsilon g$, so update is $\theta{+}v$. Unrolled: $v_t=-\epsilon\sum_i\alpha^i g_{t-i}$ — an EWMA. α=0.9 → horizon ≈10; α=0.99 → ≈100.

nesterov — look ahead first $$ \tilde\theta=\theta+\alpha v,\qquad v\leftarrow\alpha v-\epsilon\,\nabla_{\tilde\theta}J(\tilde\theta),\qquad \theta\leftarrow\theta+v $$

Only difference: gradient at $\tilde\theta$, not θ. First step identical when $v_0=0$. Faster on smooth convex problems.

momentum helps exactly when

high curvature	opposing gradients cancel → no zig-zag
small steady g	accumulates → bigger step
noisy g	averaged over horizon

the oscillation demo

$L=0.5w_1^2+10w_2^2$, ε=0.1: plain GD gives $w_2\leftarrow-w_2$ — oscillates forever; momentum (α=0.5) kills it in 2 steps. Cause: one shared ε capped by the steepest Hessian direction.

practice

SGD(lr=.01, momentum=.9, nesterov=True)

The physics of the name (تکانه)

Momentum in physics is mass times velocity; the algorithm assumes unit mass, so the new state variable $v$ is simultaneously the momentum and the velocity — the direction and speed at which the parameters move through parameter space. The hyperparameter $\alpha\in[0,1)$ is an exponential-decay (friction) coefficient, and what matters is $\alpha$ relative to $\epsilon$: the larger $\alpha$ is compared with $\epsilon$, the more previous gradients dominate the freshly computed one in setting the current direction.

Terminal velocity: deriving the 1/(1−α) headline

Feed the slide's unrolled EWMA a constant gradient $g$ — a long plateau of small, consistent slope. The velocity update has a fixed point, the geometric-series limit $v^\ast=-\dfrac{\epsilon\,g}{1-\alpha}$, so at steady state momentum behaves like plain SGD with effective rate $\epsilon/(1-\alpha)$: $10\times$ larger at $\alpha=0.9$, $100\times$ at $\alpha=0.99$. Numerically with $\epsilon=0.01$, $\alpha=0.9$, $g=1$: step sizes grow $0.01,\ 0.019,\ 0.0271,\dots\to 0.1$. This is exactly the "small but consistent gradients" case the course lists as a momentum win.

Poor conditioning: what the contour figure shows

The reference slides pair two contour plots of a quadratic loss with a poorly conditioned Hessian — a long narrow valley with steep sides. Plain SGD takes large steps across the valley and small steps along it, so its path zig-zags across the contours and "wastes time". Inside the velocity sum, the across-valley gradient components alternate sign between iterations and cancel, while the along-valley components always agree and accumulate — the momentum path cuts smoothly across contours toward the minimum. Practical consequence stated by the course: with oscillation damped, you can afford a higher learning rate than plain SGD tolerates.

Nesterov: order of operations is the exam answer

Write the loop in the course's order and mark what is new: (1) interim update $\tilde\theta\leftarrow\theta+\alpha v$ — the only line added to plain momentum; (2) gradient of the minibatch loss evaluated at the interim point $\tilde\theta$; (3) velocity update $v\leftarrow\alpha v-\epsilon g$; (4) apply $\theta\leftarrow\theta+v$. Interpretation emphasized in the course: the gradient is evaluated after the current velocity is applied, so Nesterov acts as a correction factor to standard momentum — if the inherited velocity is about to overshoot, the lookahead gradient already points back before the step is committed. $\alpha$ and $\epsilon$ keep the same roles as in the standard method.

Practice, defaults, pitfalls

Keras: momentum is a float that "accelerates SGD in the relevant direction and dampens oscillations"; nesterov is a boolean. Course example: SGD(lr=0.01, decay=1e-6, momentum=0.9, nesterov=True).
Both algorithm boxes require an explicit initial velocity ($v=0$); with it, classical and Nesterov take an identical first step.
Momentum is not just "a bigger $\epsilon$": the $\epsilon/(1-\alpha)$ equivalence holds only while gradients agree; on sign flips the two diverge completely — that is the point of the slide's oscillation demo.
Symbol clash: this chapter's $\alpha$ is the momentum coefficient; the $\alpha=k/\tau$ on the previous slide is a decay interpolation fraction. Unrelated quantities.

from: SGD & momentum slides · ch 6

CH 07 · Adaptive rates · live race, real update rules

One η can't fit an anisotropic bowl

L = 0.05·w₁² + w₂² · start (−9, 2.6)

■ SGD · ■ momentum · ■ RMSProp · ■ Adam

adagrad — accumulate forever $$ r\leftarrow r+g\odot g,\qquad \Delta\theta=-\frac{\epsilon}{\delta+\sqrt r}\odot g $$

Equalizes scales, but $r$ only grows → η → 0 → premature stall on non-convex problems.

rmsprop — forget with an ewma $$ r\leftarrow\rho r+(1-\rho)\,g\odot g,\qquad \Delta\theta=-\frac{\epsilon}{\sqrt{\delta+r}}\odot g $$

ρ=0.9. Accumulator saturates at $g^2$ → effective rate stabilizes non-zero. All ops elementwise ⊙ — never $g^Tg$.

Why per-parameter rates at all (نرخ یادگیری تطبیقی)

The course opens by calling the learning rate the most difficult hyperparameter to set: the cost is highly sensitive to some directions in parameter space and nearly insensitive to others. Momentum treats that anisotropy but adds a second hyperparameter. The escape rests on one explicit assumption — if the directions of sensitivity are axis-aligned, it suffices to keep a separate learning rate per parameter and adjust it throughout learning: a diagonal shortcut that buys curvature-like behaviour without a Hessian.

Delta-bar-delta: the full-batch ancestor

The earliest heuristic in the chapter works per coordinate: if $\partial L/\partial w_i$ keeps the same sign across iterations, that parameter's private rate should increase; if the sign flips, it should decrease. The margin sketch reduces it to sign-driven steps $w_i\leftarrow w_i-\eta_0\,\mathrm{sign}(g_i)$, moving $\pm\eta_0$ per coordinate. It is branded full-batch (GD) only because under mini-batch sampling a sign flip is usually noise, not evidence of overshoot — the rule would keep cutting rates for no reason. AdaGrad, RMSProp and Adam are the "recent incremental mini-batch methods" that make the idea noise-tolerant.

AdaGrad under the microscope

The per-coordinate effective rate implied by the update box on the slide is $$\eta_i^{(t)}=\frac{\epsilon}{\delta+\sqrt{\big(g_i^{(1)}\big)^2+\big(g_i^{(2)}\big)^2+\cdots+\big(g_i^{(t)}\big)^2}}$$ with $\delta\approx 10^{-7}$ purely for numerical stability and every operation element-wise (ضرب مؤلفه‌به‌مؤلفه). Coordinates with a large gradient history get small rates; rarely-active ones keep large rates. The annotated course copy poses two check-yourself questions: does $\eta_i^{(t)}\to 0$ as $t\to\infty$, and does $\sum_t\eta_i^{(t)}=\infty$ — i.e. test the induced schedule against chapter 6's convergence conditions. If a coordinate's gradients stay bounded away from zero, $r_i$ grows linearly in $t$, so $\eta_i^{(t)}\sim 1/\sqrt t$: it vanishes, yet its sum diverges. The same formula explains the failure mode: the accumulator also stores the large gradients of the early transient, so on non-convex problems the rate collapses before the iterate reaches a good region. Course verdict: "performs well for some but not all deep learning."

RMSProp: closed form and why forgetting fixes it

Expanding the EWMA recursion gives the handwritten closed form $$r^{(T)}=(1-\rho)\sum_{t=1}^{T}\rho^{\,T-t}\big(g^{(t)}\big)^2,$$ an exponential window of width $\approx 1/(1-\rho)$ — about the 10 most recent minibatches at $\rho=0.9$. Old history is discarded, so after a long non-convex journey RMSProp behaves like AdaGrad freshly initialized inside the current bowl; the course notes it converges rapidly once applied to a (locally) convex region, exactly where AdaGrad's guarantees hold. The stabilizer is larger here, $\delta\approx 10^{-6}$, and sits inside the root in the course's box. The PDF also spells out RMSProp with Nesterov momentum: interim point $\tilde\theta=\theta+\alpha v$, gradient there, EWMA accumulation, then $v\leftarrow\alpha v-\frac{\epsilon}{\sqrt r}\odot g$ and $\theta\leftarrow\theta+v$ — adaptivity sets the scale, momentum the direction; the mechanisms compose.

Defaults to memorize

RMSprop(learning_rate=0.001, rho=0.9) — described as the go-to practical method.
Adagrad(learning_rate=0.01) — note the 10× larger default rate; the accumulator will shrink it.

from: adaptive learning-rate slides I–II · ch 7

CH 07 · Adaptive rates · kingma & ba 2015

Adam = RMSProp + momentum + bias correction

adam — the full update $$ s\leftarrow\rho_1 s+(1{-}\rho_1)g \qquad r\leftarrow\rho_2 r+(1{-}\rho_2)g\odot g $$ $$ \hat s=\frac{s}{1-\rho_1^{\,t}},\quad \hat r=\frac{r}{1-\rho_2^{\,t}} \qquad \Delta\theta=-\epsilon\frac{\hat s}{\sqrt{\hat r}+\delta} $$

Why correct: zero-init EWMAs underestimate by $(1-\rho^t)$ — $\mathbb E[s^{(t)}]=\mathbb E[g](1-\rho_1^t)$; dividing removes the bias exactly, matters most at small $t$.

ε=0.001ρ₁=0.9ρ₂=0.999δ=1e-8

method	one-line identity
Delta-bar-delta	grow η if sign stable — full-batch only
AdaGrad	÷√(sum g²) — rate dies
RMSProp	÷√(EWMA g²) — the go-to
Adam	+ 1st moment + bias fix
AdaDelta	numerator = RMS(Δθ) — no ε to tune
AdaMax	ℓ∞ norm: u←max(ρ₂u,\|g\|)
Nadam	Adam + Nesterov look-ahead

empirics — no universal winner

Adam fastest early; tuned SGD+Nesterov often lowest at epoch 45; AdaGrad lags badly non-convex. Adaptive methods damp dropout noise.

Reading the algorithm box precisely

Details of the full Adam ("adaptive moments") box that the slide's two display lines compress: the step counter starts at $t=0$ and is incremented before the moment updates, so the first correction already uses $t=1$; both accumulators are zero-initialized ($s=0$, $r=0$); all operations are element-wise; and $\delta=10^{-8}$ is added outside the root, $\sqrt{\hat r}+\delta$ — unlike RMSProp's inside-the-root placement in the course's boxes. $s$ estimates the gradient's first moment (mean — the momentum role), $r$ its second raw moment (the RMSProp role); Keras names $\rho_1,\rho_2$ as beta_1, beta_2.

A structural contrast with chapter 6: $s$ is a convex combination of past gradients (weights $(1-\rho_1)\rho_1^{k}$ sum to 1 in the limit), so it never exceeds the scale of recent gradients, whereas the momentum velocity $v$ carries a factor $-\epsilon$ and can grow to $\epsilon/(1-\alpha)$ times a gradient. Dividing mean by RMS makes each coordinate's step roughly $\pm\epsilon$ whenever gradients are consistent — why $\epsilon=0.001$ is safe and Adam is "generally robust to the choice of hyperparameters".

Bias correction: the actual derivation

Unroll the recursion from $s^{(0)}=0$: $s^{(t)}=(1-\rho_1)\sum_{i=1}^{t}\rho_1^{\,t-i}g^{(i)}$. For an (approximately) stationary gradient, take expectations and sum the geometric series: $\mathbb E[s^{(t)}]=\mathbb E[g]\,(1-\rho_1^{t})$. The estimator is shrunk toward zero by exactly $(1-\rho_1^t)$, so dividing by that factor is an exact fix, not an approximation; identically for $r$ with $\rho_2$. Worked numbers at $t=1$ with the defaults:

$s=0.1\,g$ (10× underestimate) and $r=0.001\,g\odot g$ (1000× underestimate);
uncorrected step: $\epsilon\cdot 0.1g/\sqrt{0.001\,g^2}\approx 3.16\,\epsilon\,\mathrm{sign}(g)$ — the mismatched biases make early steps about 3× too large;
corrected: $\hat s=g$, $\hat r=g\odot g$, step $=\epsilon\,\mathrm{sign}(g)$ as designed;
as $t\to\infty$ both factors tend to 1, but with $\rho_2=0.999$ the second-moment bias stays material for roughly the first 1000 steps — far longer than the first moment's ≈10.

The rest of the family, with Keras signatures

AdaDelta: the numerator becomes an RMS of recent parameter updates, so no free step size remains — hence Adadelta(learning_rate=1.0, rho=0.95): the "learning rate" is a mere multiplier on a self-scaled step.
AdaMax: second moment tracked by an $\ell_\infty$ running maximum instead of a squared average; Adamax(learning_rate=0.002, beta_1=0.9, beta_2=0.999).
Nadam: the chapter-6 lookahead applied inside Adam; Nadam(learning_rate=0.002, beta_1=0.9, beta_2=0.999).
Adam itself ships with an amsgrad=False switch in Keras.

What the benchmark figures actually show

MNIST logistic regression, 45 epochs: SGD-Nesterov and Adam nearly coincide near cost 0.26; AdaGrad sits far above throughout.
IMDB bag-of-words logistic regression, 10,000 sparse features with dropout: SGD-Nesterov+dropout stays noisy near 0.35 while AdaGrad, RMSProp and Adam cluster near 0.25 — per-feature rates suit sparse inputs, and adaptivity damps dropout's gradient noise.
CIFAR-10 ConvNet (c64-c64-c128-1000): Adam leads the first three epochs, but by epoch 45 SGD-Nesterov and Adam both reach $\approx 10^{-3}$ training cost while AdaGrad stalls near $10^{0}$.

Closing verdict of the course: no consensus. The actively used set is SGD, SGD+momentum, RMSProp, RMSProp+momentum, AdaDelta and Adam — and familiarity with tuning the one you pick is a legitimate selection criterion.

from: adaptive learning-rate slides I–II · ch 7

CH 08 · Regularization · penalized loss

Shrink by percent or shrink by amount

l2 · weight decay $$ L+\tfrac{\lambda}{2}\textstyle\sum_j w_j^2 \;\Rightarrow\; w^{new}=(1-\eta\lambda)\,w^{old}-\eta\nabla_w f $$

Constant percentage shrink — never exact zeros. Gaussian prior, $\lambda=1/\sigma^2$. Unhalved penalty ⇒ $(1-2\eta\lambda)$ — watch the convention.

l1 · sparsity $$ L+\lambda\textstyle\sum_j|w_j| \;\Rightarrow\; w\leftarrow w-\eta\lambda\,\mathrm{sign}(w)-\eta\nabla_w f $$

Constant amount $\eta\lambda$ per step → exact zeros → feature selection. Laplace prior, $\lambda=1/b$.

never penalize biases

Biases don't multiply inputs — penalizing them just shifts outputs wrong.

diagnosis	signature	fix
underfit	both errors high	capacity / optimize
overfit	train low, val high (gap)	regularize

The course formulation, layer by layer

The reference slides write weight regularization (منظم‌سازی وزن) as a penalty over all layers: $\Omega(\theta)=\sum_k\sum_i\sum_j\big(W^{(k)}_{i,j}\big)^2=\sum_k\lVert W^{(k)}\rVert_F^2$ — the squared Frobenius norm of every weight matrix — with gradient $\nabla_{W^{(k)}}\Omega=2W^{(k)}$ (unhalved convention). The lecture's regression loss is $\mathrm{Loss}(w)=\frac{1}{N}\sum_{i=1}^{N}\big(y^{(i)}-\hat y(x^{(i)};w)\big)^2+\frac{\lambda}{2}\sum_j w_j^2$, where $\lambda$ is the regularization coefficient (ضریب منظم‌سازی). In the SGD update the shrink factor $\alpha=\eta\lambda$ is what the course literally labels "weight decay" — the name refers to the multiplicative decay $(1-\alpha)$, not to the penalty term itself.

MAP view: penalties are priors

Maximizing the posterior $p(w\mid D)\propto p(D\mid w)\,p(w)$ and taking $-\log$ gives (negative log-likelihood) $+$ ($-\log$ prior). The choice of prior fixes the penalty:

Gaussian $w_j\sim\mathcal N(0,\sigma^2)$: $-\log p(w)=\frac{1}{2\sigma^2}\sum_j w_j^2+\text{const}$, i.e. L2 with $\lambda=1/\sigma^2$. A tighter prior (small $\sigma$) means larger $\lambda$, stronger shrinkage.
Laplace $p(w_j)=\frac{1}{2b}e^{-|w_j|/b}$: $-\log p(w)=\frac1b\sum_j|w_j|+\text{const}$, i.e. L1 with $\lambda=1/b$. The Laplace density has a sharp (non-differentiable) peak at $0$ — exactly why its MAP solutions can sit exactly at zero, while the Gaussian is flat at its peak and only pulls weights near zero.

Eigenvalue shrinkage: what L2 actually does

Approximate the unregularized loss quadratically around its minimizer $w^{*}$: $\hat L(w)=L(w^{*})+\tfrac12(w-w^{*})^{\top}H(w-w^{*})$. Adding $\frac{\alpha}{2}\lVert w\rVert^2$ and setting the gradient to zero gives

$$\tilde w=(H+\alpha I)^{-1}H\,w^{*}\;\;\Rightarrow\;\; \tilde w_i=\frac{\lambda_i}{\lambda_i+\alpha}\,w_i^{*}$$

in the eigenbasis of $H$. Directions with high curvature $\lambda_i\gg\alpha$ (strongly determined by the data) are nearly untouched; directions with $\lambda_i\ll\alpha$ collapse toward zero. L2 therefore filters out the poorly determined parameter directions — the precise mechanism behind "shrink by percent."

Why L1 lands exactly on the axes

The PDF defines the (sub)gradient entrywise: $\operatorname{sign}(W^{(k)})_{i,j}=1_{W_{i,j}\gt 0}-1_{W_{i,j}\lt 0}$ — a pull of constant magnitude $\lambda$, independent of $|w|$. With a diagonal Hessian approximation the closed-form solution is soft thresholding:

$$w_i=\operatorname{sign}(w_i^{*})\,\max\Big\{|w_i^{*}|-\frac{\alpha}{H_{ii}},\;0\Big\}$$

Any weight whose unregularized optimum lies within $\alpha/H_{ii}$ of zero is clipped to exactly $0$. Geometrically, the L1 ball is a diamond whose corners sit on the axes, so the loss contours generically touch a corner first (some coordinate $=0$); the L2 ball is smooth, so the tangency point is almost never on an axis. Dynamically: L2's pull $\eta\lambda|w|$ vanishes as $w\to 0$ and never crosses zero, while L1's fixed step $\eta\lambda$ marches the weight through it (implementations clamp at $0$).

Exam traps

"Weight decay $\equiv$ L2" holds only for plain SGD. Adaptive optimizers (Adam) rescale the L2 gradient per coordinate, so the decay is no longer uniform — decoupled weight decay (AdamW) restores it.
Watch the factor of 2: the PDF's $\Omega$ has no $\frac12$, hence gradient $2W^{(k)}$; the halved form on the slide gives $(1-\eta\lambda)$.
The lecture's remedy map: underfitting (کم‌برازش, both errors high) → better optimization, an active research area; overfitting (بیش‌برازش, train–val gap) → better regularization, which the course lists broadly: norm penalties, unsupervised learning / pre-training (GANs noted in class), and stochastic dropout training — regularization is not only penalties.
Neither L1 nor L2 is ever applied to biases.

from: regularization slides · ch 8

CH 08 · Regularization · hinton 2012 · ioffe–szegedy 2015

Cripple the net, normalize the batch

dropout — forward, backward, test $$ h^{(k)}=g(a^{(k)})\odot\mathbf m^{(k)},\quad m_j\!\sim\!\text{Bern}(1{-}p) \qquad \nabla_{a}=\big(\nabla_h\odot g'\big)\odot\mathbf m $$

Same mask forward and backward (fresh mask = corrupted gradient). Output layer never masked. Test: scale by $1{-}p$, or inverted dropout $\tfrac{1}{1-p}$ at train.

why it works

Kills co-adaptation; trains an implicit ensemble of $2^{\#units}$ weight-sharing subnets; p=0.5 default. ≈ adaptive data-dependent L2.

batch norm — per unit, per mini-batch $$ \mu=\tfrac1m\textstyle\sum_i Z_i,\;\; \sigma^2=\tfrac1m\sum_i(Z_i{-}\mu)^2,\;\; \hat Z=\frac{Z-\mu}{\sqrt{\epsilon+\sigma^2}} $$ $$ H=\max\{0,\ \gamma\odot\hat Z+\beta\} $$

Between linear map and nonlinearity. γ, β can undo normalization ($\gamma{=}\sqrt{\epsilon+\sigma^2},\beta{=}\mu$) → capacity unchanged, only optimization dynamics.

test time + fine print

Inference uses running averages, not batch stats → deterministic. Batch noise = incidental regularizer; modern view (2018): benefit = smoother loss landscape, not covariate shift.

Where the mask enters — and where it does not

The PDF's dropout (ریزش) forward pass is deliberately asymmetric. With $h^{(0)}(x)=x$, the pre-activation $a^{(k)}(x)=b^{(k)}+W^{(k)}h^{(k-1)}(x)$ carries no mask (annotated بدون لایه‌ی Dropout, "no dropout layer here"); the random binary mask multiplies only the post-nonlinearity output, $h^{(k)}(x)=g(a^{(k)}(x))\odot m^{(k)}$ for $k=1,\dots,L$ (با لایه‌ی Dropout). The output layer $h^{(L+1)}(x)=o(a^{(L+1)}(x))=f(x)$ is never masked — zeroing prediction entries would corrupt the loss itself. The lecture's worked expectation: with drop probability $p=0.2$, $\mathbb E[m_j]=0\cdot p+1\cdot(1-p)=0.8$ — a unit survives with the keep probability $1-p$, the number every test-time correction is built from.

Backprop: where the mask reappears

The PDF walks the full backward pass. Top gradient before activation: $\nabla_{a^{(L+1)}}\big[-\log f(x)_y\big]=-(e(y)-f(x))$. Then for $k$ from $L{+}1$ down to $1$, the parameter gradients $\nabla_{W^{(k)}}=\big(\nabla_{a^{(k)}}\big)\,h^{(k-1)}(x)^{\top}$ and $\nabla_{b^{(k)}}=\nabla_{a^{(k)}}$ contain the mask implicitly — the stored $h^{(k-1)}$ was already masked in the forward pass (the slide's margin note: "includes the mask $m^{(k-1)}$"). The mask appears explicitly one step lower: $\nabla_{a^{(k-1)}}=\big(\nabla_{h^{(k-1)}}\big)\odot[\dots,g'(a^{(k-1)}_j),\dots]\odot m^{(k-1)}$. Consequences: a dropped unit receives zero gradient, so its weights simply do not update on that example; and the same realized mask must be reused from forward to backward — resampling in between mismatches gradient and function.

Test time: one pass approximates an exponential ensemble

At inference each mask is replaced by its expectation — the constant vector $1-p$ (i.e. $0.5$ for $p=0.5$) — making prediction deterministic. The lecture's justification: for a single hidden layer this expected-mask network computes the (renormalized) geometric mean of the predictions of all $2^{H}$ masked subnetworks. So one forward pass stands in for averaging an exponentially large ensemble of weight-sharing subnets, each trained only on the minibatches where its mask was drawn — hence the co-adaptation argument: no unit can rely on specific partners existing, so each must be independently useful. Equivalent implementations: scale activations (or outgoing weights) by $1-p$ at test, or inverted dropout, dividing by $1-p$ during training so inference code stays untouched (the framework default). Hinton, Srivastava, Krizhevsky, Sutskever & Salakhutdinov (2012) report it beating plain backprop on many datasets; the PDF adds it composes with unsupervised pre-training.

Batch norm mechanics and the covariate-shift story

The course example inserts BN inside a fully connected layer (لایه‌ی تمام‌متصل) with ReLU: after $Z=XW$, center $\tilde Z=Z-\frac1m\sum_{i=1}^{m}Z_{i,:}$ and normalize $\hat Z=\tilde Z/\sqrt{\epsilon+\frac1m\sum_i\tilde Z_{i,:}^2}$, statistics taken per unit across the $m$ examples of the mini-batch (اندازه‌ی mini-batch); $\epsilon$ only guards against $\sigma^2\approx 0$. The motivation is in Ioffe–Szegedy's title: internal covariate shift — each layer's input distribution drifts as earlier layers update, forcing continual re-adaptation; pinning the first two moments gives every layer a stationary input scale, permitting larger learning rates and less delicate initialization. $\gamma,\beta$ are per-unit and learned by backprop, and gradients also flow through $\mu$ and $\sigma^2$, which depend on every example in the batch.

Fine print worth exam points

Under BN the preceding layer's bias is redundant: centering subtracts any constant, and $\beta$ restores a learnable offset — hence the PDF's bias-free $Z=XW$.
Train/eval asymmetry is a classic bug: forgetting eval mode leaves batch statistics on, making predictions depend on batch composition.
Small batches give noisy $\mu,\sigma^2$: more incidental regularization, worse estimates.
Convention check: the PDF's $p$ is the drop probability, $m_j\sim\text{Bern}(1-p)$; some texts define $p$ as the keep probability — verify before scaling.

from: regularization slides · ch 8

Part III · Chapters 9–11

Convolutional
Networks

9 — Convolution, pooling, LeNet-5, AlexNet
10 — Backprop through conv & pooling layers
11 — Design choices: sharing spectrum, heads, ResNet

III

CH 09 · CNNs · the workhorse formula, live

Output size & parameters — drag the sliders

the two formulas that always appear $$ O=\left\lfloor\frac{W-F+2P}{S}\right\rfloor+1 \qquad \#\text{params}=F{\cdot}F{\cdot}C_{in}{\cdot}C_{out}+C_{out} $$

input W7

kernel F3

padding P1

stride S2

forward pass · weight sharing $$ Z_{r,s}=f\!\Big(\sum_{i,j} w_{i-r,\,j-s}\,x_{i,j}\Big) $$

Weights independent of position (r,s) = sharing. DL "convolution" = cross-correlation (no flip) — irrelevant for learned kernels.

blue = kernel footprint · gray ring = zero padding

why not fully connected

Images are stationary — same local statistics everywhere. LeNet C1: conv = 156 params vs FC alternative > 4.8 M.

Where the output-size formula comes from

Pad the input with $P$ zeros on each side: usable width becomes $W+2P$. A kernel of width $F$ can place its left edge at offsets $0,1,\dots,(W+2P-F)$ — that last offset is the rightmost position where the footprint still fits entirely inside. Stride $S$ keeps every $S$-th of these placements, so the count is $\lfloor (W+2P-F)/S \rfloor + 1$: the $+1$ is the starting position, the floor discards a final partial step where the kernel would hang off the edge. Verify on the slide's widget: the blue footprint occupies exactly the positions this count enumerates. Two traps: if $S \nmid (W+2P-F)$ the rightmost columns are simply never covered by any placement (information silently dropped), and the formula applies per dimension — height and width are computed independently.

The three padding modes

Valid ($P=0$): output $W-F+1$; each layer shrinks the map by $F-1$, so deep stacks of valid convs eat the image from the borders inward.
Same ($P=(F-1)/2$, odd $F$, $S=1$): output equals input. The course writes kernels in centered form with width $(2K{+}1)\times(2K{+}1)$ (عرض کرنل, "kernel width"), i.e. $F=2K+1$; then same-padding is simply $P=K$ and the sum $Z_{r,s}=f(\sum w_{i-r,j-s}x_{i,j})$ runs over $r-K \le i \le r+K$.
Full ($P=F-1$): output $W+F-1$; every placement with any overlap counts. This mode reappears in the backprop chapter as the shape of the input gradient.

Volume convolution and counting parameters

On a multi-channel input, one kernel is a $F\times F\times C_{in}$ block: it spans all input channels and produces one output map; $C_{out}$ such kernels plus one bias each give the parameter formula. Crucially the count is independent of $W$ and $H$ — that is weight sharing. Deriving the slide's LeNet numbers: C1 has $6\cdot(5\cdot 5\cdot 1)+6=156$ parameters, while a fully connected layer producing the same $6\cdot 28\cdot 28=4704$ activations from $32\cdot 32=1024$ inputs would need $4704\cdot 1024\approx 4.8\,\mathrm{M}$ weights. Note the asymmetry: parameters don't scale with image size, but activations and compute do — a common exam confusion.

Replicated features and the growing receptive field

The course frames conv layers as the replicated feature approach (مزیت ویژگی‌های تکرارشونده در تصاویر): many positioned copies of one detector, justified because image statistics are stationary. Replication across scale and orientation is possible too but "tricky and expensive," so standard CNNs replicate only across position. Using several feature types, each with its own map, lets every image patch be represented in several ways at once — the layer output is a stack of maps, not one image. Stacking layers grows the receptive field: each $F\times F$ conv adds $F-1$ pixels of context, and every stride-$s$ or pooling stage multiplies subsequent growth by $s$. This is why small kernels deep in a net see globally, supporting the edges → corners/contours → object-parts hierarchy the course illustrates with Goodfellow's Fig 1.2.

from: CNN intro slides · ch 9

CH 09 · CNNs · favorite exam distinction

Equivariant conv, invariant-ish pooling

the distinction $$ \text{equivariance: } g(\mathrm{shift}(x))=\mathrm{shift}(g(x)) \qquad \text{invariance: } g(\mathrm{shift}(x))=g(x) $$

Convolution is equivariant to translation, NOT invariant. Pooling supplies local approximate invariance.

pooling facts

Fixed rule, zero parameters. Typically 2×2 stride 2. Max ≳ average empirically. Cost: precise positions unrecoverable after a few rounds — genuine trade-off.

mcnemar's test — comparing models $$ \chi^2=\frac{(|b-c|-1)^2}{b+c}\;\gt\;3.84 $$

Paired test on disagreements; same error totals can be significant (9²/12=6.75) or not (9²/40=2.03).

max vs average · worked 4×4 → 2×2

imagenet vs mnist — 5 axes harder

1000 classes100× pixels 3D scenescluttermulti-object

Equivariant activities, invariant knowledge

The course makes the distinction with Hinton's digit example: shift a handwritten 2 and the pattern of active neurons in a conv layer shifts by the same amount — "replicated features do not make the neural activities invariant to translation; the activities are equivariant." What weight sharing buys instead is invariant knowledge: if a feature was useful at some positions during training, a detector for it exists at every position at test time. So the network's competence is translation-invariant even though its activations are not. Exam trap: "conv layers are translation invariant" is false; invariance (approximate, local) only enters through pooling and, eventually, the fully connected head.

What pooling actually buys — and costs

The mechanism: combine four neighboring replicated detectors (2×2 patch) into one output — averaging in the original formulation, with max working "slightly better" empirically. Two payoffs the slide compresses away: (1) each level gains a small amount of translational invariance — a feature moving one pixel within the pool window often leaves the pooled value unchanged; (2) an economic effect — quartering the inputs to the next stage frees budget for many more feature maps, i.e. resolution is traded for feature diversity. The cost is Hinton's standing objection: after several pooling levels, information about precise positions is gone, making it impossible to use exact spatial relationships between high-level parts for recognition — a genuine architectural weakness, not an implementation detail.

Designed invariance vs brute force on MNIST

LeNet bakes invariance knowledge into the design: local connectivity, weight sharing, pooling → about 80 test errors (the famous run: 82). Adding input transformations and other tricks (Ranzato 2008) reaches ~40. Ciresan et al. (2010) took the opposite route — inject invariance through data: generate huge amounts of carefully transformed training examples per image and train a large, "dumb" deep net on a GPU without much overfitting → ~35 errors, and ~25 with model averaging; the right answer is almost always in the top-2 guesses. Estimated human error is 20–30, "but nobody has had the patience to measure it." Lesson: invariance can be engineered into architecture or injected via augmentation, and by 2010 brute force had caught up.

McNemar's test: the full tables

Is 30 errors out of 10,000 significantly better than 40? "It all depends on the particular errors" — totals alone can't decide. Cross-tabulate the two models per test case; only the disagreements matter. The course gives two tables, both with model 1 at 30 errors and model 2 at 40:

Both wrong 29, only model 2 wrong 1, only model 1 wrong 11, both right 9959 → $\chi^2=(10-1)^2/12=6.75 \gt 3.84$: significant at the 5% level (df = 1). Model 2's errors almost contain model 1's — consistent one-sided evidence.
Both wrong 15, only model 2 wrong 15, only model 1 wrong 25, both right 9945 → $\chi^2=81/40\approx 2.03 \lt 3.84$: not significant. Errors scatter across 40 disagreements, so the 10-error gap is plausibly chance.

The $-1$ is a continuity correction; the 9,900-plus cases both models handle identically contribute nothing. This paired design is why McNemar is "much more powerful than a test that just uses the number of errors."

from: CNN intro slides · ch 9

CH 09 · CNNs · landmark architectures

LeNet-5 → AlexNet — 14 years, same recipe

lenet-5 · 1998 · 82 errors on mnist · read ~10% of US checks

32×32×1input

→

6@28×28C1 · 5×5 · 156p

→

6@14×14S2 avg pool

→

16@10×10C3 partial conn.

→

16@5×5S4

→

120C5 conv=FC

→

84F6

→

10output

alexnet · 2012 · ilsvrc top-5 16.4% vs 26.1% runner-up

227×227×3"224" is errata

→

96@55×5511×11 s4

→

256@27×27conv+pool

→

13×13conv3–5

→

256@6×6pool

→

4096 ×2FC + dropout

→

1000softmax

alexnet's 5 ingredients

ReLU · local response norm · dropout p=.5 in FC · augmentation (crops+flips, 10-patch test) · 2× GTX 580 GPUs

dimension check (exam!)

$\lfloor(224{-}11)/4\rfloor{+}1=54\ne55$; solving gives $W=227$.

canonical pattern

[conv+ReLU ×n → pool]* → flatten → FC → softmax. Spatial ↓ while channels ↑ — resolution traded for representation.

LeNet-5 beyond the boxes

The 32×32 input is the 28×28 MNIST digit padded so border strokes can sit at kernel centers. C1 (6 maps, 5×5) has 156 parameters; S2 subsamples 2×2. C3 is where the course's handwritten multi-channel formula lives: for each $c_{out}=1\dots 16$, $$Z^{c_{out}}_{ij}=f\Big(\sum_{k,l,c_{in}} W^{c_{in},c_{out}}_{k,l}\,X^{c_{in}}_{i+k-3,\,j+l-3}\Big),\quad k,l=1\dots 5,\; c_{in}=1\dots 6,$$ the $+k-3$ offset being the centered 5×5 window. Full connectivity across channels would cost $16\cdot(5\cdot 5\cdot 6)+16=2416$ parameters; the real C3 connects each map to only a subset of the six S2 maps (1516 parameters) — breaking symmetry and forcing the maps to learn different feature combinations. C5's 120 units apply 5×5 kernels to a 16@5×5 input, so each output is 1×1: convolution degenerates to full connection ($120\cdot(5\cdot 5\cdot 16)+120=48{,}120$ parameters) — but on a larger input C5 would again produce spatial maps, which is why it is still labeled conv. The output stage in LeCun's original figure is marked Gaussian: RBF-style units compare the 84-dim F6 vector against a fixed template per class.

LeNet as a deployed system

The course stresses that LeNet was "a complete system, not just a recognizer": a wide net that copes with several characters at once even if they overlap, trained end-to-end with backprop — and it read roughly 10% of North American checks. Its 82 MNIST errors are mostly cases people find easy, i.e. its failure modes differ from human ones even at near-human accuracy.

AlexNet: what the flow diagram hides

Krizhevsky's net (NIPS 2012): 7 hidden layers not counting the max-pooling layers — early layers convolutional, the last two globally connected. The architecture figure is drawn split in two horizontal streams because the net ran on two GPUs: conv1 has 48+48 maps, later stages 128+128 at 27×27 and 192/192/128 at 13×13, and the dense layers are 2048+2048 per GPU (hence "4096"), with cross-GPU links only at selected layers. Activations: ReLU in every hidden layer — trains much faster and is more expressive than logistic units — plus competitive normalization (local response normalization), which suppresses a unit when nearby units fire more strongly, helping with intensity variation.

Generalization tricks and the 2012 result

Two tricks the course singles out: (1) augmentation — train on random 224×224 patches of the 256×256 images plus left-right reflections; at test time average the opinions of ten patches (four corners + center, and their reflections); (2) dropout in the globally connected layers, which hold most of the parameters — half the hidden units randomly removed per training example, stopping units from co-adapting. ILSVRC-2012: 1.2M training images, 1000 classes, classification scored top-5, localization requiring at least 50% box overlap. Final standings — Toronto 16.4% (34.1% with localization), Tokyo 26.1%/53.6%, Oxford 26.9%/50.0%, INRIA+XRCE 27.0%, Amsterdam 29.5%. The point: hand-tuned multi-stage vision pipelines from the strongest groups lost by ~10 points to end-to-end learned features.

Why the recipe deepens

Depth works by repeated composition: first hidden layer edges, second corners and contours, third object parts, then identity (Goodfellow Fig 1.2). Empirically the course shows test accuracy rising monotonically with depth on a digit-transcription task, roughly 92.3% at 3 layers to 96% at 11 (Fig 6.6) — the quantitative case behind "same recipe, 14 years deeper."

from: CNN intro slides · ch 9

CH 10 · CNN backprop · the headline results

Shared weight ⇒ sum over every reuse

forward · 4-D kernel, stride s $$ Z_{i,j,k}=\sum_{l,m,n} V_{l,(j-1)s+m,\,(k-1)s+n}\;K_{i,l,m,n} $$

gradient w.r.t. kernel — updates the layer $$ \frac{\partial J}{\partial K_{i,j,k,l}}=g(\mathbf G,\mathbf V,s)=\sum_{m,n} G_{i,m,n}\,V_{j,(m-1)s+k,\,(n-1)s+l} $$

FC: each weight used once → single product $g_ih_j$. Conv: kernel reused at every position → sum over all positions.

gradient w.r.t. input — passes upstream $$ \frac{\partial J}{\partial V}=h(\mathbf K,\mathbf G,s) \;=\; \text{transposed conv with }180^\circ\text{-flipped kernel} $$

Same op used forward as upsampling (segmentation, generative nets). "Deconvolution" is not an inverse — it's $W^\top$, not $W^{-1}$.

what comes out of backward

conv layer	2 gradients: ∂J/∂K + ∂J/∂V
pooling layer	1 gradient: ∂J/∂V only (no params)

the universal primitive $$ \texttt{op.bprop}=\sum_i(\nabla_X\,\texttt{op.f})_i\,G_i \quad\text{(vector–Jacobian product)} $$

Generic backprop: recurse over consumers with a memo table; cost ∝ #edges ≈ forward pass. Symbol-to-symbol graphs enable higher-order grads.

worked check (know it)

V 3×3, K=[[1,0],[−1,2]], G=[[1,−1],[2,0]] → ∂J/∂K=[[−1,4],[3,0]], ∂J/∂V=[[1,−1,0],[1,3,−2],[−2,4,0]].

Index bookkeeping: how the kernel gradient falls out

Everything starts from $G_{i,j,k}=\frac{\partial}{\partial Z_{i,j,k}}J(\mathbf V,\mathbf K)$ — the upstream gradient at output channel $i$, row $j$, column $k$. In the 4-D kernel $K_{i,l,m,n}$: $i$ = output channel, $l$ = input channel, $m,n$ = row/column offsets inside the window. Because of weight sharing, a single kernel entry touches every output position $Z_{i,m,n}$, each time multiplied by the input value sitting under it at that placement, $V_{j,(m-1)s+k,(n-1)s+l}$. The chain rule therefore produces one product per placement and the total gradient is their sum over all output positions $(m,n)$ — which is itself a correlation between the input and the upstream gradient map. With stride $s$, the input samples are taken $s$ apart: the effective correlation is dilated by the stride.

Gradient w.r.t. the input, fully unpacked

The slide compresses $h$ to "transposed conv with flipped kernel"; the course PDF gives the exact constrained form:

$$h(\mathbf K,\mathbf G,s)_{i,j,k}=\sum_{\substack{l,m\ \mathrm{s.t.}\\ (l-1)s+m=j}}\;\sum_{\substack{n,p\ \mathrm{s.t.}\\ (n-1)s+p=k}}\;\sum_{q}K_{q,i,m,p}\,G_{q,l,n}$$

The constraints select exactly the output positions $(l,n)$ whose stride-$s$ window covered input cell $(j,k)$ — nothing else can carry gradient back to it.
The extra sum over $q$ exists because $V$ feeds every output channel; input-channel index $i$ sits in the kernel's second slot.
For $s=1$ the constraint reads $m=j-l+1$: the kernel index runs backwards as the output index runs forwards — that reversal is the $180^\circ$ flip. Border cells satisfy the constraint for fewer windows, which is precisely a zero-padded "full" convolution.
For $s\gt 1$, first dilate $\mathbf G$ by inserting $s-1$ zeros between entries, then full-convolve with the flipped kernel — the transposed / fractionally-strided convolution.

Check against the slide's worked example: with flipped $K=[[2,-1],[0,1]]$, the centre entry is $2\cdot1+(-1)(-1)+0\cdot2+1\cdot0=3$ — matching $\partial J/\partial V_{2,2}$.

Bias gradients follow the sharing pattern

From the PDF's bias-term slide: convolution has one bias per output channel, so $\partial J/\partial b_i=\sum_{j,k}G_{i,j,k}$ (sum over all locations, same "sum over reuse" logic as $g$); locally connected layers have one bias per output unit (gradient is $G$ itself); tiled layers share biases with the tiling pattern; for fixed-size inputs one may learn a separate bias per location.

The generic engine behind it all

Algorithms 6.1–6.6: run the forward pass over the computational graph, set $\texttt{grad\_table}[z]\leftarrow1$, then build_grad recurses over each variable's consumers (via get_operation / get_consumers / get_inputs), calls each op's bprop (a vector–Jacobian product), sums the pieces, and memoizes. Cost is proportional to the number of edges — same order as the forward pass. The memo table is exactly what kills the repeated-subexpression blow-up of naive chain rule, e.g. $\frac{dz}{dw}=f'(f(f(w)))\,f'(f(w))\,f'(w)$: store $f(w)$ once instead of recomputing. Symbol-to-symbol differentiation (Fig 6.10) appends the derivative as graph nodes, so higher-order gradients come free. A conv layer plugs into this engine by registering just the pair $(g,h)$; the MLP case (Alg 6.4) is the degenerate version: $\nabla_W J=g\,h^\top$ and $g\leftarrow W^\top g$ — one reuse position instead of many.

from: CNN backprop slides · ch 10

CH 10 · CNN backprop · routing the gradient

Max routes, average splits

pooling gradients $$ \text{max: } \frac{\partial J}{\partial V_c}=\sum_{W\ni c} G_W\,[\,c=\arg\max_{c'\in W}V_{c'}] \qquad \text{avg: } \sum_{W\ni c}\frac{G_W}{|W|} $$

Common mistake: dividing the max gradient by window size. All of $G$ goes to the argmax; losers get exactly 0 — and zero training signal.

worked ex 10.3 — each G lands on its window's max

sharing spectrum · who owns weights

fully connected	no locality, no sharing
locally connected	local, private per position
tiled (period t)	t kernels cycle; bias $C_{out}{\times}t{\times}t$
standard conv	t=1 · full sharing · bias $C_{out}$

strided conv forward $$ Z_{i,j,k}=\sum_{l,m,n}V_{l,(j-1)s+m,(k-1)s+n}K_{i,l,m,n} $$

Index shift $j{+}m{-}1\to(j{-}1)s{+}m$; s=1 recovers standard. Can replace pooling.

differentiability

Max pool: a.e. differentiable (ties = measure zero). Average: smooth everywhere.

Why pooling returns exactly one gradient

A pooling layer owns no parameters, so op.bprop has nothing to report for weights — only $\partial J/\partial V$ flows through. For max pooling, $P_W=\max_{c\in W}V_c$ gives $\partial P_W/\partial V_c=1$ if $c$ is the argmax and $0$ otherwise, so backward is a pure router: it needs the argmax indices cached from the forward pass. This is a subgradient — at an exact tie the max is not differentiable, but ties form a measure-zero set, so training ignores them (pick one winner or split by convention).

Overlapping windows accumulate

The $\sum_{W\ni c}$ in the pooling gradient is not decoration. In the slide's ex 10.3 the $2{\times}2$ windows have stride 2, so they tile disjointly and each input cell receives at most one $G$. But the course's pooling-with-downsampling picture uses width-3 windows with stride 2: adjacent windows share a cell, and a cell that wins two windows receives the sum of both upstream gradients. Average pooling backward is equivalently a convolution of $G$ with a uniform kernel of value $1/|W|$: every cell trains a little, whereas max trains only winners — a sparser but more targeted signal.

Tiled convolution, backward included

Tiled convolution (لابلای یکدیگر — "interleaved") cycles through a bank of $t$ kernels as the output position advances: $Z_{i,j,k}=\sum_{l,m,n}V_{l,j+m-1,k+n-1}\,K_{i,l,m,n,\,j\%t+1,\,k\%t+1}$. Setting $t=1$ recovers standard convolution; letting $t$ reach the output size recovers a locally connected layer — the sharing spectrum on the slide is literally a dial on $t$. Backward, the $g$-sum splits by residue class: kernel-bank member $(a,b)$ collects gradient only from output positions with $j\bmod t=a$, $k\bmod t=b$ — each kernel is updated by the subset of positions where it was actually used. Neighbouring units get different filters (position-dependent specialization) at a cost of only $t^2$ kernel copies.

Strided conv is conv fused with downsampling

The PDF's "downsampled convolution" page: a stride-$s$ convolution equals a stride-1 convolution followed by keeping every $s$-th output — but the fused form never computes the values that would be thrown away. That efficiency, plus the fact that the downsampling is learned, is why strided conv can replace pooling (the all-convolutional design). Backward through stride mirrors this: the discarded positions simply contribute nothing, which is the zero-insertion step of the transposed convolution.

Where these layers apply at all

Convolution assumes grid-structured data, and the course's data-format table is exam bait: 1-D single-channel = audio waveform, 1-D multichannel = skeleton joint angles over time; 2-D single = Fourier-transformed audio (equivariance across time and frequency — a melody an octave higher yields the same features at a different height), 2-D multichannel = RGB image; 3-D = CT volumes, or color video with time as one axis. Channels are the non-grid axis — never convolved over.

from: CNN backprop slides · ch 10

CH 11 · CNN design · three structural priors

Sparse, shared, equivariant

320×280 img · 2×1 kernel	stored floats	ops
dense matrix	> 8×10⁹	> 16×10⁹
sparse matrix	178,640	267,960
convolution	2	267,960

Sparsity fixes the op count; only sharing fixes storage — the sparse→conv gap is entirely parameter sharing.

param spectrum · 5 in → 4 out, kernel 2

FC 20local 8 tiled t=2 4conv 2

receptive field grows with depth

Width-3 kernels: layer-2 unit already sees 5 inputs. Deep stacks of small kernels capture long-range structure.

padding decides depth $$ W-(F-1)L\ \ge\ F \quad\Rightarrow\quad 32-4L\ge5 \Rightarrow L\le6 $$

No padding: lose $F{-}1$ width per layer → forced shallow. "Same" padding → arbitrarily deep.

layer anatomy — fixed order

convaffine

→

detectorReLU

→

poolinvariance

The efficiency table, decoded

The 320×280 example is the edge detector of Fig 9.6: kernel $[1,-1]$ takes horizontal differences, output $319\times280$. Each output needs 3 ops (one mul per weight, one add), hence $319\cdot280\cdot3=267{,}960$. A dense matrix mapping every input pixel to every output pixel needs $319\cdot280\cdot320\cdot280\gt 8\times10^9$ stored weights and twice that many ops. Making the matrix sparse (keep only the two nonzeros per row) fixes the op count but still stores $2\cdot319\cdot280=178{,}640$ floats, because each row keeps its own private copy of the pair — the remaining factor of $\sim\!90{,}000$ down to 2 floats is pure parameter sharing (tying). So: sparsity = fewer ops, sharing = fewer parameters; distinct mechanisms, and the exam asks you to attribute the right saving to the right one. Sparsity also runs both directions — each output reads $k$ inputs (its receptive field), and each input influences at most $k$ outputs.

Equivariance is not invariance

Convolution is equivariant to translation: shift-then-convolve equals convolve-then-shift. It is not equivariant to rotation or scale — nothing in the operation handles those. Invariance is pooling's contribution: in the Fig 9.8 experiment, shifting the detector row one pixel changes every detector value but only about half the max-pool outputs, because each pool reports "is the feature present nearby" rather than "exactly where". Cross-channel pooling (Fig 9.9) generalizes this: max-pooling over the outputs of several learned filters — e.g. detectors for a "5" at different rotations — yields invariance to learned transformations, not just translation.

Receptive fields (گسترش میدان ادراکی) and the padding dial

Fig 9.4's point: direct connections are sparse, but the indirect receptive field grows with depth (width-3 kernels: 3, 5, 7, …), and faster still through stride or pooling — deep units see most of the image, so sparse networks still capture global structure. Padding then decides whether depth is even available. The instructor's annotation on Fig 9.13: without padding, each layer loses about half the kernel width from each side. The three regimes:

valid (no padding): width shrinks by $F-1$ per layer, giving the slide's depth bound; border pixels also enter fewer windows, so they are under-represented.
same (pad to keep width): arbitrary depth, but border outputs are computed partly from padded zeros.
full (pad $F-1$): every input pixel is visited $F$ times and the output grows; rarely useful. Optimal padding usually lies between valid and same.

Terminology trap: what counts as a "layer"

Fig 9.7 gives two vocabularies. Complex terminology: one "convolutional layer" = convolution stage (affine) + detector stage (nonlinearity) + pooling stage. Simple terminology: each stage is its own layer. Papers mix these freely — when a source says "conv layer", check whether pooling is inside it. The processing order itself is fixed either way, exactly as on the slide.

from: CNN design slides · ch 11

CH 11 · CNN design · from trunk to prediction

Three heads, four families, one skip

head (trunk → 16×16×64)	extra params	var. size?
flatten + dense 1000	≈ 16.4 M	no
pool to 3×3 + dense	≈ 0.58 M	yes
fully conv + global avg pool	≈ 0	yes

family	one-line identity
spatial transducer	fully convolutional, no dense anywhere
all-convolutional	no pooling — strided conv downsamples
Inception	parallel multi-scale kernels per stage
ResNet	additive identity skips → extreme depth

y = ReLU( x + F(x) ) — gradient rides the skip undiminished

biology, briefly

Trained first-layer kernels ≈ Gabor functions (V1 simple cells) — real but limited analogy: first layer only, backprop implausible.

The three heads of Fig 9.11, step by step

All three trunks are identical: input $256\times256\times3$ → conv+ReLU to $256\times256\times64$ → stride-4 pool → conv+ReLU → stride-4 pool → $16\times16\times64$. They diverge only at the head:

Reshape + dense: flatten to a 16,384-vector, matrix-multiply to 1000 units, softmax. The dense layer's fan-in hard-codes the trunk's output size — change the input resolution and the head breaks.
Pool to a fixed grid: pool with variable-size regions down to a $3\times3\times64$ grid (576 units), then dense to 1000. Because the number of regions is fixed but their size adapts, any input resolution works.
Fully convolutional: convolve to 1000 channels, giving a $16\times16\times1000$ map of per-location class scores, then global average pooling to $1\times1\times1000$ → softmax. Essentially zero head parameters, variable input size, and the score maps double as coarse localization.

Spatial transducers and recurrent pixel labeling

When the output must scale with the input (pixel labeling / segmentation), every layer is convolutional. Fig 9.17 adds recurrence: hidden tensor $\hat{\mathbf H}^{(t)}$ is computed from the image $\mathbf X$ through kernel $\mathbf U$ and from the previous label estimate $\hat{\mathbf Y}^{(t-1)}$ through $\mathbf W$; the new estimate is $\hat{\mathbf Y}^{(t)}=\mathbf V\,\hat{\mathbf H}^{(t)}$. The same $\mathbf U,\mathbf V,\mathbf W$ are reused every step — a convolutional RNN that iteratively refines labels so neighbouring pixels' predictions can reconcile.

Family fine print worth quoting

All-Convolutional Net: no pooling anywhere; strided convolution shrinks the representation — downsampling with learned weights.
Inception: the design goal in the PDF's wording is high accuracy at low computational cost, not accuracy alone.
ResNet: blocks of layers at the same spatial size, each layer's output added to one repeatedly-updated buffer. With $y=x+F(x)$, $\partial y/\partial x=I+\partial F/\partial x$: the identity term guarantees an undiminished backward path, so "very many updates = very deep net without vanishing gradient".

Channel connectivity is its own design choice

Spatial sparsity (Fig 9.14: local vs conv vs fully connected) is only half the story. Across channels, standard convolution is dense: every output channel reads every input channel. Fig 9.15 shows partial channel connectivity — each output channel wired to a subset of input channels — cutting parameters and compute along the channel axis independently of kernel size (the ancestor of grouped convolutions).

Gabor evidence, both directions

Fig 9.18 sweeps the Gabor family over orientation, frequency and phase (white = positive weight, black = negative). Fig 9.19's punchline: first-layer kernels learned by two unrelated algorithms are both Gabor-like — many objectives converge on the same first-layer vocabulary, matching V1 simple cells. The analogy still stops there: it says nothing about deeper layers, and backprop itself is not biologically plausible.

from: CNN design slides · ch 11

Part IV · Chapters 12–14

Sequence Models

12 — RNNs: recurrence, unrolling, task typologies
13 — BPTT, vanishing gradients, LSTM & GRU
14 — Attention & Transformers: QKV, multi-head, the block

CH 12 · RNNs · three matrices, any length

Share weights across time

folded self-loop = unrolled chain · same three matrices at every step

vanilla rnn $$ h_t=\tanh\!\big(W_{hh}h_{t-1}+W_{xh}x_t\big),\qquad \hat y_t=W_{hy}h_t $$

$h_t$ is an activation, not a parameter — init to zeros, reset between sequences (stale-context bug).

4 design criteria (naive ideas fail)

variable length	✓ recurrence
long-term deps	✗ fixed window
order matters	✗ bag-of-words
share params	✗ big slotted window

param math

|V|=10k, window 20, H=100: slotted = 20 M weights; shared = 1 M, independent of length. CNN shares across space, RNN across time.

Why each naive idea dies

The lecture builds the RNN by elimination. Idea 1 — fixed window: concatenate one-hot vectors of the last two words and feed an MLP. It works until the answer lives outside the window: "France is where I grew up, but I now live in Boston. I speak fluent ___" — the evidence sits 9+ words back, unreachable by construction. Idea 2 — bag of counts: use the whole sequence as one count vector. Now nothing is out of range, but order is destroyed: "the food was good, not bad at all" and "the food was bad, not good at all" produce the identical vector. Idea 3 — a really big fixed window: order survives, but each slot owns separate weights, so "this morning" learned at positions 1–2 teaches the network nothing about "this morning" appearing mid-sentence — no transfer. The slide's four design criteria are exactly the negative image of these three failures, and the recurrence satisfies all four at once.

The recurrence, read slowly

$h_t=f_W(h_{t-1},x_t)$ — same function, same parameters, every step. That single sentence carries the whole chapter: $h_t$ is a lossy, fixed-size summary of everything in $x_1..x_t$; whatever the future needs to know about the past must survive inside those $H$ numbers. The lecture's five-line intuition is worth memorizing: hidden_state = [0,0,0,0], then for word in sentence: prediction, hidden_state = my_rnn(word, hidden_state) — an RNN is a for-loop with a carried variable, and the unrolled diagram is just that loop drawn out. Terminology guard: the 6.S191 slide calls $h_t$ the "cell state"; from ch 13 on, cell state will mean the LSTM's separate $c_t$ — keep the two uses apart.

Three matrices, three jobs

$W_{xh}$ reads the input ($D{\to}H$), $W_{hh}$ transports memory ($H{\to}H$), $W_{hy}$ reports ($H{\to}K$) — and the unrolled chain re-stamps the same three at every step. The tanh keeps the state bounded in $(-1,1)$ under endless re-application (ReLU here would let $h$ grow without limit). And remember what is and isn't learned: $h_t$ is an activation, not a parameter — initialized to zeros and reset between sequences; carrying it across unrelated sequences is the stale-context bug flagged on the slide.

What sharing buys, statistically

The param-math card's numbers (slotted 20 M vs shared 1 M) understate the win: with shared weights every timestep of every sequence is a training example for the same three matrices, so "morning" at $t{=}2$ and "morning" at $t{=}200$ both sculpt the same $W_{xh}$ rows. It is the identical argument that justified convolution in ch 9 — tie weights across space because a cat is a cat anywhere; tie across time because a word means the same thing whenever it arrives.

from: lecture — sequence modeling: intro (MIT 6.S191) · ch 12

CH 12 · RNNs · karpathy taxonomy

Four wirings, one cell

two axes only: input single/sequence × output single/sequence — cell equations never change

loss across time $$ L=\sum_{t=1}^{T}L_t \qquad \frac{\partial L}{\partial W_{hh}}=\sum_{t=1}^{T}\frac{\partial L_t}{\partial W_{hh}} $$

Each shared weight collects one gradient contribution per step used — that sum is BPTT.

synced ≠ general n→n

Aligned equal-length (POS tags, music) works directly; translation needs encoder–decoder — whole input → one bottleneck vector → decode.

Mapping the taxonomy to real tasks

1→1 is the degenerate case — a plain feed-forward net, no sequence at all; it sits in the figure as the baseline the others generalize. n→1: sentiment classification — words in, one $P(\text{positive})$ out; only the final $h_T$ feeds the classifier and the loss is a single softmax-CE (softmax_cross_entropy_with_logits in the lecture), yet gradients still reach every timestep through the recurrent chain. 1→n: captioning — an image vector seeds the state, then the net emits a sequence. n→n synced: POS tagging, and the course lab's music generation — input a sheet-music character, predict the next one at every step. n→n unaligned (translation) is the one wiring the plain diagram cannot do — see the last panel.

Next-token training is teacher forcing

For generation tasks the target at step $t$ is simply $x_{t+1}$ — the data labels itself. During training you always feed the ground-truth prefix, not the model's own guesses (teacher forcing): every step has a clean target, and one forward pass scores all positions. At generation time the loop changes: sample $\hat y_t$, feed it back as $x_{t+1}$, repeat — so early mistakes compound (exposure bias), which is why generated text drifts in a way training loss never shows. File this away: ch 14's decoder keeps exactly this trick and adds a mask so all steps can even be trained in parallel.

The loss sum is where BPTT is born

Each step contributes $L_t$ (softmax-CE against that step's target) and the deck's figure funnels every $L_t$ into one node $L=\sum_t L_t$. Because the weights are shared, $\partial L/\partial W_{hh}=\sum_t \partial L_t/\partial W_{hh}$ — the gradient of one matrix literally accumulates one contribution per use. The backward pass that computes those contributions must retrace every arrow of the unrolled graph, right to left. That retracing has a name — backpropagation through time — and its pathology (a product of Jacobians along the chain) is the entire subject of the next chapter.

Encoder–decoder and its bottleneck

Translation input and output differ in length and word order, so the lecture splits the model: an encoder consumes "the dog eats" and its final state — one fixed-size vector — becomes the initial state of a decoder, which starts from <start> and emits "le chien mange", feeding each output back as the next input. The slide explicitly labels the handed-over vector the encoding bottleneck: an entire sentence, however long, must survive in $H$ numbers. Long inputs degrade first — and this precise complaint is the opening argument of attention in ch 14.

from: lecture — sequence modeling: intro (MIT 6.S191) · ch 12

CH 13 · BPTT · werbos 1990

A product of Jacobians — geometric fate

backward recursion — direct + indirect path $$ g_t=\underbrace{W_{hy}^{\top}\delta_t^{y}}_{\text{via }\hat y_t}+\underbrace{W_{hh}^{\top}\big(g_{t+1}\odot\tanh'(a_{t+1})\big)}_{\text{from the future}} $$

Recurrent term absent only at $t{=}T$. Omitting it = most common error. Then $\partial L/\partial W_{hh}=\sum_t\delta_t^a h_{t-1}^\top$ with $\delta_t^a=g_t\odot(1-h_t^2)$.

why gradients die (or explode) $$ \frac{\partial h_k}{\partial h_t}=\prod_{i=t+1}^{k}\mathrm{diag}\big(\tanh'(a_i)\big)W_{hh},\qquad \Big\|\tfrac{\partial h_k}{\partial h_t}\Big\|\le\rho^{\,k-t} $$

$\rho\gt1$ → explode; $\rho\lt1$ (typical — $|\tanh'|\le1$) → vanish. Systematic bias toward short-term deps: "clouds → sky" survives, "France → French" dies.

exploding fix — clipping $$ \tilde g=g\cdot\min\!\Big(1,\tfrac{\text{thr}}{\|g\|_2}\Big) \quad\text{(norm, keeps direction)} $$

By value: $\mathrm{sign}(g_i)\min(|g_i|,\text{thr})$ — can bend direction. Clipping cannot fix vanishing — it only shrinks. Classic trap.

vanishing remedies, ranked

ReLU	f′=1 for a>0, but W_hh untouched
identity init	W_hh=I — drifts during training
gated cells	the only architectural fix →

feel the decay

×0.6/step → 0.13 after 4σ′ ≤ 0.25tanh′ ≤ 1

BPTT, mechanically

Run the whole sequence forward, accumulate $L=\sum_t L_t$, then run one backward pass right-to-left along the unrolled chain. Every red arrow in the lecture figure crosses the same shared matrix, so the update written on the board is $W_{hh}\leftarrow W_{hh}-\eta\sum_{i=1}^{t}\nabla_{W_{hh}^{(i)}}\text{loss}$ — one gradient contribution per timestep the weight was used, summed. Nothing here is new machinery: BPTT is ordinary backprop applied to the unrolled computational graph, where time plays the role of depth. A 100-step sequence backprops like a 100-layer network — which is exactly why it inherits the deep-net gradient pathologies.

The Jacobian product, expanded

$\partial h_k/\partial h_t=\prod_{i=t+1}^{k}\mathrm{diag}(\tanh'(a_i))\,W_{hh}$ — that's $k-t$ factors. Since $\tanh'\le 1$ (and $\sigma'\le 0.25$), unless $W_{hh}$'s spectral radius compensates, the norm bound $\rho^{k-t}$ decays geometrically: ×0.6 per step is 0.13 after 4 steps and $\sim 10^{-9}$ after 40. Distant errors arrive microscopically small, so training systematically biases parameters toward short-term dependencies — the lecture's paired examples: "the clouds are in the ___" (gap 3, learnable) vs "I grew up in France … I speak fluent ___" (gap 20+, effectively invisible). Exploding is the mirror case (many factors > 1) and fails loudly — loss spikes, NaNs — where vanishing fails silently.

Clipping — the board's worked example

With $g=\langle 1,4,-3,5\rangle$ and threshold 2, value clipping $\mathrm{sign}(g_i)\min(|g_i|,\text{thr})$ gives $\langle 1,2,-2,2\rangle$ — the ratios between coordinates changed, so the update direction bent. Norm clipping $\tilde g=g\cdot\min(1,\text{thr}/\lVert g\rVert_2)$ rescales all coordinates by one factor and keeps the direction — the usual default. Either way, clipping is a ceiling, not a floor: it can only shrink gradients, never revive them, so it addresses exploding and does exactly nothing for vanishing. That asymmetry is the chapter's classic trap question.

The three-knob remedy list, ranked

Activation: ReLU has $f'=1$ for $a>0$, so the diagonal factors stop shrinking — but the $W_{hh}$ product remains, and an unbounded state can now blow up.
Initialization: $W_{hh}=I$, biases 0 — the network starts as "copy the state forward", perfect memory at step 0; nothing prevents training from drifting away from it.
Architecture: change what the Jacobian is. Gated cells make the memory-path factor $\approx\mathrm{diag}(f_t)$ with no weight matrix in it at all — the only fix aimed at the mechanism rather than the symptom, and the subject of the next slide.

from: lecture — BPTT, LSTM & GRU (MIT 6.S191) · ch 13

CH 13 · LSTM · hochreiter & schmidhuber 1997 · hover a gate

A protected cell-state highway

memory crosses time through × and + only — never a matrix multiply

1 · forget $$ f_t=\sigma(W_f h_{t-1}+U_f x_t+b_f) $$

2 · store → 3 · update $$ i_t=\sigma(\cdot),\;\; \tilde c_t=\tanh(\cdot) \qquad c_t=f_t\odot c_{t-1}+i_t\odot\tilde c_t $$

4 · output $$ o_t=\sigma(\cdot),\qquad h_t=o_t\odot\tanh(c_t) $$

why it survives time $$ \partial c_t/\partial c_{t-1}\approx\mathrm{diag}(f_t) $$

No $W_{hh}$ factor — with $f_t\!\approx\!1$ the product needn't vanish. Two states $(c_t,h_t)$; 4× the weights of vanilla.

Forget, store, update, output — the narrative

The lecture's four verbs map one-to-one onto the figure. Forget: $f_t=\sigma(W_f h_{t-1}+U_f x_t+b_f)$ multiplies $c_{t-1}$ coordinate-wise — a per-dimension keep fraction (naming trap: $f_t\!\approx\!1$ means don't forget). Store: $i_t$ chooses which coordinates get written while $\tilde c_t=\tanh(\cdot)$ proposes the content. Update: $c_t=f_t\odot c_{t-1}+i_t\odot\tilde c_t$ — an edit, not an overwrite; contrast the vanilla cell, which replaces its entire state through $\tanh(W\cdot)$ every step. Output: $h_t=o_t\odot\tanh(c_t)$ releases a filtered view — the cell can carry information in $c$ for hundreds of steps without ever showing it in $h$.

Why σ for gates, tanh for content

A gate is a valve: $\sigma\in(0,1)$ makes the elementwise product interpolate smoothly between "block" and "pass", and its saturation at 0/1 — a bug for hidden units back in ch 3 — is a feature here, giving near-binary keep/erase decisions. Content paths ($\tilde c_t$, and the tanh on the output branch) use tanh instead: zero-centered and bounded, so the additively-accumulating cell state stays in a sane numeric range.

The gradient highway, quantitatively

Along the top rail, $c_{t-1}\to c_t$ passes through one elementwise × and one + — never a matrix multiply — so $\partial c_t/\partial c_{t-1}\approx\mathrm{diag}(f_t)$. Backprop along the $c$-track multiplies by learned gate values instead of by $\tanh'(a)\,W_{hh}$; this is Hochreiter & Schmidhuber's "constant error carousel", and the lecture's "uninterrupted gradient flow!" slide draws exactly that unbroken red arrow across cells. If training sets $f_t\approx 1$ on some coordinates, those coordinates transmit gradient essentially unattenuated across long spans. Practical corollary: initialize the forget-gate bias positive ($b_f\approx 1$) so memory starts on rather than hoping training turns it on.

Bookkeeping for the exam

Two states, two jobs: $c_t$ = long-run conveyor belt, $h_t$ = per-step working output (and what $y_t$ is read from). Confusing them is the most common written-exam slip.
Cost: four weight bundles ($f,i,\tilde c,o$), each $H\times(H{+}D)+H$ → the slide's 4× vanilla parameters; API mirror: SimpleRNN(units) → LSTM(units).
Traps: the output-path tanh is applied to $c_t$ directly (it is not a learned layer), and all three gates read $(h_{t-1},x_t)$ — none of them reads $c$.

from: lecture — BPTT, LSTM & GRU (MIT 6.S191) · ch 13

CH 13 · GRU · cho et al. 2014

Two gates, one state — the final table

gru — interpolation replaces forget+input $$ r_t=\sigma(W_r h_{t-1}+U_r x_t+b_r),\qquad z_t=\sigma(W_z h_{t-1}+U_z x_t+b_z) $$ $$ \tilde h_t=\tanh\big(W_h(r_t\odot h_{t-1})+U_h x_t+b_h\big) $$ $$ h_t=(1-z_t)\odot h_{t-1}+z_t\odot\tilde h_t $$

$z_t$ couples keep/write ($1{-}z_t$ and $z_t$ sum to 1) ≈ combined $f_t,i_t$; $r_t$ gates the candidate's view of the past; no output gate, no $c_t$.

which to pick

Empirically comparable. GRU cheaper (3× weight sets); LSTM can keep old memory and write lots of new info at once — GRU's coupling can't. Neither universally wins.

	vanilla RNN	LSTM	GRU
state	$h_t$	$c_t$ + $h_t$	$h_t$
gates	—	f, i, o + c̃	r, z + h̃
update	full overwrite	additive	interpolation
memory Jacobian	diag(tanh′)·W_hh	≈ diag(f_t)	≈ diag(1−z_t)
vanishing	severe	mitigated	mitigated
weight sets	1×	4×	3×

worked step — same inputs, both cells

$h_{t-1}{=}.5, c_{t-1}{=}.2, x_t{=}1$: LSTM → $f{=}.818, i{=}.777, \tilde c{=}.762, c_t{=}.755, h_t{=}\mathbf{.522}$ · GRU → $r{=}.818, z{=}.777, \tilde h{=}.721, h_t{=}\mathbf{.672}$

GRU as a compressed LSTM

The GRU (Cho et al. 2014 — born in the same machine-translation paper as the encoder–decoder) makes three cuts. Merge the two states into one $h_t$. Merge forget+input into a single update gate $z_t$ via interpolation, $h_t=(1-z_t)\odot h_{t-1}+z_t\odot\tilde h_t$ — keep-weight and write-weight are forced to sum to 1 per coordinate. Add a reset gate $r_t$ that scales $h_{t-1}$ inside the candidate: $r_t\to 0$ gives the candidate a fresh start ignoring the past, $r_t\to 1$ recovers a vanilla-RNN candidate. Drop the output gate entirely — the full state is exposed at every step, which is why a GRU has nowhere to hide private memory.

Reading the memory-Jacobian row

The table's decisive line: per step, vanilla multiplies gradients by $\mathrm{diag}(\tanh')\,W_{hh}$; LSTM by $\approx\mathrm{diag}(f_t)$; GRU by $\approx\mathrm{diag}(1-z_t)$. In both gated cells the factor is a learned number in (0,1) per coordinate with no weight-matrix product, so "remember for a long time" is a stable, trainable setting instead of an accident of spectral radius. Convexity bonus: since $h_t$ is a convex combination of $h_{t-1}$ and a tanh-bounded candidate, the GRU state can never blow up — boundedness by construction.

Choosing between them

Head-to-head studies (Chung et al. 2014; Greff et al.'s "LSTM: a search space odyssey", 2015) find no consistent winner — echoed by the slide's "which to pick" card. GRU: 3 weight sets vs 4, one state to carry, trains faster, often edges ahead on smaller datasets. LSTM: the decoupled $f_t,i_t$ can keep all old memory and write a lot of new content in the same step — GRU's $z$-coupling forces a trade between the two — occasionally decisive on tasks with dense long-range structure. Sensible default from the lecture: reach for GRU first on speed grounds; switch to LSTM when capacity looks like the binding constraint.

Traps, and the bridge to ch 14

Convention trap: some textbooks flip $z$ and write $h_t=z_t\odot h_{t-1}+(1-z_t)\odot\tilde h_t$ (z as the keep gate). Check which side $z$ multiplies before comparing formulas across sources.
Worked-step card: use it to verify your σ/tanh order of operations; note GRU's output (.672) exceeds LSTM's (.522) mostly because the LSTM squashes its released state through $o_t\odot\tanh(c_t)$ while the GRU exposes $h_t$ raw.
What gating did not fix: both cells still run $O(n)$ sequential steps (no parallel training) and, in seq2seq, still squeeze the source through one bottleneck vector. Those two residual complaints are precisely the case for attention — ch 14 picks up here.

from: lecture — BPTT, LSTM & GRU (MIT 6.S191) · ch 13

CH 14 · Transformers · vaswani et al. 2017

Attention — a soft dictionary lookup

scaled dot-product attention $$ \mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\Big(\frac{QK^{T}}{\sqrt{d_k}}\Big)V $$

Each query scores every key by dot product; softmax turns scores into weights; output = weighted average of values. A dictionary lookup, but soft — every entry contributes.

why divide by √d_k $$ q\!\cdot\!k=\textstyle\sum_{i=1}^{d_k} q_i k_i \;\Rightarrow\; \mathrm{Var}(q\!\cdot\!k)=d_k $$

Dot products grow with dimension → softmax saturates at one-hot → gradients vanish. Scaling keeps variance ≈ 1. Same saturation story as sigmoids in ch 3.

tensor	role	asks
Q query	what I'm looking for	"who is relevant to me?"
K key	what I advertise	"match me against queries"
V value	what I hand over	"take this if I win"

the seq2seq bottleneck it fixes

Encoder–decoder RNNs squeeze the whole source sentence into one fixed vector — long sentences degrade. Attention lets the decoder look back at all encoder states, weighted per step (Bahdanau 2014).

cost

scores n×ntime O(n²d)path length O(1)

From hard lookup to expectation

A python dict returns the single value whose key matches exactly. Attention relaxes both steps: similarity replaces equality, and the softmax turns similarities into a probability distribution $\alpha$ over positions, so the output is an expected value $\sum_j \alpha_j v_j$ rather than one winner. Because every operation (dot product, softmax, weighted sum) is differentiable, the network can learn where to look by gradient descent — no supervision on the weights themselves. The attention matrix $\alpha$ is also the model's best free interpretability tool: rows are queries, columns keys, and bright cells show which tokens informed which.

Why the √d_k derivation matters

Assume the components of $q$ and $k$ are independent with mean 0 and variance 1. Then $q\cdot k=\sum_{i=1}^{d_k}q_ik_i$ has mean 0 and variance $d_k$ — standard deviation $\sqrt{d_k}$. At $d_k=64$ raw scores of magnitude ±8 are routine, and softmax at that scale is effectively an argmax: one weight ≈ 1, the rest ≈ 0, and $\partial\alpha/\partial\text{score}\approx 0$ everywhere. Dividing by $\sqrt{d_k}$ restores unit variance, keeping the softmax in its responsive regime. This is the same saturation-kills-gradient argument made for sigmoids in ch 3 and for the softmax-CE pairing in ch 5 — a favorite exam link.

Additive vs multiplicative attention

Additive (Bahdanau 2014): score $=v^T\tanh(W_1 q+W_2 k)$ — a one-hidden-layer MLP per pair. More parameters, no saturation issue, but slow.
Multiplicative / dot-product (Luong 2015, Transformer): score $=q\cdot k$ — the whole score matrix is one $QK^T$ matrix multiply, exactly what GPUs are built for. With the $\sqrt{d_k}$ fix it matches additive quality at a fraction of the cost.

Reading the complexity pills

The $n\times n$ score matrix is attention's price: doubling sequence length quadruples time and memory — the reason long-context models need tricks (sparse, linear, or windowed attention). What you buy: any token reaches any other in one step. In an RNN the same interaction crosses $n$ recurrence steps, each a multiplication by $W_{hh}$ — the vanishing-gradient chain of ch 13. Attention's $O(1)$ path length is the structural cure, more radical than LSTM's gating patch.

from: final lecture — attention & transformers · ch 14

CH 14 · Transformers · self-attention · heads · position

The sequence attends to itself

self-attention — Q, K, V from the same X $$ Q=XW^{Q},\quad K=XW^{K},\quad V=XW^{V} $$

Every token queries every token — including itself. The three learned projections let "what I seek", "what I show" and "what I give" differ; without them attention would be plain symmetric similarity.

multi-head — h parallel subspaces $$ \mathrm{head}_i=\mathrm{Attn}(XW_i^{Q},XW_i^{K},XW_i^{V}),\quad \mathrm{MH}=\mathrm{concat}(\mathrm{head}_1..\mathrm{head}_h)W^{O} $$

$d_k=d_{model}/h$ (512/8 = 64): h views for ≈ the price of one full-width head. Heads specialize — syntax, coreference, position — emergently, never by supervision.

positional encoding — order must be injected $$ PE_{(pos,2i)}=\sin\!\frac{pos}{10000^{2i/d}},\qquad PE_{(pos,2i+1)}=\cos\!\frac{pos}{10000^{2i/d}} $$

Self-attention is permutation-equivariant — shuffle the tokens, outputs shuffle identically. Adding a position fingerprint to each embedding breaks the tie.

why sinusoids

Geometric wavelengths $2\pi\!\to\!10000\!\cdot\!2\pi$: each position gets a unique multi-frequency code, and $PE_{pos+k}$ is a linear function of $PE_{pos}$ — relative offsets are easy to learn. Learned position embeddings work equally well (BERT uses them).

exam trap

"Multi-head = more parameters" — false. Heads split $d_{model}$; total projection cost stays ≈ constant. The win is diversity of attention patterns, not capacity.

Why three projections and not zero

Drop $W^Q,W^K,W^V$ and attention degenerates to $\mathrm{softmax}(XX^T)X$: the score matrix is symmetric ($x_i\cdot x_j = x_j\cdot x_i$), every token's strongest match is itself, and "relevance" is frozen to raw embedding similarity. The projections make relevance directional and learnable: "bank" as a query can seek financial context while "bank" as a key advertises something else entirely. This asymmetry — who attends to whom is not who is attended by whom — is the point of the QKV factorization.

Multi-head arithmetic, concretely

With $d_{model}=512$ and $h=8$: each $W_i^{Q,K,V}$ is $512\times 64$, so the eight heads together cost $3\times 512\times 512$ — identical to one full-width head. Each head runs scaled dot-product attention in its own 64-dim subspace, the eight $n\times 64$ outputs are concatenated back to $n\times 512$, and $W^O$ ($512\times 512$) mixes them. One head averaging everything would wash distinct relations out; eight quarter-size heads can dedicate one to adjacent-word syntax, another to long-range coreference, etc. Empirically some heads are prunable after training — specialization is uneven.

Permutation equivariance, precisely

For any permutation matrix $P$: $\mathrm{Attn}(PX)=P\,\mathrm{Attn}(X)$ — permute the input rows and the output rows permute identically. Proof sketch: $PQ(PK)^T=PQK^TP^T$, and softmax acts row-wise, so $P$ passes through. The model therefore sees a bag of tokens; "dog bites man" and "man bites dog" are indistinguishable until positions are injected. Note the term: equivariant (output shuffles along), not invariant (output unchanged) — the same distinction as convolution vs pooling in ch 9.

The sinusoid design, unpacked

Dimension pair $2i,2i{+}1$ oscillates with wavelength $10000^{2i/d}\cdot 2\pi$: low dims spin fast (fine positions), high dims spin slow (coarse positions) — a continuous binary counter.
For fixed offset $k$, $(\sin,\cos)$ at $pos{+}k$ is a rotation of $(\sin,\cos)$ at $pos$ — a linear map independent of $pos$ — so "attend 3 tokens back" is one learned linear transform.
Sinusoids extrapolate to lengths never seen in training; learned embeddings cannot, but fit the training range slightly better.

from: final lecture — attention & transformers · ch 14

CH 14 · Transformers · attention is all you need

The block — and why it retired the RNN

residual skips (dashed) = the gradient highway

	RNN / LSTM	Transformer
max path length	O(n)	O(1)
sequential ops	O(n) — no parallel train	O(1) — fully parallel
per-layer cost	O(n·d²)	O(n²·d) — length hurts
order handling	built-in recurrence	injected via PE
long memory	gated (ch 13)	direct access

decoder — causal mask $$ \mathrm{score}_{ij}=\begin{cases}q_i\!\cdot\!k_j/\sqrt{d_k} & j\le i\\[2pt] -\infty & j\gt i\end{cases} $$

−∞ before softmax → weight exactly 0: no peeking at the future. Plus cross-attention: Q from decoder, K,V from encoder.

lineage

BERT — encoder, masked LMGPT — decoder, causalViT — patches as tokens

Inside the block, piece by piece

Each sublayer computes $\mathrm{LayerNorm}(x+\mathrm{Sublayer}(x))$. The residual sum is ch 13's constant-error-carousel idea generalized: gradients flow through the identity path untouched, so 12–96 stacked blocks train where a plain 12-layer stack of ch 5 would suffer vanishing gradients. LayerNorm normalizes across the feature dimension of each token independently — contrast with BatchNorm (ch 8), which normalizes across the batch and therefore breaks on variable-length sequences and small batches. The feed-forward sublayer is a per-token two-layer MLP, $\max(0,xW_1+b_1)W_2+b_2$, expanding $d_{model}\to 4d_{model}\to d_{model}$; it holds roughly two-thirds of the block's parameters and is where token-wise computation happens between the mixing steps.

Training vs inference in the decoder

Training uses teacher forcing (ch 12): the whole target sequence is fed at once, the causal mask guarantees position $i$ only sees $1..i$, and all positions train in parallel — this, not accuracy, is the transformer's decisive win over RNNs. Inference is still autoregressive: generate a token, append, rerun — which is why generation is slow while training scales, and why KV-caching exists (store past keys/values instead of recomputing).

Reading the comparison table honestly

$O(1)$ path length: any pair of tokens interacts in one attention hop — no gradient product over $n$ Jacobians (the ch 13 bound $\rho^{k-t}$ simply doesn't apply).
$O(n^2 d)$ per layer: for $n \lt d$ (typical sentences, $d=512$) this is cheaper than the RNN's $O(nd^2)$; it only bites at long contexts.
Sequential ops $O(1)$: an LSTM must compute step $t$ before $t{+}1$ — 1000 tokens = 1000 serial steps per layer. Attention is three matrix multiplies.

Lineage in one paragraph

Encoder-only BERT (2018): bidirectional context, trained by masking 15% of tokens and predicting them — the fill-in-the-blank objective suits understanding tasks. Decoder-only GPT (2018–): causal mask, trained to predict the next token — the objective that scales into modern LLMs. ViT (2020) chops an image into 16×16 patches, treats them as tokens, and beats CNNs given enough data — parameter sharing and locality (ch 9's founding assumptions) turn out to be learnable rather than necessary priors.

Exam traps

The mask is applied to scores before softmax, not to the weights after — post-softmax zeroing breaks the "weights sum to 1" invariant.
LayerNorm ≠ BatchNorm: axis of normalization is the standard trick question after ch 8.
"Transformers have no recurrence at all" — true for training; autoregressive generation is still a sequential loop.

from: final lecture — attention & transformers · ch 14

Finale · if you remember nothing else

The whole course in ten formulas

1 · neuron$$ y=f(\mathbf w^{T}\mathbf x+b) $$

2 · perceptron rule$$ \mathbf w\!\leftarrow\!\mathbf w+(t-a)\mathbf p,\;\; b\!\leftarrow\!b+e $$

3 · backprop δ$$ \delta_i=h'(a_i)\textstyle\sum_k w_{ki}\delta_k,\;\; \delta^{out}=y-t $$

4 · momentum$$ v\!\leftarrow\!\alpha v-\epsilon g,\;\; \theta\!\leftarrow\!\theta+v $$

5 · adam$$ \Delta\theta=-\epsilon\,\hat s\,/(\sqrt{\hat r}+\delta) $$

6 · weight decay$$ w\!\leftarrow\!(1-\eta\lambda)w-\eta\nabla f $$

7 · conv size$$ O=\lfloor(W\!-\!F\!+\!2P)/S\rfloor+1 $$

8 · bptt fate$$ \|\partial h_k/\partial h_t\|\le\rho^{\,k-t} $$

9 · lstm heart$$ c_t=f_t\odot c_{t-1}+i_t\odot\tilde c_t $$

10 · attention — where the story goes next$$ \mathrm{Attention}(Q,K,V)=\mathrm{softmax}\!\big(QK^{T}\!/\sqrt{d_k}\big)V $$

Press R and run the deck again — if you can recall each card from its label alone, you're ready. Good luck. Σ

RECALL ←→ move · T contents · R recall · M more 1/1

Artificial NeuralNetworks

Foundations

One neuron — weighted sum, then squash

Slow parts, smart whole

Two springs, one winter

Logic from a threshold

XOR breaks the line — depth fixes it

Convolution — one kernel, every position

The general neuron and its dimensions

Activation functions — click to explore

Static map vs dynamical system

The boundary is a hyperplane — and w is normal to it

Add the sample, subtract the sample

Training

Make the basis functions trainable

Backprop in four boxed steps

Non-convex, symmetric, overfittable

Noisy gradients are a feature

Velocity remembers ≈ 1/(1−α) gradients

One η can't fit an anisotropic bowl

Adam = RMSProp + momentum + bias correction

Shrink by percent or shrink by amount

Cripple the net, normalize the batch

ConvolutionalNetworks

Output size & parameters — drag the sliders

Equivariant conv, invariant-ish pooling

LeNet-5 → AlexNet — 14 years, same recipe

Shared weight ⇒ sum over every reuse

Max routes, average splits

Sparse, shared, equivariant

Three heads, four families, one skip

Artificial Neural
Networks

Convolutional
Networks