An ML/LLM Crash Course for Security People

CIS 400/600 · optional full-math reference · interactive. The gentle version of this material is the Week 2 ML Background & LLM Teaser deck and the Week 8 Neural Networks deck. These notes are the deep dive — never required, always here — covering everything in the crash-course reference deck and the details & extensions deck, with full prose, runnable code, and interactive demos. The LLM chapters (tokenization onward) are the deep material for the Weeks 10–11 sessions; CIS600 students, treat the whole page as fair game. Every widget is self-contained — open the console and read the source; none of it is magic, which is rather the point.

1 · ML foundations

1.1 The setup, the loss, and empirical risk

A trained model is a function $f_\theta$ built in two phases — training (sees the dataset, adjusts $\theta$; the attacker's lever is the data: poisoning, backdoors) and inference ($\theta$ frozen, maps $x \mapsto f_\theta(x)$; the attacker's lever is the input: adversarial examples, prompt injection). Keep the two windows separate in your head; they need entirely different defenses.

Supervised learning: given $S = \{(x_i, y_i)\}_{i=1}^n$ drawn i.i.d. from an unknown distribution $\mathcal{D}$, choose $f_\theta$ so it is right on future draws, not merely on $S$. A loss scores one prediction; the two that matter here are squared error $\tfrac{1}{2}(\hat y - y)^2$ and cross-entropy

$$ L(p, y) = -\sum_{k=1}^{C} y_k \log p_k = -\log p_c \quad (c = \text{true class}), $$

which is the negative log-likelihood of the correct class and blows up as $p_c \to 0$. Training is empirical risk minimization:

$$ R(\theta) = \frac{1}{n}\sum_{i=1}^{n} L\big(f_\theta(x_i), y_i\big), \qquad \theta^\star = \arg\min_\theta R(\theta). $$

1.2 Gradient descent

The gradient $\nabla_\theta R \in \mathbb{R}^p$ points uphill; step downhill with learning rate $\eta$: $\theta \leftarrow \theta - \eta \nabla_\theta R$. On $R(\theta) = (\theta-3)^2$ the update is $\theta \leftarrow (1-2\eta)\theta + 6\eta$, a geometric sequence with ratio $1-2\eta$: it converges iff $0 < \eta < 1$ and diverges outside. Try both regimes:

With $n$ in the billions the full gradient is hopeless per step, so estimate it on a random minibatch $B$: $\theta \leftarrow \theta - \eta \frac{1}{|B|}\sum_{i\in B}\nabla_\theta L_i$. The minibatch gradient is an unbiased estimator of the true gradient. (The systems reason you must minibatch — the GPU memory budget, ~16 bytes/parameter for mixed-precision Adam — is in the details deck; see also the memory calculator in §7.3.)

1.3 Neurons, layers, nonlinearity

A neuron is $z = w^\top x + b$, $a = \sigma(z)$; a layer of $m$ neurons is one matrix multiply $a = \sigma(Wx+b)$; an MLP chains them, $h^{(l+1)} = \sigma(W^{(l)} h^{(l)} + b^{(l)})$, ending in raw logits that a softmax turns into probabilities. Nonlinearity is not optional: two stacked linear layers collapse to one, $W_2(W_1 x + b_1) + b_2 = \tilde W x + \tilde b$, so any depth of purely linear layers can only draw a hyperplane — and no hyperplane computes XOR. One ReLU and a 2-neuron hidden layer solves it. Geometrically, each layer repositions space (affine) and then folds it (ReLU); enough folds make tangled classes linearly separable.

1.4 Forward and backward, by hand and in code

Backprop is the chain rule organized so shared sub-expressions are computed once. For a layer $z = Wh+b$, $a = \sigma(z)$, with upstream gradient $\delta = \partial L/\partial z$:

$$ \delta = \frac{\partial L}{\partial a} \odot \sigma'(z); \qquad \frac{\partial L}{\partial W} = \delta h^\top, \quad \frac{\partial L}{\partial b} = \delta, \quad \frac{\partial L}{\partial h} = W^\top \delta. $$

And at the very top, softmax + cross-entropy give the famously clean $\partial L/\partial z = p - y$. A complete, runnable implementation for the lecture's $2 \to 2 \to 2$ network:

# forward + backward for the lecture's 2-2-2 net, NumPy only
import numpy as np

def softmax(z):
    z = z - z.max()
    e = np.exp(z)
    return e / e.sum()

x  = np.array([1.0, 2.0])
y  = np.array([0.0, 1.0])                      # true class = 2
W0 = np.array([[1.0, -1.0], [0.0, 1.0]]); b0 = np.zeros(2)
W1 = np.array([[1.0,  1.0], [0.0, 2.0]]); b1 = np.array([0.0, 1.0])

# ---- forward ----
z1 = W0 @ x + b0                               # [-1, 2]
h  = np.maximum(z1, 0.0)                       # ReLU -> [0, 2]
z2 = W1 @ h + b1                               # [2, 5]
p  = softmax(z2)                               # [0.047, 0.953]
L  = -np.log(p @ y)                            # 0.048

# ---- backward ----
delta2 = p - y                                 # [0.047, -0.047]  (the p - y identity)
dW1    = np.outer(delta2, h)                   # [[0, .094], [0, -.094]]
db1    = delta2
dh     = W1.T @ delta2                         # gradient passed down
delta1 = dh * (z1 > 0)                         # ReLU gate: [0, 1] mask
dW0    = np.outer(delta1, x)
db0    = delta1

# ---- one SGD step ----
eta = 0.1
W1 -= eta * dW1;  b1 -= eta * db1
W0 -= eta * dW0;  b0 -= eta * db0

2 · Tokenization

2.1 The interface problem and the tradeoff

A network computes on numbers. The tokenizer — a fixed, deterministic program chosen and frozen before training — maps text to integer IDs that index rows of the embedding matrix $E \in \mathbb{R}^{|V| \times d_{\text{model}}}$. The choice of alphabet trades off three quantities: sequence length hurts quadratically (attention is $O(n^2 d_{\text{model}})$), vocabulary size hurts linearly (embeddings and softmax are $O(|V| \cdot d_{\text{model}})$), and out-of-vocabulary behavior determines whether inputs can fail to encode. Bytes: tiny $|V|$, longest $n$. Words: shortest $n$, huge $|V|$, heavy OOV. Subword (BPE): the engineered middle — fixed $|V|$ of 10k–100k, medium $n$, zero OOV.

2.2 Byte-pair encoding, complete

from collections import Counter

def pair_counts(corpus):
    # corpus: {tuple_of_symbols: word_frequency}
    pairs = Counter()
    for symbols, freq in corpus.items():
        for a, b in zip(symbols, symbols[1:]):
            pairs[(a, b)] += freq
    return pairs

def apply_merge(corpus, pair):
    a, b = pair
    new = {}
    for symbols, freq in corpus.items():
        out, i = [], 0
        while i < len(symbols):
            if i + 1 < len(symbols) and symbols[i] == a and symbols[i+1] == b:
                out.append(a + b); i += 2          # fuse the pair
            else:
                out.append(symbols[i]); i += 1
        new[tuple(out)] = freq
    return new

def learn_bpe(corpus, K):
    merges = []
    for _ in range(K):
        pairs = pair_counts(corpus)
        if not pairs: break
        best_count = max(pairs.values())
        best = min(p for p in pairs if pairs[p] == best_count)  # deterministic tie-break
        merges.append(best)                        # ORDER matters
        corpus = apply_merge(corpus, best)
    return merges, corpus

corpus = {tuple("low_"): 5, tuple("lower_"): 2,
          tuple("newest_"): 6, tuple("widest_"): 3}
merges, final = learn_bpe(corpus, 5)
# merges: [('e','s'), ('es','t'), ('est','_'), ('l','o'), ('lo','w')]

GPT-2 runs BPE over raw bytes: base alphabet 256, so every input encodes — $|V| = 256 + 50{,}000 \text{ merges} + 1$ (<|endoftext|>) $= 50{,}257$. A regex pre-tokenizer keeps merges from crossing word boundaries, which is why "the" and " the" (leading space) are different token IDs.

2.3 The tokenizer is a parser — and parsers disagree

Glitch tokens (SolidGoldMagikarp and friends) — vocabulary rows the model never trained — are covered in the details deck.

3 · Embeddings and attention

3.1 From IDs to vectors, and the soft dictionary

Token ID $x_t$ selects row $x_t$ of the learned table $E$; stacking rows gives the model's input $X \in \mathbb{R}^{n \times d_{\text{model}}}$. The geometry is emergent — "cat" and "dog" are near each other because that lowered the loss. Attention is then a content-addressed retrieval: a dict where matching is by similarity, not equality, and every entry contributes. Queries, keys, values are learned projections $Q = XW_Q$, $K = XW_K$, $V = XW_V$, and the score $q \cdot k = \lVert q\rVert \lVert k\rVert \cos\theta$ is pure geometry — the learning lives in the projections that shape the space.

$$ \mathrm{Attention}(Q, K, V) = \mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V $$

Compare ($QK^\top$, all pairs), scale ($1/\sqrt{d_k}$), normalize (row-wise softmax → row-stochastic $A$), read out ($AV$: each output row is a convex combination of value rows). The $\sqrt{d_k}$ is a variance argument: for unit- variance entries, $\mathrm{Var}(q\cdot k) = d_k$, so dividing by $\sqrt{d_k}$ restores unit variance and keeps softmax out of its saturated, zero-gradient regime. And softmax weights are never exactly zero (except under a $-\infty$ mask): every position always reads every other position a little.

3.2 Causal masking, code, and the security reading

Autoregressive training scores all positions in parallel, so position $i$ must not see $j > i$: set those scores to $-\infty$ before softmax and the weights become exactly zero. The mask is a time constraint, not a trust constraint — untrusted tokens sit in the lower triangle with everything else. Reference implementation (run it on the widget's numbers):

import numpy as np

def softmax(z, axis=-1):
    z = z - z.max(axis=axis, keepdims=True)
    e = np.exp(z)                                  # exp(-inf) -> 0, no NaN
    return e / e.sum(axis=axis, keepdims=True)

def attention(Q, K, V, causal=False):
    d_k = Q.shape[-1]
    S = Q @ K.T / np.sqrt(d_k)
    if causal:
        n = S.shape[0]
        S = np.where(np.triu(np.ones((n, n), dtype=bool), k=1), -np.inf, S)
    A = softmax(S, axis=-1)
    return A @ V, A

Q = np.array([[1.,0.],[1.,1.],[0.,1.]])
K = np.array([[1.,0.],[0.,1.],[1.,1.]])
V = np.array([[2.,0.],[0.,2.],[1.,1.]])
out, A = attention(Q, K, V)     # out[1] == [1.00, 1.00]

4 · The transformer block

4.1 The four ingredients

• Multi-head attention. One head = one softmax = one relationship per position. Run $h$ heads at width $d_k = d_{\text{model}}/h$ (total compute roughly constant), concatenate, and mix with $W_O$: $\mathrm{MultiHead}(X) = \mathrm{Concat}(\mathrm{head}_1,\dots,\mathrm{head}_h)W_O$.
• Position (RoPE). Attention alone is permutation- equivariant — order-blind. RoPE rotates $q$ at position $m$ by $R(m\theta)$ and $k$ at position $n$ by $R(n\theta)$; since $R(m\theta)^\top R(n\theta) = R((n-m)\theta)$, the score depends only on the relative offset $n-m$. Relative position, built into the dot product, no learned table.
• Residual stream + LayerNorm. Every sublayer is $y = x + F(x)$, so the Jacobian has an identity term and gradients cross a hundred layers undiminished. LayerNorm standardizes each token over its own features, then rescales with learned $\gamma, \beta$. Read the model as one additive bus of width $d_{\text{model}}$ that every component reads and writes — with no protected subspace.
• Feed-forward network. $\mathrm{FFN}(x) = W_2\,\phi(W_1 x + b_1) + b_2$ with $d_{ff} = 4d_{\text{model}}$, applied per token. It is the block's only per-token nonlinearity and holds two-thirds of its parameters — interpretability work reads it as key-value memory where much factual knowledge lives.

4.2 The parameter budget

Per block with $d_{ff} = 4d$: attention $= W_Q,W_K,W_V,W_O = 4d^2$; FFN $= W_1 + W_2 = 8d^2$; total $\approx 12\,d_{\text{model}}^2$ per block, so $N \approx 12 L d_{\text{model}}^2 + |V| d_{\text{model}}$. Sanity-check GPT-2 small: $12 \cdot 12 \cdot 768^2 \approx 85$M plus $50{,}257 \cdot 768 \approx 38.6$M — about 124M. The FFN, not attention, holds the weights.

5 · Language modeling and training

5.1 The objective

The chain rule factors any sequence distribution exactly: $p_\theta(x) = \prod_t p_\theta(x_t \mid x_{<t})$. The model's one job is the conditional next-token distribution, and training is maximum likelihood — equivalently cross-entropy, equivalently per-token negative log-likelihood:

$$ \mathcal{L}(\theta) = -\frac{1}{n}\sum_{t=1}^{n} \log p_\theta\big(x_t \mid x_{<t}\big). $$

No separate label set: the corpus is its own supervision ("self-supervised"), which is why the data budget is the readable internet. Three training facts that matter later:

• Teacher forcing / exposure bias. Training always conditions on the ground-truth prefix; inference conditions on the model's own output. Guarantees thin out off-distribution — which is exactly where jailbreaks operate (unusual encodings, role-play frames, long distracting prefixes). Same lesson as fuzzing.
• One pass, $n-1$ targets. The causal mask lets a single forward pass produce every conditional at once; targets are the inputs shifted left by one. (Get the shift wrong and the model learns to copy its input — loss collapses to ~0, a classic "too good to be true.")
• Perplexity. $\mathrm{PPL} = e^{\mathcal{L}}$, the geometric-mean $1/p$ — an effective branching factor. Uniform over $|V|$ gives $\mathrm{PPL} = |V|$; perfect prediction gives 1; perplexity 20 reads "as unsure as a fair 20-sided die at each step."

5.2 The corpus is a supply chain

5.3 Scaling and the "emergence" debate

Chinchilla's compute-optimal recipe (grow parameters and data in lockstep, ~20 tokens/parameter) is in the details deck. The debate to internalize is emergence: Wei et al. report abilities that appear suddenly at scale; Schaeffer et al. reply that an all-or-nothing metric can manufacture a cliff out of smooth progress. If per-token accuracy $a$ improves smoothly, exact-match on a $k$-token answer is $\approx a^k$ — try it:

6 · Alignment: SFT, RLHF, DPO

6.1 The pipeline

Pretraining never says be helpful; a base model just continues text. Alignment is post-training, and every stage is the same weights, more fine-tuning:

                     demonstrations              preference pairs
                   (instruction,response)     (x, y_win, y_lose)
                          |                          |
pretrained ──SFT──▶  SFT model  ──reward model──▶  r_phi  ──PPO──▶ aligned policy
pi_base   (X-ent)    pi_ref     (Bradley-Terry)          (KL leash)   pi_theta
                          |                                              ^
                          └────────── DPO (skip r_phi + RL) ─────────────┘

• SFT: cross-entropy on (instruction, ideal response) pairs, prompt tokens masked from the loss. Behavioral cloning — it never sees a bad answer to push away from.
• Reward model + Bradley–Terry: a scalar head $r_\phi(x,y)$; preference probability is a logistic on the reward difference, $P(y_w \succ y_l) = \sigma\!\big(r_\phi(x,y_w) - r_\phi(x,y_l)\big)$ — identifiable only up to an additive constant.
• PPO with a KL leash: $\max_\theta \mathbb{E}[r_\phi(x,y)] - \beta\,\mathrm{KL}(\pi_\theta \| \pi_{\text{ref}})$. The leash exists because $r_\phi$ is a learned proxy: optimize it too hard and Goodhart bites — gold-standard quality rises, peaks, then falls as KL grows (Gao et al.; curve in the details deck).
• DPO: substitute the implicit reward $\beta\log\frac{\pi_\theta}{\pi_{\text{ref}}}$ into Bradley–Terry; the partition function cancels, leaving one supervised-style classification loss — no reward network, no RL, offline.

6.2 What alignment is, mechanically

7 · Decoding and inference

7.1 From logits to a token

Greedy ($\arg\max$) is deterministic but degenerates into repetition; likelihood-maximizing text is not human-like text (Holtzman et al.). So we sample, with knobs: temperature rescales logits ($p = \mathrm{softmax}(z/T)$; $T \to 0$ is greedy, $T \to \infty$ is uniform, ranking never changes) and top-p truncates to the smallest set of tokens whose cumulative mass reaches $p$ — the nucleus adapts to the distribution's shape, which a fixed top-$k$ cannot.

7.2 The KV cache

Generating token $t{+}1$ needs every earlier position's $K,V$; caching them turns an $O(n^2)$ decode into $O(n)$ per step, at a memory price of $2 \cdot L \cdot n \cdot d_{\text{model}} \cdot b$ bytes per sequence. For Llama-2-7B in fp16 that is 0.5 MiB per token — 2 GiB per full 4096-token sequence, so 8 concurrent sequences carry more cache than the 13 GiB of weights. KV memory, not compute, caps serving throughput; GQA/MQA (shared KV heads) is the standard mitigation.

8 · Agents and the security surface

8.1 The loop

def agent(goal, tools, model, max_steps):
    ctx = [SYSTEM, goal]                    # the context = ONE token stream
    for t in range(max_steps):
        out = model.generate(ctx)           # THINK: sample from p_theta(. | ctx)
        act = parse_action(out)             # structured action, or a final answer
        if act.type == "final":
            return act.answer
        obs = tools[act.name](**act.args)   # ACT: harness runs it with APP authority
        ctx = ctx + [out, format_obs(obs)]  # OBSERVE: result re-enters the SAME ctx
    return "step budget exhausted"

Every agent product is a variant of these ten lines. Two load-bearing facts: the model holds no credentials — the harness does, so the security question is what the harness can do and who influences the text that drives it; and each tool observation re-enters the very context that decides the next action, with no change of type. Formally, $c_t$ concatenates trusted spans (system prompt, goal) with spans authored by tools and the world, and $\pi_\theta(a_t \mid c_t)$ conditions on all of it identically.

8.2 Tools, RAG, and the trust inversion

A tool schema (JSON Schema) is a contract on syntax; whether the call should be made is decided by the model, driven by $c_t$. The harness's validate/REGISTRY lines are the defensive foothold — ordinary, testable code where allowlists, rate limits, and approval gates attach. RAG retrieves by embedding cosine similarity ($\mathrm{score}(q,d) = \cos(E_q(q), E_d(d))$) and splices the winner into $c_t$ verbatim: similarity is not trust, and an attacker who writes a document your retrieval will rank has written into your agent's control channel without touching your servers — indirect prompt injection, the dominant agent threat.

---

Sources are cited on the corresponding slides in the main deck and details deck (Goodfellow–Bengio–Courville; Vaswani et al. 2017; Radford et al. 2019; Hoffmann et al. 2022; Holtzman et al. 2020; Ouyang et al. 2022; Rafailov et al. 2023; Gao et al. 2022; Lewis et al. 2020; Anil et al. 2024; among others). These notes are a first draft and will be revised.