ML Introduction

CIS 400/600 · Week 2 · companion to the ML Introduction deck. These notes walk the same ladder the lecture walks, in the same order, at reading speed, with six widgets in the places where a slide could only show you the answer. One corpus runs through all of it: the UCI SMS Spam Collection, 5,574 real messages, 747 spam, split with SEED = 400 into 4,458 training and 1,116 held-out. It is the same corpus and the same split the paired lab uses. Every figure below is either quoted from a slide or recomputed in your browser by the page itself; the widgets are plain JavaScript, and the source is in this file.

1 · What learning is

Deck slides 2–4.

Supervised learning starts with examples of inputs and the answers we wanted,

$$ (x_1, y_1),\ (x_2, y_2),\ \dots,\ (x_n, y_n), $$

and asks for a function $f_\theta$ with $f_\theta(x) \approx y$ on future inputs. The input $x$ is an email, a network packet, a house's square footage. The answer $y$ is spam or not, attack or benign, the sale price. And $\theta$ is the knobs we get to turn — the settings of the knobs are the model.

The word doing the work is "future". Memorizing the examples is trivial and proves nothing; the whole game is inputs you have not seen, and that gap is where overfitting and attackers both live.

The simplest model

Predict a number as a weighted sum of the features plus an offset:

$$ \hat{y} = w_1 x_1 + w_2 x_2 + \cdots + w_d x_d + b. $$

With one feature this is $\hat{y} = wx + b$, a line, and the knobs are the slope and the intercept. On the 4,458 training messages, predicting word count from character length, least squares picks $\hat{y} = 0.197x + 0.67$ with $R^2 = 0.96$; the eyeballed line of slope $0.30$ on the same slide has 19 times the squared error. Small as this family is, a large share of deployed security ML is a linear model over hand-built features, because it is fast and auditable.

The loss

Score one prediction with the squared error, and a whole candidate model by its average over the examples:

$$ \text{loss for one example} = (\hat{y} - y)^2, \qquad \text{total} = \frac{1}{n}\sum_{i=1}^{n} \left(f_\theta(x_i) - y_i\right)^2. $$

Training is the search for the knob settings with the lowest total loss. The loss is what turns "which model is better" into a number, and a number is something you can search.

2 · The normal equations

Deck slides 5–6.

Take three points, $(1,2)$, $(2,4)$, $(3,5)$. Stack one row per example, with a column of ones so the bias is just the weight on a feature that never varies:

$$ \underbrace{\begin{bmatrix} 1 & 1 \\ 2 & 1 \\ 3 & 1 \end{bmatrix}}_{X}\; \underbrace{\begin{bmatrix} w \\ b \end{bmatrix}}_{\text{parameters}} \;=\; \begin{bmatrix} w + b \\ 2w + b \\ 3w + b \end{bmatrix} \quad\text{should land near}\quad \underbrace{\begin{bmatrix} 2 \\ 4 \\ 5 \end{bmatrix}}_{y} $$

Write the total squared error and expand it:

$$ L(w) \;=\; \lVert Xw - y \rVert^2 \;=\; w^\top X^\top X w \;-\; 2\,w^\top X^\top y \;+\; y^\top y. $$

That is an ordinary quadratic in $w$. Differentiate one parameter at a time and set the result to zero:

$$ \nabla L = 2X^\top X w - 2X^\top y = 0 \qquad\Longrightarrow\qquad \boxed{\;X^\top X\,w \;=\; X^\top y\;} $$

$X^\top X$ and $X^\top y$ are numbers you compute from the data, so the boxed line is a system of linear equations — two equations in two unknowns here — and solving it gives the best line exactly.

The three points by hand

Every entry of the Gram matrix is a dot product of two feature columns:

$$ X^\top X = \begin{bmatrix} 14 & 6 \\ 6 & 3\end{bmatrix}, \qquad X^\top y = \begin{bmatrix} 25 \\ 11 \end{bmatrix}, \qquad \det = 42 - 36 = 6. $$

The $14$ is $1 + 4 + 9$, the $6$ is $1 + 2 + 3$, and the $3$ is just $n$. On the right, $25 = 1\cdot 2 + 2\cdot 4 + 3\cdot 5$ and $11 = 2 + 4 + 5$. Inverting the $2\times 2$:

$$ \begin{bmatrix} w \\ b \end{bmatrix} = \frac{1}{6}\begin{bmatrix} 3 & -6 \\ -6 & 14\end{bmatrix}\begin{bmatrix} 25 \\ 11\end{bmatrix} = \frac{1}{6}\begin{bmatrix} 9 \\ 4 \end{bmatrix} = \begin{bmatrix} 3/2 \\ 2/3 \end{bmatrix}. $$

The fitted values are $13/6$, $11/3$, $31/6$, so the residuals are $r = (\tfrac16, -\tfrac13, \tfrac16)$ and $\mathrm{SSE} = \lVert r \rVert^2 = \tfrac16 \approx 0.167$. Two sanity checks fall out of the boxed equation: $X^\top r = 0$, so the residual is orthogonal to both feature columns, and since one of those columns is all ones, the residuals sum to zero.

import torch

X = torch.tensor([[1., 1.],          # each row is [x_i, 1]
                  [2., 1.],          # the 1 is the bias column
                  [3., 1.]])
y = torch.tensor([[2.], [4.], [5.]])

print("X^T X =", (X.T @ X).tolist(), " X^T y =", (X.T @ y).flatten().tolist())

w = torch.linalg.lstsq(X, y).solution          # solves X^T X w = X^T y
r = X @ w - y                                  # the residual vector
print("w, b  = [%.6f, %.6f]" % tuple(w.flatten().tolist()))
print("SSE   = %.6f" % (r * r).sum().item())
print("X^T r =", [round(v, 7) for v in (X.T @ r).flatten().tolist()])

X^T X = [[14.0, 6.0], [6.0, 3.0]]  X^T y = [25.0, 11.0]
w, b  = [1.500000, 0.666668]
SSE   = 0.166667
X^T r = [2.4e-06, 1.4e-06]

Training is that one lstsq call. No learning rate, no epochs, no seed, and the same answer on every machine.

3 · Why we iterate anyway

Deck slide 7.

An exact solution exists, and almost nobody uses it. There are two reasons.

Scale. Solving $X^\top X w = X^\top y$ costs $O(nd^2 + d^3)$, and on the 2,000 bag-of-words features of this very corpus $X^\top X$ is singular anyway: $\mathrm{rank}(X) = 1958$ of $2001$, partly because one verbatim repeated spam template contributes four identical columns. The condition number is about $1.1 \times 10^{42}$, and an unpivoted QR raises nothing while returning garbage.

Shape. Change the loss or change the model and $\nabla_\theta L(\theta) = 0$ stops being a linear system. There is nothing left to solve, only something to search: from where you stand, step downhill, repeat.

$$ \theta \leftarrow \theta - \eta \cdot (\text{slope at } \theta) $$

On the slide's example $L(\theta) = (\theta - 3)^2$ the slope is $2(\theta - 3)$, so with $\eta = 0.1$ the update is $\theta \leftarrow 0.8\,\theta + 0.6$, and from $\theta_0 = 0$ the iterates run $0 \to 0.60 \to 1.08 \to 1.464 \to \cdots \to 3$. The learning rate $\eta$ sets the step: too small it crawls, too large it diverges.

From here on there is usually no formula at all. This update rule, run on batches, is how every model in the rest of the course is fitted.

4 · Classification by regression

Deck slides 8–11.

Label ham $0$ and spam $1$, fit the same least-squares line, then threshold the fitted value at $0.5$. No new machinery. Two features, both of them countable by hand: the length of the message in characters, and how many of those characters are digits.

	mean chars	mean digits
ham (3,861)	71.2	0.30
spam (597)	138.9	15.96

Spam carries fifty times the digits, and the fitted weights say the model noticed: $w = [0.000413,\ 0.04395,\ -0.004822]$ over $[\text{length},\ \text{digits},\ 1]$. One digit character moves the score as much as 106 characters of text do. The fit is three lines, run from inside the lab folder:

import torch, spam_lab                    # the lab's loader; SEED = 400
train, test = spam_lab.split(spam_lab.load(spam_lab.DEFAULT_DATA))

def design(rows):                         # x = [length, digits, 1]
    X = [[len(t), sum(c.isdigit() for c in t), 1.] for _, t in rows]
    return torch.tensor(X), torch.tensor([[float(l)] for l, _ in rows])

X, y = design(train); Xte, yte = design(test)
w = torch.linalg.lstsq(X, y).solution     # argmin ||Xw - y||^2
acc = ((Xte @ w >= 0.5).float() == yte).float().mean()
print([round(v, 6) for v in w.flatten().tolist()], "%.2f%%" % (100 * acc))
# [0.000413, 0.04395, -0.004822] 96.33%

On the 1,116 held-out messages, 150 of them spam, that scores 96.33% accuracy, 98.23% precision and 74.00% recall. Before being impressed: always saying "ham" scores 86.56% on the same test set, 966 messages out of 1,116. Accuracy is a number that has to beat something.

The three ways it breaks

1. The output is not a probability. Test scores run over $[-0.004,\ 1.597]$, and 4.21% of them fall outside $[0,1]$. Nothing in the model was ever told to stay in range.
2. Squared error punishes being confidently right. Target $1$, prediction $1.597$, loss $(0.597)^2 = 0.356$; a message scraping over the line at $0.51$ costs only $(0.49)^2 = 0.240$. The loss is more upset by the message it got emphatically right.
3. So points far out on the correct side drag the boundary toward them. Twenty constructed messages at 800 characters and 120 digits, added to the 4,458 training messages, swing the fit enough to drop test recall from 74.00% to 34.67%. Logistic regression on the same contaminated data does not move.

The threshold was never justified

Nothing about $0.5$ came out of the fit; it is where the two labels' midpoint happens to sit. Since the boundary just slides along the score, moving the threshold trades the two error types against each other, and the widget below is the whole trade on the real held-out set.

Sliding the threshold buys back recall, and it fixes neither of the other two defects. What we want next is an output that is a probability by construction, and a loss that stops caring once a point is confidently right.

5 · Logistic regression

Deck slides 12–15.

Security questions are rarely "predict a number". They are attack or benign, and the useful answer is a probability. Keep the weighted sum and squash it into $(0,1)$:

$$ \sigma(z) = \frac{1}{1 + e^{-z}}, \qquad p(\text{attack} \mid x) = \sigma\!\left(w^\top x + b\right). $$

Invert that and the model is a straight line in the log-odds:

$$ \log \frac{p}{1-p} \;=\; w^\top x + b. $$

So every weight is an odds multiplier, which is why a logistic model can be audited coefficient by coefficient. The two-feature fit has $w_{\text{dig}} = 0.6237$: one more digit character multiplies the odds of spam by $e^{0.6237} = 1.87$. And $\sigma$ is monotone, so $p \ge 0.5$ exactly when $w^\top x + b \ge 0$ — the boundary is still a straight line. The shape of the decision did not change. What changed is the scale the answer is reported on, and the loss used to fit it.

Where the closed form dies

Least squares had a formula because setting the gradient to zero left a linear system. Here $\sigma$ wraps around the unknown $w$, so no rearranging gets $w$ back out on its own, and there is nothing left to solve.

loss	model	$\nabla L = 0$	what you do
squared	linear	a linear system	a formula
cross-entropy	linear	not a linear system	descend
anything	a network	not a linear system	descend, no guarantee

Row two is still convex, so descent gets the global optimum and the only cost is that you have to iterate. Row three is where that guarantee goes, and we get there in section 9.

The same linear map on 2,000 features

Same corpus, same split, now with 2,000 word-count features instead of two hand-counted ones. Both models below fit the identical linear map. Only the loss differs:

model   = nn.Linear(2000, 1)                      # z = w.x + b, the LOGIT
loss_fn = nn.BCEWithLogitsLoss()                  # sigma applied inside the loss

2,000 features, same linear map	acc	prec	recall	F1	FP
exact least squares (lstsq)	97.22	94.07	84.67	89.12	8
logistic regression (BCE)	98.48	99.26	89.33	94.04	1

Accuracy moved 1.26 points, which on features this easy is close to noise. The error that costs you moved from eight false positives to one, and in spam filtering a false positive is somebody's real mail deleted.

Because the output is now a probability, the threshold follows from cost instead of habit. A missed attack costing a hundred times a false alarm puts it at $0.0099$. The familiar $0.5$ is what you get when you assume the two costs are equal, and that assumption is usually made by not thinking about it.

6 · Softmax and cross-entropy

Deck slides 16–17.

The spam filter answered one yes/no question. Plenty of real questions are $K$-way: what is this flow doing — benign, port scan, brute force, exfil? Take $K$ raw scores $z = (z_1,\dots,z_K)$, called logits, exponentiate so everything is positive, then divide by the total so it sums to one:

$$ \mathrm{softmax}(z)_k = \frac{e^{z_k}}{\sum_{j=1}^{K} e^{z_j}}, \qquad p_k > 0, \qquad \sum_{k=1}^{K} p_k = 1. $$

  class          logit z_k      e^{z_k}          p_k
  benign            +2          7.389056       0.6439
  port-scan         +1          2.718282       0.2369
  brute-force        0          1.000000       0.0871
  exfil             -1          0.367879       0.0321
                           sum = 11.475217   sum = 1.0000

Four exponentials and one division. The logits come from exactly the same place the spam score did — a weighted sum of features — except that there is one score per class instead of one score total.

Scoring the answer is one line, at any $K$:

$$ H(y, p) = -\sum_{k=1}^{K} y_k \log p_k \;=\; -\log p_{k^\star} \quad\text{when } y \text{ is one-hot at the true class } k^\star. $$

The negative log of the probability you gave the right answer. Permuting the same four logits shows the range:

prediction	logits	$p_{\text{exfil}}$	loss $-\ln p$
confident and correct	$(-1, 0, 1, 2)$	0.6439	0.4402
hedging (uniform)	$(0, 0, 0, 0)$	0.2500	1.3863 $= \ln 4$
confident and wrong	$(2, 1, 0, -1)$	0.0321	3.4402

Being confidently wrong is unbounded, so a handful of such points can dominate a batch. And $K = 2$ is not a different model: the softmax of the two logits $(0, z)$ is $(1 - \sigma(z),\ \sigma(z))$, so the spam filter of the last section is this section at $K = 2$.

7 · Where x comes from

Deck slides 18–20.

One-hot encoding

Numbering a log's categorical levels — TCP=0, UDP=1, ICMP=2 — asserts not just an order but a metric: $\text{ICMP} - \text{UDP} = \text{UDP} - \text{TCP}$, and ICMP is "twice" UDP. Suppose the malicious rate goes up and then down: 0.10, 0.90, 0.20. Every line $\hat y = ax + b$ is monotone, so no line can follow that. Solving the normal equations on the three points gives $a = 0.05$, $b = 0.35$, predicting 0.35 / 0.40 / 0.45 against a truth of 0.10 / 0.90 / 0.20, with SSE $0.375$. That fit is exact, not undertrained — it is the best line available, and it is wrong on all three protocols. The failure is in the representation.

One-hot gives each level its own axis. Then $X = I_3$, so $X^\top X = I$ and the normal equations return $w = X^\top y = (0.1, 0.9, 0.2)$: each weight is just that protocol's rate, and the fit is exact.

import torch, torch.nn.functional as F
idx = torch.tensor([0, 1, 2, 1, 0])        # TCP UDP ICMP UDP TCP
X = F.one_hot(idx, num_classes=3)
# [[1,0,0],[0,1,0],[0,0,1],[0,1,0],[1,0,0]]
# X.shape (5, 3)   X.dtype torch.int64  -> .float() before nn.Linear

The cost is one parameter per level. Fine for three protocols; a port field with 65,536 of them gets an embedding instead. The 2,000-dimensional bag-of-words vector used above is a sum of one-hots, one axis per vocabulary word.

One recipe, many models: GLMs

Everything so far is the same three-part recipe. A generalized linear model is a weighted sum, then a link function chosen to match what you are predicting; the loss follows from the link.

Predict	Weighted sum, then…	Name
a number	…nothing (identity)	linear regression
a probability	…sigmoid	logistic regression
a count (events/hour)	…exponential	Poisson regression

Keep the shape of that table in mind for section 9, where a neuron turns out to be one row of it.

Where the features come from

Someone has to decide what $x$ is, and in security that decision is the system:

• Email and phishing: sender-domain age, URL entropy, mismatch of display name against address.
• Network: packets per flow, byte histograms, timing intervals, port fan-out.
• Malware: imported API calls, section entropy, opcode n-grams.

Hand-built features	Learned features (Week 8 onward)
auditable, cheap	found by the model itself
capped by human imagination	often stronger
legible to attackers	opaque, with new failure modes

Neither column dominates. The choice is a design decision with a threat model attached.

8 · Measuring honestly

Deck slides 21–23.

Never grade a model on the examples it trained on.

  all labeled data
  ├── training set  (fit the knobs here)
  └── test set      (touch ONCE, at the end — this is the grade)

Training loss can be driven to nearly zero by memorizing, which proves nothing. The test set is an estimate of performance on future inputs, and it is the thing anyone is actually buying.

Overfitting

A model with enough capacity scores perfectly on training data by memorizing it, while learning nothing usable. On twelve real messages, a degree-eleven polynomial passes through every point: training error 0.00, held-out error 244. A straight line on the same twelve misses every point slightly: 6.69 training, 6.24 held out. The line is worse where it was fitted and better everywhere else.

The fixes are one line each: more data, a simpler model, a penalty for complexity (regularization: add $\lambda \cdot (\text{size of weights})$ to the loss), stop training early.

Base rates

Your IDS is 99% accurate. An alert just fired. Is it an attack?

Suppose 1 in 10,000 events is a real attack and the detector is 99% accurate in both directions. Out of 1,000,000 events:

	flagged	not flagged
100 attacks	99	1
999,900 benign	9,999	989,901

An alert is a real attack $99/(99 + 9{,}999) = 0.98\%$ of the time. Ninety-nine percent of the alert queue is false, at "99% accuracy". The two numbers that survive this are precision (of what we flagged, how much was real) and recall (of what was real, how much did we flag), quoted at the threshold you deploy.

9 · Past the linear wall

Deck slides 24–28.

The wall

Four points: $(0,0)$ and $(1,1)$ in one class, $(0,1)$ and $(1,0)$ in the other. No straight line separates them. Tilt the line however you like and one point lands on the wrong side, and the reason is two additions. The class-1 points require $w_2 + b > 0$ and $w_1 + b > 0$; adding those gives $w_1 + w_2 + 2b > 0$. The class-0 point $(0,0)$ forces $b \le 0$, so $w_1 + w_2 + b > -b \ge 0$, which contradicts the requirement $w_1 + w_2 + b \le 0$ coming from $(1,1)$.

Training nn.Linear(2, 1) on it settles at 50% accuracy, and a brute-force sweep of 226,981 weight settings never beats 75%. That 50% is the global optimum of a convex problem, so nobody tunes their way out of it.

A neuron

Every model on this page has been a weighted sum followed by a squash. Drawn as one unit, that gets a name. The word is historical branding, not a claim about brains.

the squash	the model	where you met it
none	linear regression	the least-squares fit
sigmoid	logistic regression	the spam filter
$\max(0, z)$, called ReLU	a hidden unit	new today

The bag-of-words spam filter is nn.Linear(2000, 1) and a sigmoid: 2,001 parameters, one neuron. Two of the three rows are models already fitted on real data earlier on this page.

Stacking

Feed a layer of neurons into another neuron and you have a network. Nothing in the data says what the middle column should hold, so the network invents those features itself; stated in the right features, the problem gets easy and the last neuron finishes it with a straight line. The wall falls exactly this way: two ReLU units and nine parameters, set by hand with no training at all, compute XOR.

Why the squash matters

Pull the squash out and stacking buys nothing, because a straight line of a straight line is a straight line. In one variable it is arithmetic:

$$ 3(2x + 1) + 4 \;=\; 6x + 7. $$

The same algebra runs with matrices and says the same thing. A hundred layers with nothing between them collapse into one layer drawing one boundary, and you are back at the four unseparated points. ReLU is the standard choice, and the only property that matters today is that it is not a straight line.

What depth costs

Width is how many units stand in one layer; depth is how many hidden layers are stacked; hidden means nobody observes those layers, and no label ever says what the third one should have held. That last point is most of why these models are hard to audit.

Depth pays when the thing you are predicting is built in stages — bytes, then opcodes, then functions, then behaviour. It buys nothing on our spam features, where "contains ringtone" is already the abstraction the decision needs. What it always costs is parameters. One neuron over the 2,000-word vocabulary is $2001$: one weight per word, plus a bias. Sixteen hidden units feeding one output is $16 \cdot 2001 + 17 = 32{,}033$, and 32,000 of that sits in the first matrix. So "how big is my network" is mostly "how big is my vocabulary".

Today in one screen

Deck slide 29.

$$ \text{model } f_\theta \;\longrightarrow\; \text{loss scores it} \;\longrightarrow\; \text{fit } \theta \;\longrightarrow\; \text{grade on held-out data} $$

  ONE corpus: 5,574 SMS, 4,458 train / 1,116 test, 13.4% spam

  exact least squares ─▶ logistic (BCE) ─▶ softmax + CE ─▶ one hidden layer
  lstsq, closed form     descent, convex   the same loss   32,033 params
  97.22%   8 FP          98.48%   1 FP     spam is K = 2   98.57%, untrained

  always-say-ham baseline: 86.56% = 966/1116 — the number every claim must beat

• A closed form needs a squared loss and a linear model. Change either and stationarity stops being a linear system; at 2,000 features $X^\top X$ is singular anyway, rank 1958 of 2001.
• One-hots encode a category, and a bag of words is a sum of them. Cross-entropy over a softmax handles any number of classes; spam is the $K = 2$ case.
• XOR needs a hidden layer: no line does it. A trained linear model scores 50%, the best one that exists 75%, and two ReLU units with 9 parameters get it exactly. Delete the ReLU and a hundred stacked layers collapse into one layer drawing one boundary.
• Parameters are what depth and width cost: one neuron on bag of words is 2,001, sixteen of them 32,033, and 99.9% of that sits in the first matrix.
• Base rates and held-out data decide whether a detector is usable, not accuracy.

The last rung is the honest one. The hidden layer scored 98.57% against logistic regression's 98.48% — 0.09 points, about one message out of 1,116 — and that number is the lab's own run, quoted as a baseline, not something trained here. The hype filter points inward too.

---

Everything on this page follows the ML Introduction deck and stops where it does. Backpropagation and training dynamics are Week 8; adversarial examples are Week 9. These notes are a first draft and will be revised.