You will train four spam classifiers on real SMS messages and compare them. The point is not the accuracy score. The point is to see, on a real dataset, what a model is, how training works, and why “99% accurate” is a nearly useless claim until you look at precision and recall.

There are two versions of the code. They use the same features and produce almost the same numbers.

  • spam_lab.py is pure Python. No installs. It runs anywhere you have python3, including a locked-down lab machine.
  • spam_torch.py is the same thing in PyTorch, so you can run it on your laptop’s GPU. Setup for a MacBook Pro is below.

Run it (no setup)

python3 spam_lab.py

That trains all four models and prints a report in about one second. To score your own message:

python3 spam_lab.py --classify "WINNER!! You have won a £1000 prize. Txt CLAIM to 80086"

The data

The SMS Spam Collection: 5,574 real text messages, hand labeled. 747 are spam, 4,827 are ham (legitimate). So 13.4% of messages are spam. Keep that number in mind; it decides how to read every result below.

A few real examples from the file:

ham    Ok lar... Joking wif u oni...
ham    Nah I don't think he goes to usf, he lives around here though
spam   Free entry in 2 a wkly comp to win FA Cup final tkts 21st May 2005. Text FA to 87121...
spam   WINNER!! As a valued network customer you have been selected to receive a £900 prize...

We split the data 80/20 into a training set (4,458 messages) and a test set (1,116). Every model learns from the training set only. Every number reported is measured on the test set, which the model never saw during training. This is the one rule you cannot break: grading a model on the data it trained on tells you nothing.

Features: bag of words

A model does arithmetic, not English, so each message becomes a vector of word counts. We take the 2,000 most common tokens in the training set, and represent a message by how many times each of those tokens appears. We keep !, $, £, #, and ? as tokens, because they carry a lot of the spam signal and a plain word-splitter would throw them away.

That is the whole featurizer. tokenize() and build_vocab() in spam_lab.py are about fifteen lines together.

The four models

All four take the same word-count vector and produce a spam score. They differ in the shape of the function and how they are trained.

Linear regression. Fit a straight line to the 0/1 labels by least squares, then call anything scoring above 0.5 spam. This is the wrong tool: it fits a line to a yes/no answer, so its output is an unbounded score, not a probability. It also destabilizes easily. Try --epochs 25 with a larger learning rate and watch the F1 fall apart. It is here so you can see why we reach for the next model.

Logistic regression. Same linear score, but squashed through a sigmoid into a real probability, and trained by gradient descent on cross-entropy. This is the update derived in lecture: the gradient of the loss with respect to the score is just (predicted − actual). It is fast, and its weights are readable (see below).

Naive Bayes. Count how often each word shows up in spam versus ham, apply Bayes’ rule, assume words are independent. This is the classic spam-filter method, the one behind Paul Graham’s “A Plan for Spam.” No gradient descent at all, just counting.

Neural network. One hidden layer of 16 ReLU units, trained by backpropagation. This is the same machinery as the neural-networks lecture, only wider. Because messages are short, the features are sparse, so even this trains in a few seconds of pure Python.

Results

From an actual run of python3 spam_lab.py:

                 accuracy  precision  recall    F1
linear             98.12     99.24    86.67    92.53
logistic           98.21     97.79    88.67    93.01
naive Bayes        98.75     97.89    92.67    95.21
neural net         98.39     96.48    91.33    93.84

Read these carefully, because the accuracy column is a trap.

  • A model that labels everything ham, without looking at the message at all, scores 86.6% accuracy on this data, because 86.6% of messages are ham. So “98% accurate” is only about 11 points better than a model that does nothing.
  • Precision is: of the messages we flagged as spam, how many really were. High precision means few false alarms. A false alarm here means a real message sent to the spam folder, which is worse than missing a spam.
  • Recall is: of the real spam, how much we caught. Naive Bayes has the best recall (92.7%) and the best F1, so on this dataset it is the model to beat.
  • Linear regression has the highest precision but the worst recall: it is cautious, so it rarely false-alarms but misses more spam.

This is the base-rate lesson from lecture, made concrete. When one class is rare, accuracy is dominated by the common class and hides everything that matters.

What logistic regression learned

The model’s weights are just numbers attached to words. Large positive weight means “this word pushes toward spam.” From the same run:

most spammy tokens:  £(+3.36) text(+2.71) txt(+2.30) 150p(+2.22) ringtone(+2.14) reply(+2.06) stop(+1.93) uk(+1.91)
most hammy tokens:   my(-1.39) gt(-1.37) lt(-1.36) #(-1.15) da(-1.13) i'll(-1.10) amp(-1.07) him(-1.07)

Nobody told the model that £, txt, 150p, and ringtone are spammy. It found them by counting. The hammy list is mostly ordinary conversational words (my, him, i'll). This readability is why logistic regression is still deployed: you can inspect exactly why a message was flagged.

Sample predictions

$ python3 spam_lab.py --classify "WINNER!! You have won a £1000 prize. Txt CLAIM to 80086 now"
[logistic regression]  P(spam) = 1.000  ->  SPAM

$ python3 spam_lab.py --classify "hey are we still on for lunch tomorrow?"
[logistic regression]  P(spam) = 0.002  ->  ham

The full run also prints the model’s mistakes. The interesting misses are spam that avoids the usual vocabulary, and ham that happens to use it. That is exactly where an attacker who wanted to slip past the filter would aim, which is the subject of the adversarial-ML week: the model’s weights are a map, and anyone can read it.

The security reframe

Every step here is an attack surface later in the course.

  • The training set is an input. Whoever can add labeled messages to it can move the weights. That is data poisoning.
  • The features and weights are legible. An attacker can see which words to avoid and which to add. That is evasion.
  • The reported accuracy is meaningless without the base rate. A vendor quoting “99% accurate” on a rare-event detector is quoting a number a do-nothing model nearly matches. That is the hype filter.

The PyTorch version (MacBook Pro)

spam_torch.py reuses the same featurizer and trains the linear, logistic, and neural-net models with PyTorch. It picks the fastest device your machine has: the Apple GPU (MPS) on Apple Silicon, CUDA on an NVIDIA box, or the CPU.

Setup:

cd labs/spam-classification
./setup_macbook.sh          # creates .venv, installs torch, reports the device
source .venv/bin/activate
python spam_torch.py

Or with make:

make setup     # one time
make torch      # train the PyTorch models
make run        # the pure-Python version, no venv needed

Notes for a MacBook Pro:

  • On Apple Silicon (M1/M2/M3/M4), the default pip install torch wheel includes the MPS backend. The script uses it automatically and prints device: mps. No CUDA, no special index URL.
  • On an Intel Mac, PyTorch runs on the CPU. This dataset is small, so a run still finishes in seconds.
  • If setup_macbook.sh warns that your Python is not arm64, you installed an Intel build of Python under Rosetta. Install an arm64 python3 (python.org or Homebrew) to get the GPU backend.
  • Force a device if you want to compare: python spam_torch.py --device cpu.

The PyTorch numbers will not match the pure-Python numbers to the decimal, because the optimizer (Adam), batching, and weight initialization differ. They land in the same place: high-90s accuracy, Naive Bayes and logistic regression near the top, linear regression cautious and behind. If they diverge wildly, that is a bug worth finding, not a result worth trusting.

Things to try

  • Change --max-features. Does a bigger vocabulary help, or just slow it down?
  • The dataset is 13.4% spam. Weight the loss toward the rare class (pos_weight in BCEWithLogitsLoss) and watch recall rise and precision fall. Which error is worse for a spam filter? For a malware detector?
  • Write a message that you think is obviously spam but that the model misses. What did you avoid? You have just done manual evasion.

Attribution

Dataset: Almeida, T.A., Gómez Hidalgo, J.M., Yamakami, A. “Contributions to the Study of SMS Spam Filtering: New Collection and Results.” DOCENG’11, 2011. Distributed for research and education; see data/UCI-README.txt.