How emerging AI systems change the practice of security, in both directions: AI as a tool for attack and defense, and attacks on AI systems themselves. We study classical foundations (reverse engineering, memory attacks, automated exploit generation) and build up to the state of the art (AI agents for vulnerability discovery, adversarial ML, prompt injection, jailbreaking, and LLM/agent escapes). Because this class discusses a rapidly-emerging area, content will be adjusted on-demand to emergent concerns.
Meetings: Tu/Th 5:00–6:20 PM · CST 4-201 · Aug. 25 – Dec. 8, 2026 · Kristopher Micinski (kkmicins@syr.edu). See the syllabus (or the official PDF) for official course policies and grading.
Lecture
Discussion day
Guest lecture
Research presentation
Exam
Project / HW
Slides, notes, and assignments are posted as we reach each topic.
Schedule
Week 1 · Foundations of Security
-
8/25Course Introduction & the Security + AI Landscape
- Slides
- Demo: adversarial images, FGSM vs. PGD — run in Colab · source
-
8/27Security Foundations: CIA, Threat Modeling, Information Flow
- Slides
- Information Flow Demo App
- Anderson, Security Engineering, Ch. 1
- Thompson, "Reflections on Trusting Trust" (CACM 27(8), 1984) — mandatory
- Bell & LaPadula, "Secure Computer Systems: Mathematical Foundations" (1973) — required for graduate students
Week 2 · ML Background & the AI Threat Surface
-
9/1Machine Learning Background: Regression, Classification, Evaluation
- Slides
- Interactive notes — work through it yourself
- Linear and logistic regression, GLMs, overfitting, base rates
- Lab: spam classification, four ways
-
9/3Neural Networks and LLM Intro
- Slides
- Demo: tokens, next-token prediction, and the scratchpad — run in Colab · new to Colab? start here
- Optional — 3Blue1Brown, "Backpropagation, intuitively" (Ch. 3) and "Backpropagation calculus" (Ch. 4)
- Panfilov et al., "Stealing Reasoning Traces from Proprietary LLM APIs" (2026)
- Optional deep-dive: interactive notes
Week 3 · LLM Agents, Tool Use & Agent Security
-
9/8LLM Agents & Tool Use: The Agent Loop, Trust Boundaries, and the Control/Data Collapse
-
9/10Securing Agentic Systems: Capabilities, Information Flow & What Defenses Actually Buy
- Slides
- Odersky, Zhao, Xu, Bračevac & Pham, "Securing Agents With Tracked Capabilities" (CAIS '26) — read §1–3 for Thursday (open-access PDF)
- Presentation #1 — Odersky et al., "Securing Agents With Tracked Capabilities" (TACIT)
Week 4 · Reverse Engineering & Memory Attacks
-
9/15Reverse Engineering I: Binary Analysis & AI-Assisted Decompilation (tentative)
- Hands-on: Ghidra
- Aleph One, "Smashing the Stack for Fun and Profit" (Phrack 49, 1996)
- Shoshitaishvili et al., "SoK: The Art of War" (I–IV)
-
9/17Reverse Engineering II: Stacks, Buffer Overflows, ROP (tentative)
- Presentation #2 — Shacham, "The Geometry of Innocent Flesh on the Bone" (ROP)
Week 5 · Mitigations & Automated Exploit Generation
-
9/22Exploit Primitives, Mitigations & Symbolic Execution (tentative)
- ASLR, DEP/NX, canaries, CFI — then path conditions, constraint solving, and a solved input
- KLEE (OSDI '08); AEG (NDSS '11); Anderson, Ch. 4 (Access Control)
-
9/24AI-Assisted Exploitation: LLM-Guided Fuzzing & the Bug→Exploit Gap (tentative)
- Presentation #3 — Cha et al., Mayhem (S&P '12)
- Fang et al. (2024) + critiques — demand the denominator
Week 6 · AI Agents for Vulnerability Discovery & Code Understanding
-
9/29Agent-Computer Interfaces, Cyber-Reasoning Systems, DARPA AIxCC (tentative)
- Yang et al., SWE-agent (2024)
-
10/1AI Agents for Code Understanding & Auditing at Scale (tentative)
- Presentation #4
- Xu et al., "When LLMs Meet Cybersecurity" (2025)
Week 7 · Exam 1
-
10/6Exam 1 Review (tentative)
- Worked problems from Weeks 1–6; bring questions
-
10/8Exam 1 (in class)
- Focus topics TBA
Week 8 · Fall Break; Deep Learning & Adversarial ML (guest)
-
10/13Fall Break — no class
- University Fall Break, Mon 10/12 – Tue 10/13
-
10/15Deep Neural Networks and Adversarial Machine Learning (tentative)
- Guest lecture: Ed Raff — author of Inside Deep Learning
- Presentation #5
Week 9 · AI & Network Security: Detection and Evasion
-
10/20ML-Based Intrusion Detection: Traffic Analysis & Feature Engineering (tentative)
- Mirsky et al., Kitsune (NDSS '18); Anderson, Ch. 21
-
10/22Adversarial Evasion (FGSM, PGD) & Defenses (tentative)
- Presentation #6
- Goodfellow et al. (2015); Carlini & Wagner (2017)
Week 10 · AI-Powered Social Engineering & Exam 2
-
10/27LLM Phishing, Voice Cloning & Deepfakes — and Defending Against Them (tentative)
- Schmitt & Flechais (2024)
- Human detection rates; provenance and out-of-band verification
-
10/29Exam 2 (in class)
- Focus topics TBA
Week 11 · Prompt Injection & Jailbreaking
-
11/3Prompt Injection: Direct vs. Indirect — the Control/Data Collapse (tentative)
- Greshake et al. (2023)
-
11/5Jailbreaking: Taxonomies, Universal/Transferable Attacks, Why Defenses Fail (tentative)
- Presentation #7
- Zou et al., GCG (2023)
Week 12 · LLM & Agent Escapes
-
11/10Agent Escapes: Tool-Use Breakout, Code Execution, Exfiltration, MCP Risk (tentative)
- Emerging-attacks module
-
11/12Self-Propagating Attacks & AI Worms; Poisoning & the Model Supply Chain (tentative)
- Presentation #8
Week 13 · Securing AI Systems & Red-Teaming
-
11/17Threat-Modeling LLM Apps: Trust Boundaries, Tool-Use Risk, Exfiltration (tentative)
- OWASP Top 10 for LLM Applications (2025)
- HW4 assigned (write-up TBA)
-
11/19Adaptive Attacks & Building an Evaluation Harness (tentative)
- Presentation #9 (final slot)
- Nasr, Carlini, et al. (2025), revisited
Thanksgiving Break · 11/22–11/29 · no class
Week 14 · Ethics, Policy & Exam 3
-
12/1Policy, Ethics, Disclosure & the Law: Dual-Use, CFAA, Responsible Disclosure (tentative)
-
12/3Exam 3 (in class)
- Focus topics TBA
Week 15 · Class Project Presentations
-
12/8Class Project Presentations (tentative)
- Last day of classes
-
FinalsOptional Final Exam
- Final-exam period (Dec 10–15)
- Cumulative and optional — replaces your lowest midterm; skip it if you are satisfied with the three