How emerging AI systems change the practice of security, in both directions: AI as a tool for attack and defense, and attacks on AI systems themselves. We study classical foundations (reverse engineering, memory attacks, automated exploit generation) and build up to the state of the art (AI agents for vulnerability discovery, adversarial ML, prompt injection, jailbreaking, and LLM/agent escapes). Because this class discusses a rapidly-emerging area, content will be adjusted on-demand to emergent concerns.
Meetings: Tu/Th 5:00–6:20 PM · CST 4-201 · Aug. 25 – Dec. 8, 2026 · Kristopher Micinski (kkmicins@syr.edu). See the syllabus (or the official PDF) for official course policies and grading.
Lecture
Discussion day
Guest lecture
Research presentation
Exam
Project / HW
Slides, notes, and assignments are posted as we reach each topic.
Schedule
Week 1 · Foundations of Security
-
8/25Course Introduction & the Security + AI Landscape
-
8/27Security Foundations: CIA, Threat Modeling, Information Flow
- Slides
- Anderson, Security Engineering, Ch. 1
- Thompson, "Reflections on Trusting Trust" (CACM 27(8), 1984) — mandatory
- Bell & LaPadula, "Secure Computer Systems: Mathematical Foundations" (1973) — required for graduate students
Week 2 · ML Background & the AI Threat Surface
-
9/1Machine Learning Background: Regression, Classification, Evaluation
- Slides
- Interactive notes — work through it yourself
- Linear and logistic regression, GLMs, overfitting, base rates
-
9/3Neural Networks and LLM Intro
- Slides
- Optional — 3Blue1Brown, "Backpropagation, intuitively" (Ch. 3) and "Backpropagation calculus" (Ch. 4)
- Panfilov et al., "Stealing Reasoning Traces from Proprietary LLM APIs" (2026)
- Optional deep-dive: interactive notes
Week 3 · LLM Agents, Tool Use & Agent Security
-
9/8LLM Agents & Tool Use: The Agent Loop, Trust Boundaries, and the Control/Data Collapse
-
9/10Securing Agentic Systems: Capabilities, Information Flow & What Defenses Actually Buy
- Slides
- Odersky, Zhao, Xu, Bračevac & Pham, "Securing Agents With Tracked Capabilities" (CAIS '26) — read §1–3 for Thursday (open-access PDF)
- Presentation #1 — Odersky et al., "Securing Agents With Tracked Capabilities" (TACIT)
Week 4 · Reverse Engineering & Memory Attacks
-
9/15Reverse Engineering I: From Source to Binary — Compilation, ELF/PE, and What Gets Thrown Away
- Slides
- Hands-on: Ghidra
- Aleph One, "Smashing the Stack for Fun and Profit" (Phrack 49, 1996)
- Shoshitaishvili et al., "SoK: The Art of War" (I–IV)
-
9/17Reverse Engineering II: Disassembly, Decompilation, Ghidra in Practice & Where AI Comes In
- Slides
- Presentation #2 — Shacham, "The Geometry of Innocent Flesh on the Bone" (ROP)
Week 5 · Mitigations & Automated Exploit Generation
-
9/22Exploits, System-Level Defenses, and Coverage-Guided Fuzzing
- Slides (deck runs across 9/22 and 9/24)
-
9/24Symbolic Execution, SMT Solving & Mayhem: Automatic Exploit Generation on Binaries
- Slides (continued from 9/22)
- Presentation #3 — Cha et al., Mayhem (S&P '12)
- Discussion: Cha, Avgerinos, Rebert & Brumley, "Unleashing Mayhem on Binary Code" (S&P '12) — read before class
Week 6 · AI Agents for Vulnerability Discovery & Code Understanding
-
9/29Agent-Computer Interfaces, Cyber-Reasoning Systems, DARPA AIxCC
- Slides
- Lab: find and exploit a memory-safety bug three ways (fuzzing, symbolic execution, ret2win). run in Colab · new to Colab? start here
- Yang et al., SWE-agent (2024)
-
10/1AI Agents for Code Understanding & Auditing at Scale
- Slides
- Presentation #4
- Li, Dutta & Naik, "IRIS: LLM-Assisted Static Analysis for Detecting Security Vulnerabilities" (ICLR '25) — read before class
Week 7 · Exam 1
-
10/6Exam 1 Review
- Class project announced — topics & scoping
- Worked problems from Weeks 1–6; bring questions
-
10/8Exam 1 (in class)
- Focus topics TBA
Week 8 · Fall Break; Deep Learning & Adversarial ML (guest)
-
10/13Fall Break — no class
- University Fall Break, Mon 10/12 – Tue 10/13
-
10/15Deep Neural Networks and Adversarial Machine Learning
- Guest lecture: Ed Raff — author of Inside Deep Learning
Week 9 · LLM Security
-
10/20Introduction to LLM Security & Prompt Injection
- Slides (deck runs across 10/20 and the following LLM-security sessions)
-
10/22Poisoning & Distillation Attacks on LLMs
- Slides
- Presentation — Carlini et al., "Stealing Part of a Production Language Model" (ICML 2024)
Week 10 · Securing LLM-Integrated Applications & Network Security
-
10/27Securing LLM-Integrated Applications (tentative)
- Class project proposal due — Tue 10/27, 11:59 PM
- Class project work begins — six weeks, through 12/8
-
10/29Network Security + AI (tentative)
- Presentation — information flow (reading TBA)
Week 11 · Web Security & Exam 2
-
11/3Web Security + AI (tentative)
-
11/5Exam 2 (in class)
- Covers Weeks 9–11
Week 12 · AI for Program Analysis & Reverse Engineering
-
11/10AI for Reverse Engineering (tentative)
-
11/12Guest Lecture: Rajvardhan Oak (Meta)
- Guest lecture: Rajvardhan Oak, Meta — topic TBA
Week 13 · Information Flow & Securing AI Systems
-
11/17Information-Flow Control for AI Agents (tentative)
-
11/19AI-Powered Social Engineering (tentative)
Thanksgiving Break · 11/22–11/29 · no class
Week 14 · Ethics, Policy & Exam 3
-
12/1Policy, Ethics & the Law (tentative)
-
12/3Exam 3 (in class)
- Focus topics TBA
Week 15 · Class Project Presentations
-
12/8Class Project Presentations (tentative)
- Class-project deliverables due
- Last day of classes
-
FinalsOptional Final Exam
- Final-exam period (Dec 10–15)
- Cumulative and optional — replaces your lowest midterm; skip it if you are satisfied with the three