This is Exam 1 for CMSC 311. This exam contains 105 points. It will be graded out of 100.

You may write your answers in whichever way you want: printing this exam and writing on paper, writing inside of this document, typesetting solutions in latex, etc…

All I care is that you do the following:

  • Don’t Google for any of the answers online
  • Take no more than 8 hours on the test
  • Cite any sources you use

You are allowed to use the internet under the following circumstances: you are accessing a manual for a specific program or a reference we’ve looked up in class. For example, you may look up the GDB manual and x86 instruction call reference, but you may not look up how to find the base address of a function using GDB on StackOverflow. If you are in doubt, write me an email. In other words, you may not Google for answers on-demand.

  • (10 points) Define, compare, and contrast probabilistic defenses versus deterministic defenses. Give at least one example of each. Are they both useful?

  • (10 points) This question has two subparts.

    • 5/10 points: Give an example of a small snippet of code that has a spatial memory safety bug.

    • 5/10 points: Describe under what circumstances (perhaps but not necessarily using example code) a program would exhibit a temporal memory safety bug.

(Stack Smashing / Buffer Overflows)

Consider the following piece of code:

// Assume that the address of `shellcode` without ASLR is 0x100000000000ff00
char shellcode[] = 
"\x48\x31\xd2"                                  // xor    %rdx, %rdx
"\x48\xbb\x2f\x2f\x62\x69\x6e\x2f\x73\x68"      // mov	$0x68732f6e69622f2f, %rbx
"\x48\xc1\xeb\x08"                              // shr    $0x8, %rbx
"\x53"                                          // push   %rbx
"\x48\x89\xe7"                                  // mov    %rsp, %rdi
"\x50"                                          // push   %rax
"\x57"                                          // push   %rdi
"\x48\x89\xe6"                                  // mov    %rsp, %rsi
"\xb0\x3b"                                      // mov    $0x3b, %al
"\x0f\x05";                                     // syscall

char *foo(int x, char *z) {
  // Assume that `buffer`'s address is 0x200000000000ff00
  // Assume that the saved instruction pointer is stored at 0x200000000000ff20
  char buffer[8];       // Line A
  // Line B
  while (x > 0) {
    buffer[x] = z[x];

int main(int argc, char **argv) {
  char *hello = foo(strlen(argv[1]), argv[1]);
  // ...
  // You will describe something here.
  return 0;

To simplify your task in the following, assume that you can call the program with binary command-line arguments. In other words, assume that you can give the program a command-line argument containing the byte 0 (even though you couldn’t typically because that’s the end of a C string).

  • (5 Points) Give me command line arguments to the program that would cause it to crash.

  • (10 Points) (Stack smashing) For now, assume that ASLR has been turned off in our system. This means that each time the program starts, the static data segment, code segment, stack, etc.. will all get the same value each time the program is run. For this problem, assume that the buffer shellcode is located at address 0x10000000ffaa0000.

Describe an input that would cause the program to execute the code in shellcode.

Payload injection

  • (10 points) Now assume that the shellcode variable is removed from the program. We can’t play the same trick as in the last problem. Instead, we must inject the shellcode via the program’s input. You may still assume that ASLR and NX are turned off, but for this problem, assume that the base address of buffer is 0x200000000000ff00.` Describe an input that would inject the shellcode and redirect execution to it.

  • (10 points) How would you change the program to eliminate this attack?

  • (5 points) Describe what the NX bit does. Why does it make it hard to launch the attack described in problem 5?


(5 points) A “NOP-sled” is a long sequence of NOPs before a payload that is inserted into the program.

... thousands more ...
"\x48\x31\xd2"                                  // xor    %rdx, %rdx
"\x48\xbb\x2f\x2f\x62\x69\x6e\x2f\x73\x68"      // mov	$0x68732f6e69622f2f, %rbx
"\x48\xc1\xeb\x08"                              // shr    $0x8, %rbx
  • Describe why a NOP-sled might be useful in the presence of ASLR or in the scenario in which you weren’t certain where the program would be loaded (e.g., because of variations in environment variables).

Heap exploits

Assume that we had the following structure.

struct person {
    // Address base+0
    unsigned int is_administrator;
    // Address base+4
    char username[50];
    // Address base+56
    char *(*login_shell)(char *);  // Pointer to C function that accepts a
                                   // string (char *) as an argument and returns a string (char *)

Assume the compiler lays out structured data for person in the manner described in the comments. I.e., if the base address of the struct is at address 0xAA00, then the address of the login_shell variable will be stored at 0xAA38. You may assume the function system is stored at the (8-byte) address 0x000000AAAAAA0000.

Assume that after you are logged in, the program calls the function pointed to by login_shell with the data stored in username.

  • (10 points) Describe what you could put in username so that you effected a call to system("/bin/sh").

ASLR / attack theory

  • (10 points) Assume that ASLR is enabled. Are stack-canaries and other memory-safety controls still relevant? If they are still relevant, write an example program which demonstrates a vulnerability that would exist even with ASLR. If not, explain how ASLR fixes the problem stack canaries solve. (Big Hint: think about data-only attacks, consider writing something like a password-checking program that calls a function to copy a string.)

Control-flow integrity

Consider the following program:

void foo() {
  char buf[200];
  // .. 
  bar(); // Line 1
  // .. 

void bar() {
  char buf2[100];
  // .. 
  return; // Line 3;

void system() {
  // .. 
  • (10 points) Control-flow integrity is a compile-time (or runtime, in some cases) instrumentation of the program.

    • 2/10 points: What does control-flow integrity do, and why is it helpful?

    • 4/10 points: In the above program, how would the following be transformed:
      • The call to bar() at line 1
      • The return from bar() at line 3
    • 2/10 points: How does CFI ensure the program can’t call system?

    • 2/10 points: Assume we had provably perfect CFI. Would we have then ensured memory-safety?

Systems Security Concepts

  • (5 points) Programs that are Set-UID inherit the privileges of their owners when they are called. For example, the program ping uses raw ICMP sockets to check the status of remote hosts, which requires the program to have the privileges of the root user.

In 3-6 sentences, detail why such Set-UID programs warrant extra attention from us.

  • (5 points) Consider that Process A contains secret data somewhere in its memory space. For example, maybe it is a server process that stores grades for the course. Sometimes, Process A might want to run untrusted code (e.g., downloaded from the internet). If done by naively running code in the same process space, this lead to security vulnerabilities, because the untrusted code will have accesss to all of process A’s memory.

What operating-system principle could be used to allow process A to run this untrusted code in a way so that the untrusted code does not gain access to process A’s data?