Binary Reverse Engineering

CSci 5980/8980: Manual and Automated Binary Reverse Engineering

Instructor: Stephen McCamant
Office: 4-225E Keller Hall
E-Mail: mccamant@cs.umn.edu
Home page: http://www.cs.umn.edu/~mccamant
Office hours: Wednesdays 1-12pm and TBD. Also by appointment, or available for a quick talk when my office door is open

Class Schedule: Mondays and Wednesdays, 9:45-11:00am, Rapson Hall 43

Course Overview:
Software binaries, also called executables, object code, or machine code, are the output of compilation and the final form of programs for execution on a CPU. Binary formats are designed for efficiency, rather than for ease of modification. Nevertheless there are a number of situations where it is valuable to be able to understand code that is available only in binary form: when inter-operating with commercial software, when investigating security vulnerabilities, or when understanding malicious software (malware). This task is called "reverse engineering" of software because it is about recovering an understanding of software closer to what the original developers had. Some subtasks of binary reverse engineering are called disassembly and decompiling, corresponding to reversing the processes that assemblers and compilers do when software is compiled.

This course introduces the area of reverse engineering of binary code from both manual and automated perspectives: we will start by looking at binary code mostly by hand, and then see which things can and can't be automated by open-source and research tools. The first part of the course will cover manual reverse engineering skills through lectures and homework assignments; then the later stages of the class will transition to a research seminar in which we read and discuss research papers about automatic binary reverse engineering techniques. Students will complete a substantial final project in groups of up to three. The project should address a novel and generalizable problem, either by performing reverse engineering or by researching tools to improve reverse engineering capabilities. Near the end of the semester, groups will submit final reports, and give an in-class presentation.

Prerequisites:
The only prerequisite of the course is a basic familiarity with C, assembly language, and low-level programming. In the UMN CS undergraduate program, these topics are covered in CSci 2021, and similar classes at other institutions would also be sufficient. Students with other backgrounds should discuss with the instructor.

Paper Readings:
The primary readings for the second part of the course will be in the form of research papers. I'll post links to these papers to the course web site. Some are completely public downloads; others are licensed to the University via the libraries so you can access them directly if you're coming from a campus IP address, or from off campus you can use the library's proxy service and bookmarklet.

Textbooks:
Most of the papers we read will cover material that is too new to appear in any textbooks, but you may find some textbooks useful for background reference. Two books that are commonly used in 5271 and 5471 respectively are convenient for this purpose because their online versions are available for viewing free of charge:

• Ross Anderson's Security Engineering, Second Edition (John Wiley & Sons, 2008 or online at the author's web page)
• Manezes, Van Oorschot, and Vanstone's Handbook of Applied Cryptography

Grading breakdown:
15%: Reading questions
10%: Class attendance and discussion
15%: In-class paper presentation(s)
10%: Hands-on demo assignment
50%: Research project (including report and presentation)

Exams: There will be no exams.

Project: A major component of the course is large research project. More details on the project will be on a separate page.

Assignments: The assignments for the class include a short writing assignment related to each paper we read, plus presenting papers in class and presenting a hands-on demo of how to use a research system. More details on the assignments are on a separate page.

External Sources:
Most assignments in the class will allow or even encourage the use of resources beyond the course readings and lecture notes, such as you might find in the library or on the Internet. However it is an important academic value, which we enforce rigorously in this class, that it is never acceptable to use another's work without properly acknowledging it. In writing assignments, you should acknowledge any external sources of inspiration or code directly in your answer; in the course project report, you should acknowledge resources and related work in the same was as you would in an academic paper. Failure to do so constitutes plagiarism.

Academic Integrity Policies: By the nature of this class, we will often discuss techniques that could be used to compromise the security of certain computer systems. However, IT IS VERY IMPORTANT THAT YOU NEVER APPLY THESE TECHNIQUES TO A COMPUTER WITHOUT THE PERMISSION OF THE COMPUTER'S OWNER. In particular you should never attempt to attack the security of computers that belong to CSE Labs, the department, the University, or an unsuspecting classmate. If we learn that a student has unethically exploited a vulnerability discussed in class, THAT STUDENT WILL FAIL. This is in addition to any University-level, department-level or legal penalties such an action may be subject to.

More generally, you are expected to do your own academic work and cite sources as appropriate. Failing to do so is scholastic dishonesty. Scholastic dishonesty includes, but is not limited to: plagiarizing; cheating on assignments or examinations; engaging in unauthorized collaboration on academic work; taking, acquiring, or using test materials without faculty permission; submitting false or incomplete records of academic achievement; acting alone or in cooperation with another to falsify records or to obtain dishonestly grades, honors, awards, or professional endorsement; altering, forging, or misusing a University academic record; or fabricating or falsifying data, research procedures, or data analysis. A student found responsible for scholastic dishonesty will at a minimum receive a grade of 0 for the assignment in question and be reported to the campus-wide Office for Community Standards (OCS). More serious offenses will receive a grade of F (or N) for the course and be subject to additional sanctions from the University. You should also read this page about academic conduct in computer science.

Other Applicable Policies: There are a number of other University-wide policies that apply to this course which you should be familiar with. This list is an abridged summary of longer policies which you can find linked from a University-wide page:

• Students are required to abide by the Student Conduct Code, which among other things prohibits disruptive classroom conduct.
• Personal electronic devices should be used with caution in the classroom lest they interfere with your or other students' learning.
• Students will not be penalized for absence during the semester due to unavoidable or legitimate circumstances. Such circumstances include verified illness, participation in intercollegiate athletic events, subpoenas, jury duty, military service, bereavement, and religious observances. The requirement of verification for absences due to illness is waived for a single episode absence that did not require professional treatment, and did not lead to missing an important in-class event such as an exam.
• The University considers that accepting compensation for taking and distributing classroom notes violates shared norms and standards of the academic community.
• Sexual harassment is not acceptable in the University setting.
• The University provides equal access to and opportunity in its programs and facilities, without regard to race, color, creed, religion, national origin, gender, age, marital status, disability, public assistance status, veteran status, sexual orientation, gender identity, or gender expression.
• The University of Minnesota is committed to providing equitable access to learning opportunities for all students, including making reasonable accommodations. If you have, or think you may have, a disability that might affect your participation in class please contact the Disability Services office. If you are registered with DS and have a current letter requesting reasonable accommodations, please contact your instructor as early in the semester as possible to discuss how the accommodations will be applied in the course.
• As a student you may experience a range of mental health concerns or stressful events which may interfere with learning. You can learn more about the broad range of confidential mental health services available on campus via the Student Mental Health website.
• Within the scope and content of the course as defined by the instructor, academic freedom includes the freedom to discuss relevant matters in the classroom and conduct relevant research. Students are free to take reasoned exception to the views offered in any course of study and to reserve judgment about matters of opinion, but they are responsible for learning the content of any course of study for which they are enrolled. (Adapted from The AAUP Joint Statement on Rights and Freedoms of Students.)