Reverse engineering and malware analysis

UPJŠ study programs

Credits

2

Ending with:

classified evaluation

Range:

1C/14C

Semester:

winter

Teachers

N/A

About the subject

The course focuses on the systematic process of reverse engineering and malware analysis, which aims to reveal the internal architecture, logic and functionality of software even without access to the original source code. Students will gradually become familiar with static analysis, which includes working with assembler or tools such as Ghidra, and subsequently with dynamic analysis to observe the behavior of malicious code in a controlled environment (sandbox). An important part of the course is an in-depth examination of the Linux and Windows operating systems, their memory structures, API calls and ELF and PE binary formats. Attention is also paid to advanced techniques that malware uses to hide itself and make analysis difficult, such as polymorphism, obfuscation, code injection into other processes or anti-debugging mechanisms. Another important block is the security of IoT devices and reverse engineering of firmware in embedded systems with regard to specific architectures such as ARM. Individual lectures are structured in a way that provides new facts, while important aspects of the subject are repeated and gradually developed, with the aim of ensuring a comprehensive understanding of reverse engineering and malware analysis by students.

Target group

Students of master's degree programs in computer science and applied computer science.

Learning Objectives

  • Understanding Code Transformation: Understand how logic from higher-level programming languages ​​(such as C) is transformed into low-level assembly, specifically during compilation (e.g. GCC on Linux).
  • Identifying Hidden Mechanisms: Learn to identify cryptographic routines and other hidden functions in machine code.
  • Analyzing Malware Defense Techniques: Be able to analyze and overcome advanced techniques that malware uses to hide itself and make analysis difficult, such as obfuscation, anti-debugging, and anti-VM (anti-virtualization) mechanisms.
  • Adopting a "dirty hands" approach and low-level principles: Teach students to synthesize knowledge of the internal structures of operating system kernels, processor architectures, and binary formats (such as ELF and PE), as relying solely on automated GUI tools for modern threat analysis is not enough.
  • Improving hybrid analysis skills: Master the combination of static analysis (e.g. in Ghidra to gain insight into the structure) with dynamic analysis (e.g. debugging with Frida to observe real-world code behavior and decipher context in an isolated environment).
  • Investigating modern and specific threats: Gain skills for reverse engineering and emulating IoT and embedded device firmware and become familiar with modern attack techniques such as eBPF rootkits and fileless malware.

Brief course outline

  1. Introduction to the study of reverse engineering and malware analysis.
  2. Memory architecture analysis and binary vulnerability exploitation. s of firmware of single-purpose devices
  3. Basic aspects of low-level language (assembler) and low-level machine code analysis.
  4. Machine code conversion to instructions and source code — disassembling and decompiling.
  5. Dynamic analysis of machine code and program behavior.
  6. Advanced code analysis and dynamic debugging of machine code.
  7. Advanced reverse engineering techniques in the Linux operating system.
  8. Advanced malware analysis techniques in the Linux operating system environment.
  9. Fundamentals of reverse code analysis in Windows OS.
  10. Advanced reverse code analysis techniques and malware analysis in Windows operating system.
  11. Obtaining and analyzing firmware of single-purpose devices.
  12. Advanced reverse analysis of single-purpose device firmware.

Lectures

Conditions for completing the course

Ending with: classified evaluation

Final exam

Course methodology

Recommended literature

    1. SIKORSKI, Michael — HONIG, Andrew. Practical Malware Analysis: The Hands-On Guide to Dissecting Malicious Software. San Francisco : No Starch Press, 2012.
    2. EAGLE, Chris — NANCE, Kara. The Ghidra Book: The Definitive Guide. San Francisco : No Starch Press, 2020.
    3. DENNIS, Andriesse. Practical Binary Analysis: Build Your Own Linux Tools for Binary Instrumentation, Analysis, and Disassembly. San Francisco : No Starch Press, 2019.
    4. HALE LIGH, Michael — CASE, Andrew — LEVY, Jamie — WALTERS, Aaron. The Art of Memory Forensics: Detecting Malware and Threats in Windows, Linux, and Mac Memory. Indianapolis : Wiley, 2014.
    5. SVAJCER, Vanja. Mastering Malware Analysis: The Complete Malware Analyst’s Guide to Combating Malicious Software, APT, Cybercrime, and IoT Attacks. 2nd ed. Birmingham : Packt Publishing, 2022.
    6. CHEN, Aditya — ZADDACH, Jonas — COSTIN, Andrei. Firmware Security: Vulnerabilities, Exploits, and Best Practices for IoT Devices. Hoboken : Wiley, 2025.

     

    Academic and research platforms

    1. MITRE ATT&CK Framework: https://attack.mitre.org/
    2. MITRE CWE (Common Weakness Enumeration): https://cwe.mitre.org/
    3. VirusTotal: https://www.virustotal.com/