In-Depth Malware Analysis: Unpacking & Memory Forensics

Why packing matters

Most malware distributed in the wild is packed. A packer compresses, encrypts, or otherwise transforms the original executable so that its code and data are unreadable on disk. When the packed file runs, a small unpacking stub executes first, restores the original payload in memory, and transfers control to it.

Packing defeats signature-based antivirus because the on-disk bytes change completely, producing a different hash for every packed variant. It also defeats static analysis because strings, imports, and code are hidden until runtime. The analyst must recognise packed binaries and unpack them before meaningful analysis can begin.

Identifying packing — four indicators

Four structural anomalies, observable without executing the file, reliably indicate packing:

Indicator	What to look for
High entropy	Shannon entropy above 7.0 (on a 0–8 scale) in the .text or primary code section. Normal compiled code sits between 5.0 and 6.5. High entropy in .rsrc alone is less suspicious because legitimate apps store compressed images there.
Truncated import table	The IAT contains only LoadLibraryA and GetProcAddress. A normal Windows program imports dozens of API functions. The stub uses these two to resolve all other imports at runtime.
Non-standard section names	Sections named UPX0/UPX1, .aspack, .themida, .petite, or random strings. Standard compilers produce .text, .data, .rdata, .rsrc.
VirtualSize ≫ SizeOfRawData	In section headers, a section's VirtualSize (size in memory) is much larger than its SizeOfRawData (size on disk). The OS reserves empty memory that the stub fills with the decompressed payload.

Tools like Detect It Easy (DIE), PEiD, and Exeinfo PE identify known packers by matching signatures against internal databases. DIE also displays per-section entropy graphs.

Manual unpacking — the ESP breakpoint technique

For UPX and many other simple packers, the ESP breakpoint is the fastest path to the Original Entry Point (OEP).

The principle: the unpacking stub starts by PUSHA (or several PUSH instructions) to save register state. This decreases ESP. After the stub finishes unpacking, it executes POPA (or matching POPs) to restore state, raising ESP back to its original value. At that exact moment, ESP holds its initial value — and the next instruction is the OEP.

ESP breakpoint procedure

1. Open the packed binary in x64dbg. Run to the entry point.
2. Note the value of ESP. Switch to the dump pane, navigate to that ESP value.
3. Right-click on the four-byte word at ESP. Select Breakpoint → Hardware, on access → Dword.
4. Press F9 (Run). The unpacking stub executes. When it finishes its POPA and writes back to that stack location, the hardware breakpoint fires.
5. The current EIP is at, or one or two instructions before, the OEP. Look for a JMP to a far address — that is the jump from stub to OEP.

Memory dumping and IAT reconstruction

At the OEP, the unpacked code is in memory but not on disk. Dump it:

1. Use the Scylla plugin in x64dbg. Click PE Rebuild with the OEP set to the address you reached.
2. Click IAT Autosearch. Scylla scans memory for the unpacked Import Address Table.
3. Click Get Imports. Scylla resolves each import and rebuilds the IAT.
4. Click Dump to write the unpacked binary to disk, then Fix Dump to attach the rebuilt IAT.

The output is a fully unpacked, runnable PE that can be analysed with all the static-analysis tools again.

Memory forensics with Volatility

Sometimes you cannot manually unpack — perhaps the malware is fileless, perhaps it self-destructs after running. Memory forensics extracts evidence from a snapshot of RAM.

Acquiring the dump

On a live VM: winpmem, DumpIt, or VMware's .vmem file (created when the VM is suspended) all produce a memory image.

Save the image off the VM before reverting the snapshot.

Volatility plugin workflow

# What process are we looking at?
vol -f mem.dmp windows.pslist

# Hidden processes (cross-view detection)
vol -f mem.dmp windows.psscan

# Loaded DLLs per process
vol -f mem.dmp windows.dlllist --pid 4480

# Handles open by a process — files, registry keys, mutexes
vol -f mem.dmp windows.handles --pid 4480

# Network connections
vol -f mem.dmp windows.netscan

# Injected code — the workhorse for hollowing detection
vol -f mem.dmp windows.malfind

# Dump a process's memory for further analysis
vol -f mem.dmp windows.memmap --pid 4480 --dump

malfind is the workhorse. It scans every process for memory regions with execute permission that are not backed by an image file on disk — the signature of injected code, hollowing, or shellcode. It dumps each suspicious region for offline disassembly.

Workflow example

1. pslist — find a suspicious process (unusual name, unusual parent, unusual path).
2. dlllist against that PID — check for DLLs not on disk or loaded from temp paths.
3. malfind — find injected code in the process or elsewhere.
4. netscan — correlate any network connections originating from the suspicious process.
5. memmap --dump — extract the suspicious memory for offline disassembly with Ghidra.

What you should be comfortable with after this lesson

• Recognising a packed binary from two of the four indicators within sixty seconds
• Performing manual unpacking with the ESP breakpoint and reaching the OEP
• Reconstructing the IAT with Scylla and producing a clean dump
• Running the core Volatility plugins (pslist, malfind, netscan) and interpreting their output