Advanced Unpacking Defences

Beyond the ESP Breakpoint

Standard packers (UPX, ASPack) are defeated by the ESP breakpoint technique. Commercial protectors and custom packers add defences that require more sophisticated approaches. This topic covers the defences themselves and the counter-techniques.

Unpacking defended malware is like picking a bank vault with laser grids, pressure plates, and a guard who changes the combination every hour. You need to know what defences exist before you even approach the door.

Stolen Bytes

Definition: The first few instructions from the OEP are copied into the unpacking stub and executed there. The tail jump jumps past these instructions in the original code, so the dumped file is missing its prologue.

Unpacking stub executes:
  PUSH EBP          ← stolen from OEP (executed in stub)
  MOV EBP, ESP      ← stolen from OEP (executed in stub)
  SUB ESP, 0x20     ← stolen from OEP (executed in stub)
  JMP [actual_OEP+3instructions]  ← tail jump skips the stolen bytes

Result after dump:
  OEP address:
    SUB ESP, 0x20   ← WRONG — first 2 instructions missing
    [rest of code]

The dumped file crashes immediately because the function prologue is incomplete.

Bypass:

1. Trace the unpacking stub carefully — all instructions executed before the tail jump are the stolen bytes
2. Note each stolen instruction and its bytes
3. After dumping, open the dumped file in a hex editor
4. Manually insert the stolen bytes at the beginning of the OEP function
5. Fix the IAT with Scylla as normal

Nanomites

Definition: Conditional jump instructions in the original code are replaced with INT 3 (0xCC) opcodes. An encrypted lookup table stores the original jump information (opcode, target). At runtime, each INT 3 triggers an exception — the packer''s SEH handler looks up the entry and executes the correct jump.

Original code:
  JNZ  0x004012A0           ; normal conditional jump

After nanomite application:
  INT  3                    ; 0xCC replaces the JNZ
  ; exception handler looks up: opcode=JNZ, target=0x004012A0
  ; and transfers control appropriately

A memory dump produces code full of 0xCC bytes instead of conditional jumps — completely unusable without the lookup table.

Bypass:

1. Identify the exception handler that processes nanomite INTs
2. Trace the lookup table format (how the handler maps INT 3 location → original opcode + target)
3. Script replacement of each 0xCC with the correct conditional jump instruction
4. Use x64dbg scripting or IDAPython to automate the reconstruction

This is time-consuming but systematic. Themida is the primary commercial packer using nanomites.

PE Header Erasure

After unpacking completes, some packers zero the first 0x1000 bytes of the loaded image — destroying the DOS header and PE signature.

; After OEP transfer, packer cleanup code:
    lea  edi, [image_base]       ; start of PE in memory
    xor  eax, eax
    mov  ecx, 0x400              ; zero 1024 DWORDs = 4096 bytes
    rep  stosd                   ; fill with zeros

Memory dumpers that rely on the PE header to reconstruct the file fail completely.

Bypass:

1. Dump before the header erasure code runs (set breakpoint earlier in the unpack sequence)
2. Or: reconstruct the PE header manually from a template of the same compiler version
3. PE-Bear can reconstruct a valid header from the section table alone in many cases

Guard Pages

The packer marks the unpacked memory pages as PAGE_GUARD. Any read access by a memory dumper triggers a STATUS_GUARD_PAGE_VIOLATION exception — handled by the packer to re-encrypt the page before the dumper reads it.

Memory permissions after unpacking:
  0x00400000 - 0x00450000: PAGE_EXECUTE_READ | PAGE_GUARD

Bypass:

1. In x64dbg: Memory Map → select the guarded region
2. Right-click → Set Page Guard to Off (removes the guard)
3. Then dump with Scylla normally

Anticipatory Unpacking Methodology

Rather than reacting to each trick as encountered, anticipate the packer''s behaviour based on its identified family:

Step 1: IDENTIFY
  → Run DIE/PEiD → get packer name and version

Step 2: RESEARCH
  → Search "unpacking [packer] [version] tutorial"
  → Note: stolen bytes? Nanomites? Header erasure? Guard pages?

Step 3: PREPARE
  → Enable ScyllaHide (VMProtect profile)
  → Set _NO_DEBUG_HEAP=1
  → Configure exception passthrough
  → Enable Break on TLS Callback

Step 4: EXECUTE
  → Apply ESP breakpoint technique or VirtualAlloc breakpoint
  → Reach OEP

Step 5: DUMP AND FIX
  → Remove guard pages if present
  → Dump with Scylla before header erasure (if applicable)
  → Restore stolen bytes if needed
  → Fix IAT

Step 6: VALIDATE
  → Open in DIE → should show compiler, not packer
  → Open in PEStudio → entropy < 7.0, full IAT present
  → Run in x64dbg → executes without immediate crash

Packer Reference for Anticipatory Analysis

Packer	Stolen Bytes	Nanomites	Header Erasure	Guard Pages
UPX	No	No	No	No
ASPack	Sometimes	No	No	No
Themida	Yes	Yes	Yes	Sometimes
VMProtect	Yes	Partial	Sometimes	No
Custom	Unknown	Unknown	Unknown	Unknown

For custom packers, apply the methodology iteratively — each defence encountered is diagnosed and bypassed in sequence.

Process Doppelgänging and Process Ghosting

Two advanced techniques that abuse NTFS features to evade AV scanning:

Process Doppelgänging (NTFS transactions):

1. Open a transaction (TxF)
2. Write malicious payload to a file within the transaction
3. Create a section from the transacted file
4. Roll back the transaction — file appears as if it was never written to disk
5. Create process from the section — AV cannot read the "deleted" file

Process Ghosting (delete-pending state):

1. Create a file
2. Mark it for deletion (opens a delete-pending handle)
3. Write payload to the file
4. Create a section from the file
5. Close the delete-pending handle — file is deleted
6. Create process from the section — AV finds no file to scan

Detection: Both techniques are detected by Volatility malfind and pslist/psscan discrepancies. The process exists but its backing file is absent.