Reversing Malicious Code: Process Hollowing & API Hooking

Why injection and hooking matter

Modern malware almost never executes as a standalone process. Standalone processes are easy to terminate, easy to block, and easy to attribute. Instead, malware injects its code into other processes — often legitimate, signed, OS-level processes — and hooks API functions to redirect execution and hide its presence.

This lesson covers the four major injection techniques you will encounter in samples and the three layers of API hooking. Recognising these patterns during dynamic analysis is the difference between an analyst who says "the malware did something" and one who says "the malware did WriteProcessMemory into PID 4480 with shellcode at offset 0x10000."

Code injection — the four techniques

Code injection is the act of inserting and executing code within the address space of another running process, typically to evade detection or leverage the target process's privileges and trust relationships.

1. Classic DLL injection

The textbook approach. Five Windows APIs in sequence:

OpenProcess          → handle to target
VirtualAllocEx       → allocate memory in target's address space
WriteProcessMemory   → write the path to our DLL into that memory
CreateRemoteThread   → spawn a thread in the target that calls LoadLibraryA(our_path)
LoadLibraryA         → executes inside the target, loading our DLL

When LoadLibraryA completes, our DLL's DllMain runs inside the target process. Detection indicators: cross-process memory writes and remote thread creation. Procmon and Process Hacker flag these as CreateRemoteThread events.

2. Reflective DLL injection

A more sophisticated version. Instead of writing the DLL path and asking the target to load it via LoadLibrary, the malware writes the entire DLL bytes into the target and includes its own loader stub. The stub maps the PE manually, performs base relocations, resolves imports, and calls DllMain. The OS never knows a DLL was loaded — the target's PEB module list does not contain it.

Detection indicators: RWX memory regions (rare in legitimate code), suspicious memory pages whose backing image is not on disk.

3. Process hollowing

The technique behind much of the most evasive crimeware. Steps:

1. Create a target process (commonly a legitimate one like svchost.exe) in a suspended state.
2. Call NtUnmapViewOfSection to remove the original image from the target's address space.
3. Allocate fresh memory at the same base address.
4. Write the malicious payload there.
5. Update the target thread's context so EIP/RIP points at the payload's entry point.
6. Resume the thread.

The result: a process that appears in Task Manager as svchost.exe, signed by Microsoft, with the legitimate command line — but whose memory contains entirely different code. This is particularly effective against security tools that whitelist by process name.

4. APC injection

QueueUserAPC queues an Asynchronous Procedure Call against a thread in the target process. When the target thread enters an alertable wait state (most threads do, eventually), the queued APC fires — running our code. Quieter than CreateRemoteThread because there is no new thread, only an alert on an existing one.

Detection indicators: APC queue modifications visible only to kernel-level monitoring or very specific EDRs.

API hooking — three layers

API hooking is intercepting calls to operating system or library functions by modifying the call path, so that malicious code executes instead of, before, or after, the original function.

Rootkits use API hooking to hide their presence. By intercepting NtQueryDirectoryFile or NtQuerySystemInformation, they can filter results to remove malicious files and processes from directory listings and Task Manager. The malware is still present — but the OS appears not to see it.

Layer 1 — IAT hooking

The simplest. Each PE module has an Import Address Table — an array of pointers to the imported functions. The malware overwrites the IAT entry for, say, MessageBoxA. Now every call to MessageBoxA from this module goes to the malware's function instead. The malware can log the call, modify arguments, or skip the call entirely.

IAT hooks are easy to detect: an IAT pointer that does not point into the kernel32 module is suspicious. Modern endpoint detection tools scan for this.

Layer 2 — Inline hooking (function patching)

The malware overwrites the first few bytes of the target function in memory with a JMP to its own code. Original first bytes are saved so the hook can call the original after doing its work.

; Original first bytes of MessageBoxA:
mov  edi, edi
push ebp
mov  ebp, esp
...

; After inline hook:
jmp  malware_handler
nop
push ebp
mov  ebp, esp
...

Detection: scan critical APIs for the absence of their normal prologue. JMP as the first instruction of a Windows API is suspicious.

Layer 3 — SSDT hooking (kernel-mode)

The System Service Descriptor Table maps system call numbers to kernel function pointers. A rootkit driver patches the SSDT so calls like NtQuerySystemInformation are routed through the rootkit before reaching the real kernel handler. This is comprehensive — every process making the call is intercepted, not just one.

Modern Windows protects the SSDT with PatchGuard (Kernel Patch Protection) on x64. SSDT hooking still appears in older malware analyses and on x86 systems.

Why this matters during dynamic analysis

When you watch Procmon and see this pattern:

malware.exe → CreateProcess(svchost.exe, suspended)
malware.exe → WriteProcessMemory(svchost.exe, ...)
malware.exe → ResumeThread(svchost.exe)

You are not watching three unrelated events. You are watching process hollowing in real time. Recognising the pattern shortcuts the analysis: you now know to dump the memory of svchost.exe to recover the actual payload, and you know to ignore the legitimate-looking command line and signing because they are deliberate camouflage.

What you should be comfortable with after this lesson

• Naming all four injection techniques and the API sequence each uses
• Distinguishing IAT, inline, and SSDT hooking by mechanism and detection difficulty
• Reading a Procmon log and identifying process hollowing from event sequence alone
• Explaining why memory dumping is essential when hollowing is suspected