How malware intercepts Windows API calls using inline trampoline hooks and IAT hooks — the mechanics of each, what trampolines are, rootkit hiding via NtQueryDirectoryFile, and detection tools.
Sign in to mark this article as read and track your progress.
API hooking intercepts calls to operating system or library functions by modifying the call path. The malware's code executes instead of — or before/after — the original function.
Why malware uses hooks:
Overwrites the first bytes of the target function with a JMP to the malware's handler. The original bytes are saved in a trampoline — a small code stub the handler can call to execute the original function.
; Target function: NtQueryDirectoryFile at 0x7C910000
; Malware writes a JMP to its hook handler at the start
; Original bytes (first 5 bytes):
; MOV EAX, 0x91 ; syscall number
; MOV EDX, 0x7FFE0300
; After hook installation:
; JMP hook_handler ; 5-byte relative JMP
; (remaining original bytes intact after offset 5)
; Trampoline stub (saved by malware before patching):
trampoline:
MOV EAX, 0x91 ; original first bytes (saved)
MOV EDX, 0x7FFE0300 ; original second instruction
JMP 0x7C910005 ; jump past the patched bytes to continue
The hook handler:
Refer to REMA eBook 2026, Figure 3.12 for the inline hook diagram showing the call flow before and after hook installation.
A rootkit hooks NtQueryDirectoryFile. When an application lists a directory:
The user's Explorer and dir command see a clean directory. The malware files exist on disk but are invisible through the OS.
Modifies the Import Address Table entries in the target process to redirect API calls to a replacement function. Simpler to implement than inline hooking — no code patching required.
Before IAT hook:
IAT entry for CreateFile → 0x7C801000 (real CreateFileA in kernel32.dll)
After IAT hook:
IAT entry for CreateFile → 0x00401500 (malware's hook function)
Any call to CreateFile via the IAT now goes to the malware handler instead.
IAT hooks are detected by comparing IAT entries against the actual addresses of the corresponding functions in the loaded DLLs. If an IAT entry points outside the DLL it claims to import from, it is hooked.
Tools: API Monitor, x64dbg Imports panel, Scylla (checks IAT integrity).
| Tool | Method | What It Finds |
|---|---|---|
| GMER | Compares function prologues against disk copy | Inline hooks in kernel and user mode |
| Rootkit Revealer | Cross-view comparison | Discrepancies between API output and raw disk/registry |
| x64dbg + HookShark | Prologue byte comparison | Inline hooks in loaded DLLs |
Volatility ssdt | Scans SSDT for modified entries | Kernel-level SSDT hooks |
| Property | Inline Trampoline | IAT Hook |
|---|---|---|
| What is modified | Function code bytes in memory | Import table pointer |
| Persistence across DLL reload | No — reloading the DLL removes the hook | Yes — IAT entry persists until process exits |
| Difficulty | Higher — requires code patching | Lower — pointer replacement only |
| Detection | Prologue byte comparison | IAT pointer validation |
| Bypasses | ASLR makes address prediction harder | Only affects code using IAT (not direct calls) |
In x64dbg:
1. Run the sample
2. View → Modules → select a suspicious DLL
3. Right-click → Find References to find all imports
4. Check each API of interest:
- Ctrl+G → type API name → Enter
- Examine first 5-10 bytes
- Normal: MOV EAX, syscall_number / PUSH EBP / MOV EBP,ESP
- Hooked: JMP 0x???????? (points outside the DLL's address range)
A JMP as the very first instruction of any API function is strong evidence of an inline hook.
ScyllaHide (covered in Unit 6) itself installs hooks to neutralise anti-debugging checks. This means hook detection tools may flag ScyllaHide's own hooks during analysis. When using ScyllaHide, expect to see additional JMP instructions at the start of NtQueryInformationProcess, IsDebuggerPresent, and related functions.