OS-Level Exploit Mitigations

Why Mitigations Matter for Analysts

Modern operating systems do not rely on programmers writing bug-free code. They deploy hardware and software mitigations that make exploitation harder even when a vulnerability exists.

Analysts need to understand these mitigations because:

• Malware that bypasses them uses specific, recognisable techniques (ROP chains, information leaks)
• Understanding the mitigation explains why certain malware behaviours exist
• Bypass attempts leave forensic artefacts that appear in dynamic analysis

ASLR — Address Space Layout Randomisation

ASLR randomises the base addresses of the executable image, stack, heap, and loaded DLLs on each process execution. An attacker who discovers a buffer overflow cannot hardcode a jump address because the target address changes on every run.

Entropy by architecture

Architecture	Entropy (Stack)	Brute-force Feasibility
32-bit Windows	8–9 bits	Feasible in seconds to minutes
64-bit Windows (HEASLR)	17+ bits	Infeasible
64-bit Linux (PIE)	28+ bits	Infeasible

32-bit ASLR is weak. Malware targeting 32-bit systems sometimes brute-forces ASLR by repeatedly crashing and restarting the target process until a predictable address is hit.

ASLR bypass techniques

Technique	Mechanism
Information disclosure	Leak a pointer to calculate actual base address
Partial overwrite	Modify only the lower bytes (unaffected by ASLR randomisation)
Non-ASLR modules	Target DLLs compiled without /DYNAMICBASE flag
Brute force	Feasible on 32-bit systems with low entropy
Heap spray	Fill large memory region with shellcode — increases probability of hitting it

What to look for

In dynamic analysis, a crash followed by a retry loop in the malware suggests ASLR brute-forcing. An information disclosure followed by a calculated jump target suggests a leak-based bypass.

DEP — Data Execution Prevention

DEP marks the stack and heap as non-executable. The CPU refuses to execute instructions fetched from pages without the Execute permission, enforced via the NX (AMD) or XD (Intel) bit in page table entries.

Even if an attacker injects shellcode into a data region, the CPU raises an access violation when attempting to execute bytes from a non-executable page.

DEP bypass: Return-Oriented Programming (ROP)

ROP chains together small snippets of existing executable code called gadgets — sequences ending in RET — that already exist in loaded modules.

Stack layout for a ROP chain:
┌──────────────────────────────┐
│  Address of gadget 1         │  ← return address (overwritten)
│  Address of gadget 2         │  ← popped by gadget 1's RET
│  Address of gadget 3         │  ← popped by gadget 2's RET
│  Address of VirtualProtect() │  ← called to make shellcode executable
│  Arguments to VirtualProtect │
└──────────────────────────────┘

No injected code executes — only existing code in executable regions. DEP is bypassed entirely.

ROP gadgets look like this in disassembly:

pop  eax     ; ← this is the gadget
ret          ; ← RET chains to next gadget address on stack

Finding ROP gadgets in loaded DLLs is automated by tools like ROPgadget and mona.py.

ASLR + DEP together

DEP prevents code injection. ASLR prevents reliable gadget addressing. Defeating both requires:

1. An information disclosure to find module base addresses (defeats ASLR)
2. A ROP chain using gadgets from those modules (defeats DEP)

Malware that uses both bypass techniques is sophisticated and likely from a professional threat actor.

Stack Canaries

A random value — the canary — is placed between local buffers and the saved return address during function prologue. Before RET executes, the epilogue verifies the canary value. If corrupted, the process terminates immediately.

Stack layout with canary:
┌─────────────────────────┐
│  Return Address          │  ← target of overflow
│  Saved EBP               │
│  Stack canary (random)   │  ← verified before RET
│  Local buffer            │  ← overflow starts here
└─────────────────────────┘

An overflow must corrupt the canary before reaching the return address — triggering immediate termination.

Bypass

Requires knowing the canary value in advance via an information disclosure vulnerability — reading the canary value from memory before triggering the overflow, then including the correct value in the exploit payload.

Compiler flags

Compiler	Flag	Notes
MSVC	/GS	Default since Visual Studio 2003
GCC	-fstack-protector	Protects functions with arrays
GCC	-fstack-protector-strong	Protects more functions

Summary

Mitigation	Defeats	Bypass Technique
ASLR	Hardcoded addresses	Info leak, brute force (32-bit), non-ASLR modules
DEP/NX	Shellcode injection	ROP chain using existing executable code
Stack canary	Return address overwrite	Info leak to read canary value first

ASLR and DEP are most effective together. Bypassing both requires an information leak and a ROP chain — significantly raising the bar for exploitation.