Evasion at the Code Level: Oligomorphic, Polymorphic, and Metamorphic Malware

Why Malware Rewrites Itself

Signature-based antivirus works by matching byte patterns. Change the bytes, and the signature fails. Three techniques describe how malware changes its own appearance to break this matching. They sit on a spectrum of increasing complexity and evasion power.

Refer to REMA eBook 2026, Figure 1.5 for the full evasion tree diagram showing all three techniques side by side with detection difficulty.

The Spectrum

All three techniques share one property: the malware produces a different binary on each infection while preserving identical payload behaviour. They differ in what changes and how much changes.

Property	Oligomorphic	Polymorphic	Metamorphic
What mutates	Decryptor stub only	Decryptor stub only	Entire binary
Variant count	Fixed (tens to hundreds)	Unlimited (mutation engine)	Unlimited (rewrite)
Encrypted payload	Unchanged	Unchanged	N/A — code itself changes
Hash per copy	Rotates within set	Unique every time	Unique every time
AV evasion	Temporary	Defeats hash signatures	Defeats hash + pattern
Detection approach	Signature all stubs	Emulation, heuristics	Behavioural, semantic
Real example	Whale (1990)	Storm Worm (2007)	Virut

Oligomorphic Malware

The malware carries a small, fixed set of decryptors — typically a few dozen to a few hundred. On each infection, it picks one at random.

Simple to implement, but the AV vendor only needs to write signatures for the entire set and the evasion is defeated. The Whale virus (1990) is a textbook oligomorphic example: it carried roughly 30 different decryption routines.

Think of it as a spy who has only 30 different disguises memorised. Once law enforcement photographs all 30, the disguises stop working.

Polymorphic Malware

The malware generates a new decryptor on every infection using a mutation engine. The encrypted payload stays functionally identical, but the outer shell — the decryption stub — is different each time, producing a new hash with every copy.

Think of a student who rewrites the same exam answers in different words for every submission. A plagiarism checker that compares word patterns will miss the duplication, even though the meaning is identical.

Storm Worm (2007) was a polymorphic email worm that changed its decryptor with every message, defeating hash-based AV at scale.

How the mutation engine works

A polymorphic engine typically applies three or more transformations to the decryptor:

• Instruction reordering — swap independent instructions
• Register substitution — replace EAX with EBX throughout
• Junk insertion — add NOP sleds or equivalent no-op sequences
• Equivalent instruction substitution — INC EAX instead of ADD EAX, 1

The encrypted payload that follows the stub is untouched — only the stub varies.

Metamorphic Malware

The malware rewrites its entire code structure on each generation, not just the decryptor. Instructions are reordered, equivalent instructions are substituted, dead code is inserted, and register assignments are shuffled. The binary is completely different each time, but the behaviour is identical.

Virut is a well-known metamorphic virus that used instruction permutation and register reassignment to produce unique copies.

Metamorphic instruction substitution example

A metamorphic engine replaces instructions with functionally equivalent alternatives. The CPU produces the same result in every case, but the byte sequence differs, so a fixed signature cannot match across generations.

; Generation 1          ; Generation 2
ADD  EAX, 1             INC  EAX

; Generation 3          ; Generation 4
SUB  EAX, -1            LEA  EAX, [EAX+1]

All four instructions add 1 to EAX. A YARA rule matching the bytes 83 C0 01 (the encoding of ADD EAX, 1) catches Generation 1 but misses the other three entirely.

Detection Strategies

Against oligomorphic

Write signatures for the entire known set of stubs. The fixed pool is the weakness — once catalogued, the evasion fails permanently.

Against polymorphic

Emulation: the AV engine simulates execution of the sample in a lightweight sandbox until the decryptor runs and the payload decrypts itself, then scans the decrypted payload.

Heuristics: flag any binary that contains a self-modifying loop writing to its own code section and then executing the modified region.

Against metamorphic

Behavioural analysis: ignore byte patterns entirely. Monitor what the code does at runtime — process injection, registry persistence, network beaconing — rather than what it looks like on disk.

Semantic analysis: convert code to an intermediate representation, normalise equivalent instructions, and compare the normalised form to known malware logic.

Practical Implication for Analysts

When a sample produces zero or few VirusTotal detections but behaves suspiciously in a sandbox, evasion is the likely explanation. The next step is not more static analysis — it is dynamic analysis and manual debugging to reach the real payload.

Key point: A "clean" VirusTotal result does not mean a sample is benign. It may mean the malware has successfully evaded signature matching. Always combine static results with dynamic behaviour.