DLL Analysis

DLL vs EXE

Both use the PE format. The difference is in execution:

• An EXE has an entry point (main or WinMain) and runs as an independent process
• A DLL exports functions for other programs to call. It cannot execute on its own. When loaded, the OS calls DllMain with reason code DLL_PROCESS_ATTACH

Malware authors place their payload inside DllMain under the DLL_PROCESS_ATTACH case. The malicious code runs the instant the DLL is loaded — before any exported function is called. A DLL injected into svchost.exe or explorer.exe does not appear as a separate process in Task Manager.

Refer to REMA eBook 2026, Figure 3.9 for the execution path comparison between EXE and DLL malware.

DllMain Structure

BOOL WINAPI DllMain(HINSTANCE hinstDLL, DWORD fdwReason, LPVOID lpvReserved) {
    switch (fdwReason) {
        case DLL_PROCESS_ATTACH:
            // Payload runs HERE — immediately on load
            break;
        case DLL_THREAD_ATTACH:
            break;
        case DLL_THREAD_DETACH:
            break;
        case DLL_PROCESS_DETACH:
            break;
    }
    return TRUE;
}

In disassembly, look for a CMP on the second argument (fdwReason) followed by a switch dispatch. The DLL_PROCESS_ATTACH case (value 1) is the code path that runs on injection.

Identifying DLL Files

In PEStudio or PE-Bear, a DLL is identified by:

• Characteristics flag in the File Header: IMAGE_FILE_DLL (0x2000) is set
• Extension is typically .dll, .ocx, .cpl, or .drv — but malware often uses .dat, .log, or no extension at all
• No AddressOfEntryPoint pointing to main — it points to DllMain instead

Always verify by examining the Characteristics bitmask, not the file extension.

Export Analysis

The Export Directory lists functions the DLL makes available to callers. In PEStudio: navigate to Exports tab.

Export types and their implications

Export Type	Example	Implication
ServiceMain	Windows service DLL	Load via sc create / sc start
DllRegisterServer	COM object	May register BHO or shell extension for persistence
DllInstall	Installation routine	Often invoked via rundll32
Start, Install, Run	Generic names	Primary payload entry point
Ordinal only (#1, #2)	Export names stripped	Harder to triage — test each ordinal

When exports are ordinal-only

Export names are stripped to hinder static analysis. Strategy:

1. Open in PEStudio → Exports tab → note ordinal numbers
2. Test: rundll32.exe suspect.dll,#1
3. Monitor with Procmon/Process Hacker
4. Repeat for each ordinal until payload activates

Analysing DLLs with rundll32

rundll32.exe is a built-in Windows utility that loads a DLL and calls a specified exported function. Analysts use it to execute DLL payloads without writing a custom loader.

rundll32.exe suspect.dll,ExportedFuncName
rundll32.exe suspect.dll,#1

Loading in x64dbg

1. File → Open → browse to rundll32.exe
2. In "Arguments" field: suspect.dll,ExportedFuncName
3. Set breakpoint on DllMain before running
4. F9 to run — debugger pauses when DllMain executes

When DllMain has no export

Some DLLs execute payload purely in DllMain and export only dummy functions. In this case:

• Call any export — the payload runs during load before the export function executes
• Set a breakpoint at the DLL's base address + AddressOfEntryPoint offset

DLL Side-Loading

DLL side-loading exploits the Windows DLL search order. When an application loads a DLL by name without a full path, Windows searches:

1. Application directory
2. System directory (System32)
3. Windows directory
4. Current directory
5. Directories in PATH

The attacker places a malicious DLL with the same name as a legitimate DLL in the application directory. Windows loads the malicious one first because the application directory is searched before System32.

Legitimate: C:\Windows\System32\version.dll
Malicious:  C:\Program Files\LegitApp\version.dll  ← loaded first

Detection: Procmon filter Operation = LoadImage — the path column shows which DLL was loaded. If a system DLL loads from the application directory, that is side-loading.

MITRE ATT&CK: T1574.002 — Hijack Execution Flow: DLL Side-Loading