Debugging Concepts

What is Debugging in Malware Analysis?

Debugging is controlled execution. The analyst decides when to start, when to stop, what to inspect, and — if needed — what to change. It is the primary technique for reaching code that static analysis cannot see.

Stepping

Four stepping modes control how the debugger moves through code:

Operation	Key (x64dbg)	What Happens
Step Into	F7	Execute one instruction. Follow into CALLs. Use when you want to examine a function's internals.
Step Over	F8	Execute one instruction. Run called functions to completion without entering them. Use when the function is uninteresting and you only care about the return value in EAX.
Run to Cursor	F4	Resume until EIP reaches the selected instruction. Faster than stepping through many instructions manually.
Run	F9	Resume until a breakpoint fires or the program exits.

Rule of thumb: Step Over by default. Step Into only when you need to examine what a specific function does internally.

Breakpoints

Software Breakpoints (INT 3)

Overwrites the first byte of the target address with 0xCC (the INT 3 opcode). Default type — set with F2 in x64dbg.

Limitation: malware can scan its own code for 0xCC bytes and detect the debugger. When analysing anti-debug-aware samples, prefer hardware breakpoints.

Hardware Breakpoints

Uses CPU debug registers DR0–DR3. Up to 4 addresses simultaneously. Triggers on execution, read, or write access. Invisible to the program — no code bytes are modified.

Set in x64dbg: right-click any address → Breakpoint → Hardware → Execute/Read/Write.

When to use: always prefer hardware breakpoints when analysing self-defending malware. They defeat the 0xCC scanning anti-debug check.

Conditional Breakpoints

Pauses only if a user-defined expression evaluates to true. Example: break at address 0x401234 only when EAX == 0.

When to use: isolating specific loop iterations, catching a specific file path in a CreateFileA call, or triggering only on the 500th call to a function.

Set in x64dbg: right-click breakpoint → Edit → add condition.

Practical example — catching registry persistence

1. Ctrl+G → type "RegCreateKeyExA" → press Enter
2. F2 to set a software breakpoint on the first instruction
3. F9 to run the sample
4. When breakpoint fires, examine the stack:
   - [ESP+8] = hKey (HKEY_CURRENT_USER or HKEY_LOCAL_MACHINE)
   - [ESP+C] = lpSubKey (registry path string pointer)
5. Follow lpSubKey in the dump to read the exact registry path

Total time to identify persistence mechanism: under one minute.

Exceptions

When the CPU encounters an error (division by zero, invalid memory access, INT 3), it raises an exception. The debugger gets first chance to handle it before the program's own handler runs.

Malware abuses this: it triggers exceptions deliberately and installs Structured Exception Handlers (SEH) that redirect execution to hidden code paths the disassembler cannot see.

x64dbg configuration for malware analysis: Go to Options → Preferences → Exceptions. Add INT 3 and single-step exceptions to the "pass to program" list. This prevents SEH-based anti-debugging tricks from disrupting analysis.

Modifying Execution

The debugger's ability to modify program state is the analyst's most powerful bypass tool:

Technique	When to Use
Change a register value	Set EAX to 0 after IsDebuggerPresent returns 1
NOP out a jump	Replace JNZ with 0x90 0x90 to force a specific code path
Modify memory	Write fake C2 response bytes into a buffer to observe how the malware processes them
Set EIP manually	Right-click → "Set New Origin Here" to skip sleep calls or time-consuming loops

NOP patching example

; Original — exits if debugger detected
CALL  IsDebuggerPresent
TEST  EAX, EAX
JNZ   exit_malware     ; <-- NOP this jump (replace bytes with 0x90 0x90)

; After NOP patch — always falls through to payload
CALL  IsDebuggerPresent
TEST  EAX, EAX
NOP
NOP
; payload code runs here regardless of debugger

In x64dbg: select the JNZ instruction → right-click → Binary → Fill with NOPs.

Caution: Patching instructions changes program behaviour and can cause crashes or misleading results if dependencies break. Always work on a VM snapshot so you can revert.

The Analysis Loop

Most malware analysis sessions follow this repeating loop:

1. Set breakpoints on suspicious API calls (CreateFile, RegSetValue, URLDownloadToFile)
2. Run (F9)
3. When breakpoint fires, read arguments from the stack or registers
4. Note the IOC (file path, registry key, URL)
5. Modify execution if needed (patch anti-debug check, skip sleep)
6. Continue (F9) to the next breakpoint
7. Repeat until the full behaviour is mapped

This loop — run, observe, note, continue — is the core skill of dynamic code analysis. Everything else is tooling around this pattern.