How to identify XOR encoding, standard cryptographic algorithms (AES, RC4, MD5), and custom Base64 alphabets in malware binaries using constant detection, signsrch, and FindCrypt2.
Sign in to mark this article as read and track your progress.
Malware protects its configuration (C2 addresses, encryption keys, campaign identifiers) and secondary payloads by encrypting them and decrypting only at runtime. The analyst needs to:
Think of cryptographic constants as a chef''s secret recipe written on a napkin left on the counter. The algorithm tries to hide what it''s cooking, but the ingredients list gives it away.
XOR is the most common obfuscation method in malware. It is symmetric: applying the same key twice recovers the original data.
Plaintext XOR Key = Ciphertext
Ciphertext XOR Key = Plaintext
XOR is like a light switch. Flip it once, the light turns on. Flip it again with the same switch, it turns off. The key is knowing which switch to flip — and malware authors often leave that switch in plain sight in the code.
The key is a single hardcoded byte visible directly in disassembly:
xor ecx, ecx ; counter = 0
decrypt_loop:
mov al, [esi+ecx] ; load encrypted byte
xor al, 0x55 ; XOR with key 0x55 ← KEY IS HERE
mov [esi+ecx], al ; store decrypted byte
inc ecx
cmp ecx, edx
jl decrypt_loop
Static decryption in Python:
key = 0x55
encrypted = bytes([0xA3, 0x7F, 0x12, 0xE8, ...]) # from binary
decrypted = bytes([b ^ key for b in encrypted])
print(decrypted)
The key is a multi-byte sequence applied cyclically. More resistant to frequency analysis than single-byte XOR:
lea esi, [encrypted_buf]
lea edi, [key_buf] ; 4-byte key: DE AD BE EF
decrypt:
mov al, [esi+ecx]
mov ebx, ecx
and ebx, 3 ; ebx = ecx mod 4 (cycles key)
xor al, [edi+ebx] ; XOR with cycling key byte
mov [esi+ecx], al
inc ecx
cmp ecx, edx
jl decrypt
Python decryption:
key = bytes([0xDE, 0xAD, 0xBE, 0xEF])
decrypted = bytes([b ^ key[i % len(key)] for i, b in enumerate(encrypted)])
If the key is not visible in disassembly (e.g., derived at runtime), brute-force all 256 possibilities. A valid key produces readable ASCII or a recognisable binary header:
for key in range(256):
result = bytes([b ^ key for b in encrypted])
if b'http' in result or b'MZ' in result:
print(f"Key: {key:#04x} → {result[:50]}")
When malware uses real cryptographic algorithms (AES, RC4, MD5), reverse-engineering the implementation is impractical. Instead, search for characteristic constants — every algorithm has a fingerprint.
| Algorithm | Fingerprint | Identifying Constants |
|---|---|---|
| MD5 | Initialisation vector | 0x67452301, 0xEFCDAB89, 0x98BADCFE, 0x10325476 |
| SHA-1 | Initialisation vector | MD5 constants + 0xC3D2E1F0 |
| AES | S-Box | 256-byte substitution table starting 0x63, 0x7C, 0x77, 0x7B... |
| RC4 | KSA loop | 256-iteration loop filling S[i] = i (no fixed constants — recognise by loop pattern) |
| Base64 | Alphabet string | Standard 64-char alphabet ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/ |
| CRC32 | Polynomial | 0xEDB88320 or lookup table of 256 DWORD values |
| Blowfish | P-array | Starts with 0x243F6A88, 0x85A308D3... (fractional digits of π) |
Scans loaded binary for cryptographic constants and annotates matching addresses automatically:
FindCrypt2 results:
0x00403050: AES_Te0 (AES encryption S-Box)
0x00404050: MD5_IV (MD5 initialisation vector)
0x00405000: CRC32_table (CRC32 lookup table)
Navigate to the annotated address to find the function that uses the algorithm.
signsrch suspicious.exe
Example output:
offset signature
0x004050 AES S-Box
0x004150 AES inverse S-Box
0x004250 AES Rcon
DIE also detects cryptographic primitives in its analysis output alongside packer identification.
Malware sometimes uses a non-standard Base64 alphabet to defeat standard decoders:
Standard: ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/
Custom: ZYXWVUTSRQPONMLKJIHGFEDCBAzyxwvutsrqponmlkjihgfedcba9876543210-_
Standard Base64 decoders fail on custom alphabets. The approach:
.data or .rdata)CyberChef supports custom alphabets: use "From Base64" operation → click alphabet field → paste the custom alphabet found in the binary.
1. Run FindCrypt2 or signsrch on the binary
→ identifies algorithm immediately if standard constants are present
2. If no constants found, search for XOR loops:
→ look for tight loops with XOR [reg+counter], imm8
→ extract key byte from the immediate value
3. If multi-byte key: extract key array from .data section
4. Decrypt statically in Python using extracted key + ciphertext
5. Verify decryption: decrypted content should contain readable strings,
a PE header (MZ), or recognisable network protocol
6. If algorithm is AES or RC4:
→ find the key schedule or KSA function
→ set breakpoint after key initialisation in x64dbg
→ read the key from memory at runtime