Automating pattern-based de-obfuscation of x64 code using IDA plugins

Question

Trying to deobfuscate x64 code. You can skim/skip most of this, since it mainly exists to prove I have tried and exhausted all avenues.

Burning Purpose

To massively automate the deobfuscation within a [memory dump] of a 64-bit Windows game.

Methods currently in use

PCRE driven byte replacement

Works fine with simple obsfucation like Variant 1 & 2

Original (Variant 1):

48 8D 64 24 F8        - lea rsp,[rsp-08]         ; Stack -= 8
48 89 2C 24           - mov [rsp],rbp            ; Push RBP
48 8D 2D 156A5A00     - lea rbp,[7FF749022784]   ; Put JMP target in RSP
48 87 2C 24           - xchg [rsp],rbp           ; Pop RBP (RBP restored)
48 8D 64 24 08        - lea rsp,[rsp+08]         ; Stack += 8 (Balanced)
FF 64 24 F8           - jmp qword ptr [rsp-08]   ; JMP (target)

Original (Variant 2):

48 89 6c 24 f8         - mov [rsp-0x8], rbp
48 8d 64 24 f8         - lea rsp, [rsp-0x8]
(rest as per Variant 1)

Deobsfucated:

90 90 90 .. ..        - (Variant 1: NOP * 9, Variant 2: NOP * 10)
90 90                   NOP * 2   ; Pad instruction to preserve
                                  ; RIP of next instruction
E9 ?? ?? ?? ??        - JMP NEAR 
90 90 90 .. ..        - NOP * 13

De-obfuscation script:

#!/usr/bin/env sh
#            |------------- 11 bytes--------| |-- 5 bytes--| |---------------- - 13 bytes---------|     
# Signature: 48 8D 64 24 F8 48 89 2C 24 48 8D 2D ?? ?? ?? ?? 48 87 2C 24 48 8D 64 24 08 FF 64 24 F8 (29 bytes)
# Translate: 90 90 90 90 90 90 90 90 90 90 90 E9 ?? ?? ?? ?? 90 90 90 90 90 90 90 90 90 90 90 90 90
#    
#            |------------- 12 bytes ----------| |-- 5 bytes--| |---------------- - 13 bytes---------|
# Signature: 48 89 6c 24 f8 48 8d 64 24 f8 48 8d 2d ?? ?? ?? ?? 48 87 2c 24 48 8d 64 24 08 ff 64 24 f8 (30 bytes)
# Translate: 90 90 90 90 90 90 90 90 90 90 90 90 e9 ?? ?? ?? ?? 90 90 90 90 90 90 90 90 90 90 90 90 90

xxd -ps Game_Dumped.exe | 
    sed -e 's/\(..\)/\1 /g' | 
    tr '\n' ' '             |
    perl -p -e "s/48 8d 64 24 f8 48 \
89 2c 24 48 8d 2d (.. .. .. ..) \
48 87 2c 24 48 8d 64 24 08 ff 64 24 f8/90 90 90 90 90 \
90 90 90 90 90 90 e9 \1 90 90 90 90 90 90 90 90 90 90 \
90 90 90/g ; s/48 89 6c 24 f8 48 8d 64 24 f8 48 8d 2d\
 (.. .. .. ..) 48 87 2c 24 48 8d 64 24 08 ff 64 24 \
f8/90 90 90 90 90 90 90 90 90 90 90 90 e9 \1 90 90 90 \
90 90 90 90 90 90 90 90 90 90/g" |
    xxd -r -ps > Game_Dumped_NOP2.exe

The 3rd variety

143fe7a26 48 89 6c 24 f8         mov     [rsp-8], rbp
143fe7a2b 48 8d 64 24 f8         lea     rsp, [rsp-8]
143fe7a30 e9 60 71 8d ff         jmp     loc_1438beb95

1438beb95 48 8d 2d 10 b0 3a fd   lea     rbp, sub_jump_target
1438beb9c 48 87 2c 24            xchg    rbp, [rsp]
1438beba0 48 8d 64 24 08         lea     rsp, [rsp+8]
1438beba5 ff 64 24 f8            jmp     qword ptr [rsp-8]

This is the same as the Variant 2, except it has been broken into two sections. Planned solution will involve distorm3's flow control flags, and rewriting jmp to 0x143fe7a26.

Variations on stack manipulation

48 89 E0             mov     rax, rsp
48 05 F8 FF FF FF    add     rax, 0FFFFFFFFFFFFFFF8h ; Add
48 89 C4             mov     rsp, rax
48 89 1C 24          mov     [rsp], rbx

What a horrifically long way to effect to decrement RSP by 8. But by now, I am fairly used to it, and it wouldn't bother me, but I have just enabled the "Stack Pointer" in IDA [6.8]'s General Options, and realising that IDA isn't including lea rsp, [rsp+-8] in it's calculations of the stack pointer, which is stopping it properly analysing the code.

RSP Bytes               Disassembly
--- ------------------- -------------------------------
000 48 89 E0            mov     rax, rsp
000 48 05 F8 FF FF FF   add     rax, 0FFFFFFFFFFFFFFF8h
000 48 89 C4            mov     rsp, rax ; It tracked this
-20 48 89 1C 24         mov     [rsp], rbx
-20 48 83 EC 20         sub     rsp, 20h ; and this
000 48 8B 41 10         mov     rax, [rcx+10h]
000 48 89 4C 24 F8      mov     [rsp-8], rcx
000 48 8D 64 24 F8      lea     rsp, [rsp-8] ; but not this
000 48 8B 1C 24         mov     rbx, [rsp]

I am also starting to suspect that there are going to be many permutations of all these techniques, and I need to start addressing the problem in IDA.

The problem is, the only sample source I can find uses IDAPython idaapi low-level functions, so the code is ridiculously long, and as I am replacing a 5 byte instruction with a 4 byte one, I cannot find a way to alter the operand I have inadvertently created. (Fortunately, in this case, it's just CLC).

Update: I have fixed this issue, and the solution has drastically reduced the size of my script. The pertinent fix is below:

def replace_pattern(ea):
    search = [0x48, 0x8d, 0x64, 0x24, 0xf8]
    replace = [0x48, 0x83, 0xec, 0x08, 0x90]
    current = []
    for i in xrange(5):
        current.append(idaapi.get_byte(ea+i))
    if 0 == cmp(search, current):
        for i in xrange(5):
            # fixed: replace put_byte with patch_byte
            idaapi.patch_byte(ea+i, replace[i])
        return 1
    return 0

[original code] Incidentally, the example code was written by our own Rolf Rolles

import idaapi
import idc

# Planned task: replace
#     48 8d 64 24 f8          lea    rsp,[rsp-0x8]
# with
#     48 83 ec 08             sub    rsp,0x8
#     90                      nop
#
# Actual result:
# Replaced:  48 8d 64 24 f8   lea    rsp,[rsp-0x8]
# with:   :  48 83 ec 08      sub    rsp,0x8
#            f8               clc
#
# Verdict, close enough, but way too much code involved.

def match_pattern(ea):
    search = [0x48, 0x8d, 0x64, 0x24, 0xf8]
    replace = [0x48, 0x83, 0xec, 0x08, 0x90]
    current = []
    for i in xrange(5):
        current.append(idaapi.get_byte(ea+i))
    if 0 == cmp(search, current):
        return 1
    return 0

    # Note: I thought I might be able to simply rewrite
    #       at a byte level, but it threw an exception.
    #
    #    for i in xrange(4):
    #        idaapi.put_byte(ea+i, replace[i])

class deobfu_hook(idaapi.IDP_Hooks):
    def __init__(self):
        idaapi.IDP_Hooks.__init__(self)
        self.n = idaapi.netnode("$ X86 Deobfuscator Modifications",0,1)

    def custom_ana(self):
        # Check first two bytes "by hand" for speed
        b = idaapi.get_byte(idaapi.cmd.ea)
        if b == 0x48: # First byte
            b = idaapi.get_byte(idaapi.cmd.ea+1)
            if b == 0x8d: # Second byte
                # Discard speed, do a full match
                if match_pattern(idaapi.cmd.ea, 0, 0):
                    # If matched, supply all required values for 
                    # SUB RSP,8 - Surely there is an easier way!
                    idaapi.cmd.itype = 0xd1
                    idaapi.cmd.size = 4
                    idaapi.cmd.auxpref = 0x1810
                    idaapi.cmd.segpref = 0
                    idaapi.cmd.insnpref = 0x48
                    idaapi.cmd.flags = 2

                    idaapi.cmd.Op1.type = 1
                    idaapi.cmd.Op1.offb = 0
                    idaapi.cmd.Op1.offo = 0
                    idaapi.cmd.Op1.flags = 8
                    idaapi.cmd.Op1.dtyp = 7
                    idaapi.cmd.Op1.reg = 4
                    idaapi.cmd.Op1.phrase = 4
                    idaapi.cmd.Op1.value = 0
                    idaapi.cmd.Op1.addr = 0
                    idaapi.cmd.Op1.specval = 0
                    idaapi.cmd.Op1.specflag1 = 0
                    idaapi.cmd.Op1.specflag2 = 0
                    idaapi.cmd.Op1.specflag3 = 0
                    idaapi.cmd.Op1.specflag4 = 0

                    idaapi.cmd.Op2.type = 5
                    idaapi.cmd.Op2.offb = 3
                    idaapi.cmd.Op2.offo = 0
                    idaapi.cmd.Op2.flags = 8
                    idaapi.cmd.Op2.dtyp = 7
                    idaapi.cmd.Op2.reg = 0
                    idaapi.cmd.Op2.phrase = 0
                    idaapi.cmd.Op2.value = 8
                    idaapi.cmd.Op2.addr = 0
                    idaapi.cmd.Op2.specval = 0
                    idaapi.cmd.Op2.specflag1 = 0
                    idaapi.cmd.Op2.specflag2 = 0
                    idaapi.cmd.Op2.specflag3 = 0
                    idaapi.cmd.Op2.specflag4 = 0

                    return True
        return False

class deobfu_t(idaapi.plugin_t):
    flags = idaapi.PLUGIN_PROC | idaapi.PLUGIN_HIDE
    comment = "Deobfuscator"
    wanted_hotkey = ""
    help = "Runs transparently"
    wanted_name = "deobx86"
    hook = None

    def init(self):
        self.hook = None

        self.hook = deobfu_hook()
        self.hook.hook()
        print("deobfu init")
        return idaapi.PLUGIN_KEEP

    def run(self, arg):
        pass

    def term(self):
        print("deobfu term")
        if self.hook:
            self.hook.unhook()

def PLUGIN_ENTRY():
    print("PLUGIN_ENTRY:deobfu")
    return deobfu_t()

Where do you want to go today?

I want a better solution, and I am not afraid to code it. I am how-ever in need of some starting advice, plus I need to make sure I am not re-coding the wheel here.

I have since written some other IDAPython code using the higher level idautils to create call trees, and collate xrefs and such. But I don't know how to rewrite the actual disassembled code at that level. There is one example in the IDAPython repo: https://github.com/pfalcon/idapython/blob/master/examples/ex_idphook_asm.py but that's

1. Full of silly bugs (I fixed them)
2. Hooks the Assembly command, not the disassembly process

I have reviewed the answer to a similar question at Creating IDA Pro debugger plugins - API documentation and examples?. I have looked at various examples of quite nice IDAPython code that can:

• Add comments while disassembling
• Change the line color while disassembling

But, I have seen nothing about actually changing instructions.

I have not purchased The IDA Pro Book because I do not live in the U.S. and I do not want to wait n weeks for on-demand printing and delivery. I am not adverse to writing an .idc, as I'm quite familiar with C (more-so that with Python), although I suspect despite the shallower learning curve (and assumedly ready examples) it would be harder (long term) than using higher level IDAPython code. (I'm learning Python as I go, but .. well, didn't we all?)

Because the code I am working with is exclusively 64-bit, there is little (basically no) pre-existing deobfu samples or code out there.

And so here I find myself, asking for your patient guidance. (Very patient if you actually read all of that).

PS: I took the time to document everything I had done, because I know how little respect we all have for people who don't even attempt something before hollering for help.

PPS: OMG 2nd most avid member of RE is Igor Skochinsky, I can only bow in humility.

Answer 1

There are two scenarios to be distinguished, which you (implicitly) mention as well:

I. Obfuscated pattern with no jmps in between interrupting the control flow,

II. Obfuscated pattern with interrupting jmps.

Definitely there are several possibilities to remove the obfuscation. I will try to describe one of them which I (partly) applied.

Scenario I is straightforward (and with that one I made some successful tries and wrote a program for it):

1. Identify the pattern "by hand", write down the bytecode sequence, and write down the - simpler - bytecode replacement.

2. Read the binary into the own programmed "remove obfuscation" application.

3. Let the program search for all occurences of the obfuscation bytecode.

4. Let the program replace all found occurrences with the (hopefully simpler) replacement. You might gain a lot of space for possible patches.

5. Very important: Test the modified (i.e. simplified) program

Scenario II is the more difficult part, as the breaks can show up anywhere and no unique bytecode sequence is possible any more for an obfuscation pattern. I would probably proceed in the following way, however not having tried it yet:

1. Independent from obfuscation, find a solution to remove the "breaks" in the code, i.e. try to remove the usually many, many useless jmp statements. In case you can solve this, it has the big additional advantage that the code is not broken any more in Ida by the jmps. Maybe there exist already some automated solutions, although presently not known to me.

2. Apply Scenario I

As to my experience, the Scenario I is the lesser frequent case, although having been encountered as well. The following - real - obfuscated x64 code sequence demonstrates the importance of removing the "code breaks". In this sequence there are no less than six breaks where the program each time jmps between addresses separated far distant from each other.

    loc_startObfuscatedSequence:                        
                    lea     rsp, [rsp-8]
                    mov     [rsp], rbp
                    mov     rbp, offset loc_target1
                    xchg    rbp, [rsp]    
                    mov     [rsp-8], rcx
                    jmp     loc_break1  

    loc_break1:     lea     rsp, [rsp-8]
                    jmp     loc_break2

    loc_break2:     mov     [rsp-8], rdx
                    jmp     loc_break3

    loc_break3:     lea     rsp, [rsp-8]
                    jmp     loc_ break4

    loc_ break4:    mov     rcx, [rsp+10h]
                    mov     rdx, offset loc_target2
                    cmovz   rcx, rdx
                    jmp     loc_break5

    loc_break5:     mov     [rsp+10h], rcx
                    lea     rsp, [rsp+8]
                    mov     rdx, [rsp-8]
                    jmp     loc_break6

    loc_break6:    lea     rsp, [rsp+8]
                    mov     rcx, [rsp-8]
                    retn

If I interpreted this correctly, the obfuscated sequence is a replacement for the simple

jz  loc_target2
jmp loc_target1

More complex patterns like the one of the example might contain simpler ones. As a consequence, you would start with the simple patterns and then move from simpler to more complicated ones, increasing the readablility of the code (and making Ida more happy) with each de-obfuscation step.

Answer 2

import struct

def readWord(array, offset):
    return struct.unpack_from("<i", bytearray(array), offset)[0]

def writeWord(array, offset, word):
    array[offset:offset+4] = bytearray(struct.pack("<i", word))

class Obfu(object):
    # Copyright 2016 Orwellophile LLC. MIT License.
    def __init__(self, start, end):
        self.start = start
        self.end = end
        self.eip = start
        self.patterns = []

    def append(self, searchText, replaceText, search, replace, replaceFunction = None):
        self.patterns.append([search, replace, replaceFunction])

    def replace_pattern(self, search, replace, replaceFunction, ea):
        searchSize = len(search)
        replaceSize = len(replace)
        # buf = idc.GetManyBytes(ea, ItemSize(ea))
        failed = 0
        for i in range(searchSize):
            if search[i] != -1 and idaapi.get_byte(ea+i) != search[i]:
                failed = 1
                break
        if not failed:
            if replaceFunction != None:
                original = []
                for i in range(max(searchSize,replaceSize)):
                    original.append(idaapi.get_byte(ea+i))
                replace = replaceFunction(search, replace, original, ea)
                replaceSize = len(replace)
            for i in range(replaceSize):
                if replace[i] != -1:
                    idaapi.patch_byte(ea+i, replace[i])
            print "patched %i bytes at 0x%x" % (replaceSize, ea)
            MakeCode(ea) # else it turns it into data
        return not failed

    def _patch(self, ea):
        for pattern in self.patterns:
            if self.replace_pattern(pattern[0], pattern[1], pattern[2], ea):
                return 1
        return 0

    def get_next_instruction(self):
        while self.eip <= self.end:
            yield [hex(self.eip), self._patch(self.eip)]
            self.eip += ItemSize(self.eip)

# Usage

obfu = Obfu(SegStart(ScreenEA()), SegEnd(ScreenEA()))

# Patch factory
def generate_patch1(jmpTargetOffset, oldRip, newRip):
    def patch(search, replace, original, ea):
        result = [0xcc]*len(original) # preallocate result with 0xcccccc...

        jmpTarget = readWord(original, jmpTargetOffset)
        adjustTarget = oldRip - newRip
        jmpTarget = jmpTarget + adjustTarget
        result[0] = 0xE9 # JMP rel32off
        writeWord(result, 1, jmpTarget)
        return result
    return patch

# Uses Patch Factory generated callback
obfu.append("""
        0:  48 8d 64 24 f8          lea    rsp,[rsp-0x8]
        5:  48 89 2c 24             mov    QWORD PTR [rsp],rbp
        9:  48 8d 2d 00 00 00 00    lea    rbp,[rip+0x0]        # 0x10
        10: 48 87 2c 24             xchg   QWORD PTR [rsp],rbp
        14: 48 8d 64 24 08          lea    rsp,[rsp+0x8]
        19: ff 64 24 f8             jmp    QWORD PTR [rsp-0x8]
        """, "test of replacement function (patch callback)",
        [0x48, 0x8D, 0x64, 0x24, 0xF8, 0x48, 0x89, 0x2C, 0x24, 0x48, 0x8D, 0x2D, -1, -1, -1, -1, 0x48, 0x87, 0x2C, 0x24, 0x48, 0x8D, 0x64, 0x24, 0x08, 0xFF, 0x64, 0x24, 0xF8], 
        # unused:
        [0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0xE9, -1, -1, -1, -1, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90],
        generate_patch1(0x0c, 0x10, 0x05)
        )

# Byte patch
obfu.append("""
        0:  48 89 6c 24 f8          mov    QWORD PTR [rsp-0x8],rbp
        5:  48 8d 64 24 f8          lea    rsp,[rsp-0x8]
        a:  48 8d 2d 00 00 00 00    lea    rbp,[rip+0x0]        # 0x11
        11: 48 87 2c 24             xchg   QWORD PTR [rsp],rbp
        15: 48 8d 64 24 08          lea    rsp,[rsp+0x8]
        1a: ff 64 24 f8             jmp    QWORD PTR [rsp-0x8]
        """, 
        "jmp ${ readWord(0x0d) + (oldRip = 0x11) - (newRip = 0x05) }",
        [0x48, 0x89, 0x6c, 0x24, 0xf8, 0x48, 0x8d, 0x64, 0x24, 0xf8, 0x48, 0x8d, 0x2d, -1, -1, -1, -1, 0x48, 0x87, 0x2c, 0x24, 0x48, 0x8d, 0x64, 0x24, 0x08, 0xff, 0x64, 0x24, 0xf8],
        [0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0xe9, -1, -1, -1, -1, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90, 0x90]
        )

# This patch disables one of the longer patches, keep at end of list
obfu.append(  "lea    rsp,[rsp-0x8]",   # First two arguments aren't used
        "", # "sub    rsp,0x8; nop",    # They're just there to remind you
        [0x48, 0x8d, 0x64, 0x24, 0xf8], 
        [0x48, 0x83, 0xec, 0x08, 0x90])

# Test single replacement with: 
# obfu._patch(address)
#
# or uncomment next lines to patch everything:
#
# x = obfu.get_next_instruction()
# count = 0
# while x.next(): count = count + 1