Here’s what the problem statement is
vm1.exe implements a simple 8-bit virtual machine (VM) to try and stop reverse engineers from retrieving the flag. The VM’s RAM contains the encrypted flag and some bytecode to decrypt it. Can you figure out how the VM works and write your own to decrypt the flag? A copy of the VM’s RAM has been provided in ram.bin (this data is identical to the ram content of the malware’s VM before execution and contains both the custom assembly code and encrypted flag).
You can download vm1.exe from MalwareTech’s site. The password to the zip file is MalwareTech.
Okay if you are completely new to custom VM reversing I suggest you go and see how custom VM’s are implemented. I wrote a simple custom VM in C to help you understand here.
Now let’s load our challenge in IDA and start our analysis.
Here we can see 1FBh bytes of memory being allocated in the heap and the handle to the allocated heap is moved to Dst. After that there is a call to memcpy and the parameters passed as source is an offset to unk_404040 which is the ram of the Virtual Machine, Destination is Dst and the size passed is again 1FBh (507). Next there is a call to sub_4022E0 which is probably used to implement the VM and decrypt the flag. Then the flag is sent to MD5::digestString to create a MD5 hash and messagebox is being called with a Caption “We’ve been compromised!”.
Now let’s take a look at subroutine sub_4022E0 to figure out how the VM works.
Here we see 0 being moved to var_1 and code moves to loc_4022EA and moves 1 to EAX and there is a test eax, eax and a jz (jumo zero) insruction. Since eax is compared with eax itself the jump will not be taken. Moving forward,
var_1 is moved to ecx and Dst (base of the memory) is moved to edx. Then [edx+ecx+0FFh] i.e [Dst+0+255] 255th byte is moved to eax which is further moved to var_10. Now var_1 is incremented by 1.
And [eax+edx+0FFh] i.e [Dst+1+255] 256th byte is moved to ecx which is further moved to var_C and var_1 is incremented by 1 making it 2 (var_1 = 2).
Again [ecx+eax+0FFh] i.e [Dst+2+255] 257th byte is moved to edx which is further moved to var_8 and var_1 is again incremented by 1 making it 3 (var_1 = 3).
Now the three values read (var_10, var_C and var8) are passed as parameters to sub routine sub_402270. This is repeated for all the bytes in the ram of VM(unk_404040). We can make a guess that one of these variable is opcode and other two are operands to it. Let’s figure out what the VM does with these 3 variables by taking a look at the subroutine sub_402270.
Okay here the passed parameters are accepted as var_4 = var10 , arg_4 = var_C and arg_8 = var_8. We see value of var_4 being compared to 1, 2, and 3 to execute different statements. The graph represents a switch statement. Revise here. So this means var_4 is the opcode and arg_4 & arg_8 are the two operands to the opcode. So we now know,
var_4 –> Opcode
arg_4 –> Operand 1
arg_8 –> Operand 2
So there are 3 possible opcodes 1, 2 and 3. Lets analyse what happens when the opcode is 1,
It is moving operand 2 (arg_8) to position pointed by operand 1 (arg_4). i.e
ram[operand 1] = operand 2;
When the opcode is 2,
Opcode 2 only deals with the first operand. It is moving the value pointed by operand 1 to byte_404240. So the VM is reading a byte and storing it for further usage. It’s like moving a byte to a register so we can assume loc_4022D5 as a register to the VM. This can be represented as :
vm_register = ram[operand 1];
When the opcode is 3,
Here the previously stored value (byte_404240) is moved to edx and similar to opcode 2 the value pointed by operand 1 (var_4) is moved to ecx. These two values are XOR’ed and stored at same address that is pointed by operand 1. This can be represented as
ram[operand 1] = ram [operand 1] XOR vm_register;
If the opcode is anything else than 3 then the execution in the VM will stop and return.
Now that we know exactly what the VM does let’s write a program to recreate the VM to decode the FLAG 😉
Executing the program we get,