Blog

Malware authors often use their own protections or packers to prevent automated unpacking of their executables. This is an attempt to delay development of antivirus (AV) signatures or in-depth behavioral analysis. Scramblers, tools designed to modify the packed binary to help throw off signature-based scanners, are often used as well. In fact, a public protector called [MSLRH] not only packs the executable and protects it using anti-debug, anti-dump, and stolen-byte mechanisms, but also offers the option to choose which protector the executable should appear as in PEiD and other signature-based scanners. [MSLRH] then places the typical byte sequence of that packer at the entry point of the executable, likely throwing off some analysts and automated unpackers. This type of tool is also used by legitimate software authors to protect their intellectual property.

Screenshot of [MSLRH] and its signature selection options:


Because there are so many protectors, packers, and scramblers on the Internet, it can be tricky to determine what you are dealing with. Because the actual malware and its impact on an individual or company is usually of more interest than the packer, I will describe a well-known unpacking trick that will work on a large subset of packers. This is called the ESP trick. The ESP trick is used by many reversers, and appears in the numerous OllyScript unpacking scripts available online. This post is intended to explain how and why the technique works.

About the ESP trick

When you open a packed executable in OllyDbg, you typically see a message like this:


In most cases, a packer/protector includes a PUSH instruction (e.g., PUSHAD) somewhere near the entry point (EP), as shown above. The purpose of PUSHAD is to place all of the registers (EAX, ECX, etc.) onto the stack. Why does this matter? Its purpose is to create a backup of the registers before the unpacking routine is run so that the unpacking stub can utilize all of the registers without fear of changing a value that may be needed later. Once this routine is finished and all sections are unpacked/de-protected, it can then retrieve these original register values off the stack via a POPAD instruction.

As a result, you can receive an unpacked binary on many packers by doing the following:
- Execute the PUSHAD.
- Place a hardware breakpoint on ESP.
- Run.
- Look for a breakpoint that hits. If it doesn't seem to lead to OEP, resume execution and wait for next hit.

Before giving a visual step by step, let me explain why you would place a hardware breakpoint on the ESP register. ESP is the stack pointer. It always points to the top of the stack. By placing a breakpoint on ESP, you ensure that you are notified when these original register values are retrieved again. It is important to place a hardware breakpoint, because a standard software breakpoint places an int3 interrupt instruction at the specified location, thereby modifying the code.

This example uses calc.exe, packed with UPX, for the following reasons:
- UPX is not commercial software.
- UPX is often used to compress both malicious and non-malicious software, because it is a great compression utility.
- The calc.exe file can be found on any Windows machine, so you can use it to follow along.

UPX has a decompress option that can be used on UPX-compressed executables. There are also several scramblers that modify the UPX-packed executable to prevent the use of the decompress option. In cases like this, when an automatic unpacking tool (like the UPX decompress option) does not work, as shown below, the ESP trick is quite useful.



How to use the ESP trick

To begin, execute the PUSHAD by stepping over it. Next, right-click the ESP value and click Follow In Dump:


In the dump window in the lower left, highlight the first dword (ESP's value) and set a hardware breakpoint:


To resume execution, click Run. You should hit the breakpoint shown below (scroll up a bit to see the result):


Note that the POPAD instruction was used just as discussed above. This JMP instruction takes you to the Original Entry Point (OEP), so that you can dump the process or analyze it. Future blog postings will address dumping the debugged process, and with issues found in other packers, such as IAT destruction.

Conclusions:

Malware authors, like many other software authors, make an effort to protect their creations with home-brewed or commercial solutions. When packers, protectors, or scramblers are used to help confuse analysis and antivirus signatures, threat analysis can be delayed. This may lead to users being exposed to a malware threat for some period of time. By better understanding both how these technologies work and how we can combat them, we can better protect ourselves.

Security Researcher: Joren McReynolds