Automated malware unpacking
Αυτοματοποιημένο ξεπακετάρισμα κακόβουλου λογισμικού

Bachelor Dissertation
Author
Olymbios, Andreas
Ολύμπιος, Αντρέας
Date
2026-07View/ Open
Keywords
Dynamic analysis ; Malware unpacking ; Packers ; Write-then-execute ; Memory dumpAbstract
This thesis presents a tool that detects and captures the unpacked code of packed Windows executables while they run. Packing hides a program's real code by keeping it compressed or encrypted, then restoring it in memory only when the program runs. The tool does not rebuild a working executable from the packed file. Instead, it watches the target process as it runs, finds the point where memory that was written earlier gets executed, and writes that memory to disk so an analyst can inspect it.
The detection method is write-then-execute, the standard heuristic in the generic unpacking literature. The tool runs a 32-bit target under the Windows debug API and enforces a write-xor-execute policy on memory pages. When a page is written and later executed, that marks an unpacking event. The tool captures the process image at that moment, clears its record for that page, and keeps going. This way packers that decode in several stages are followed through each stage instead of being stopped at the first. Code that is unpacked into memory allocated at runtime is covered by hooks on VirtualAlloc and VirtualProtect.
The tool was evaluated on a public dataset of benign binaries packed with twenty-four commodity packers, five samples each, and cross-checked against the original unpacked files. On the packers that unpack inside the process image, it recovered the original code in most or all sampled files, including MEW, MPRESS, FSG, NSPack and WinUpack, with partial recovery on others such as UPX, RLPack and ASPack. This was confirmed by matching the dumps against the original files, and the recovered dumps entropy dropped from roughly 7.9 to as low as 3.8. The evaluation also mapped where the method stops: virtualization protectors and bundlers such as Themida and Molebox, and packers that unpack through paths the current hooks do not observe. These are reported in full rather than left out.


