1. Introduction: When Malware Hides Its Real Code

A malware sample may appear relatively small, contain very few recognisable strings, and reveal little about its purpose when examined in a disassembler. Yet, once executed, it may create additional processes, establish network connections, modify files, or load a much larger body of code into memory.

This behaviour can indicate that the sample uses a packer.

Packing is a technique that transforms an executable's contents into a different representation and adds code capable of restoring or loading those contents at runtime. Software developers may use packers for legitimate compression and protection, but malware authors also use them to make analysis more difficult.

Commercial and well-known packers can sometimes be identified through recognisable signatures. Custom-packed malware presents a greater challenge because its packing routine may not match familiar patterns.

For malware analysts, the objective is not merely to identify the packer. It is to understand how the sample transforms itself, what code becomes available during execution, and what malicious actions follow.

Dynamic analysis is particularly useful because it reveals behaviour that may not be visible from the original executable alone.

2. Understanding Custom-Packed Malware

A typical packed executable contains an altered representation of its original code, along with a loader or unpacking routine.

When the executable runs, the loader may reconstruct the original code in memory and transfer execution to it. Depending on the packer, this can involve decompression, decryption, memory allocation, relocation, or other transformations.

Custom packers may use several approaches:

• Encryption: The original code or data is stored in an encrypted form and decrypted during execution.

• Compression: Executable contents are compressed to reduce their stored size.

• Custom loading routines: The executable restores or maps code using non-standard loading logic.

• Obfuscation: Instructions, control flow, or data structures are transformed to make analysis more difficult.

• Multi-stage unpacking: One stage reconstructs or launches another stage, which may contain the actual payload.

These techniques can appear individually or in combination.

Packing alone does not prove that a file is malicious. Some legitimate applications use executable protection and compression. Analysts should assess the sample's provenance, behaviour, execution context, and other available evidence before reaching a conclusion.

3. Static Analysis vs. Dynamic Analysis

Static analysis examines a sample without executing it. Dynamic analysis observes the sample while it runs in a controlled environment.

Both methods are valuable, and combining them usually provides a more complete understanding of packed malware.

Analysis areaStatic analysisDynamic analysisExecution requiredNoYesString visibilityMay be limited by encryption or obfuscationRuntime-generated strings may become visibleCode inspectionExamines the stored executableCan examine code reconstructed in memoryNetwork behaviourMay reveal embedded indicators or functionsCan reveal attempted connections and protocol behaviourFile and registry activityUsually inferred from codeCan be observed during executionMain limitationHidden code may remain difficult to interpretBehaviour may change depending on the environment

Static analysis helps establish the sample's initial structure and identify suspicious characteristics. Dynamic analysis then tests hypotheses about how the program behaves.

For custom-packed malware, this combination is often more effective than relying on either method independently.

4. Preparing a Safe Dynamic Analysis Environment

Before executing a suspicious sample, analysts should prepare an isolated environment designed to limit the risk of malware escaping into production systems.

A typical malware-analysis lab may include:

• A dedicated virtual machine with a disposable snapshot.

• Controlled network access or an isolated virtual network.

• Monitoring tools for process, file, registry, and network activity.

• Memory acquisition and analysis capabilities.

• A method for recording timestamps and correlating events.

• A clean baseline for identifying changes caused by execution.

Useful tools include Process Monitor for Windows activity, Process Explorer for process inspection, Wireshark for network traffic, and x64dbg for debugging. Memory-forensics frameworks such as Volatility can support subsequent examination of acquired memory.

Tool selection depends on the operating system, sample architecture, available telemetry, and research objectives.

Safety note: Do not execute unknown malware on a normal workstation or a production-connected machine. A virtual machine alone is not a complete security boundary. Use appropriate isolation, restrict shared folders and clipboard integration, and control network connectivity.

5. Dynamic Analysis Workflow for Custom-Packed Malware

A disciplined workflow helps analysts distinguish normal unpacking behaviour from subsequent malicious activity.

Stage 1: Establish the baseline

Before execution, record the sample's cryptographic hashes, file size, file type, architecture, and available metadata.

Examine the PE headers, imported functions, section layout, entropy, and printable strings where applicable.

High entropy, unusual section names, limited imports, or a small initial code section may suggest packing. However, none of these characteristics is conclusive on its own.

The baseline provides a reference for later comparison.

Stage 2: Observe initial execution

Execute the sample only within the controlled analysis environment and record its initial behaviour.

Monitor:

• Process creation and parent-child relationships.

• Loaded modules and DLL activity.

• File creation, modification, and deletion.

• Registry operations and configuration changes.

• Memory allocation and protection changes.

• Network connections and DNS requests.

The purpose is to establish what the program does immediately after execution and whether it begins preparing additional code.

Stage 3: Identify unpacking behaviour

Some packed programs allocate memory, transform data, and execute code from a region that differs from their initial executable image.

An analyst may observe memory regions being written to and subsequently marked executable, unexpected changes in thread execution, or code appearing in memory that is not represented by a conventional loaded module.

These observations can provide evidence of runtime unpacking.

However, legitimate software also uses dynamic memory and generates executable code. Analysts should correlate several observations rather than classify a sample as malicious based on one memory characteristic.

Stage 4: Examine the reconstructed code

Once runtime activity indicates that code has been reconstructed, memory inspection can help determine what the program has loaded.

Depending on the sample, investigators may be able to identify:

• Recognisable executable headers or section structures.

• New strings, API references, or configuration values.

• Code regions associated with subsequent execution.

• Embedded URLs, domain names, or file paths.

• Additional executable components that were not visible in the original file.

Not every unpacked payload will resemble a complete executable image. Some samples use shellcode, manually mapped modules, or custom data structures.

Recovered memory content should therefore be interpreted in the context of its structure and execution behaviour.

Stage 5: Correlate unpacking with malicious actions

Recovering code is only part of the investigation. Analysts must determine what the code actually does.

Examine activity following the suspected unpacking stage, including process launches, persistence attempts, credential-access indicators, file encryption, security-tool interference, and outbound communications.

Where appropriate, correlate these findings with operating-system logs, endpoint telemetry, DNS records, firewall events, and packet captures.

The timeline should connect the initial execution, memory transformation, code execution, and any observed impact.

Stage 6: Preserve and document evidence

Save relevant logs, screenshots, memory captures, packet captures, and recovered artefacts according to the lab's evidence-handling procedures.

Calculate hashes for extracted files and document how each artefact was obtained.

Clearly distinguish observed facts from hypotheses. For example, a suspicious memory region may support the conclusion that unpacking occurred, but it does not independently prove that data was exfiltrated.

6. Recognising Common Indicators of Runtime Unpacking

The following indicators can help guide an investigation. None should be treated as a standalone verdict.

IndicatorWhy it mattersWhat to validateHigh-entropy executable sectionsMay indicate compressed or encrypted contentCompare sections and examine other packing indicatorsLimited or unusual importsMay reflect a minimal loader that resolves functions dynamicallyInspect runtime module and API activityExecutable private memoryMay contain dynamically generated or reconstructed codeCorrelate memory writes, protection changes, and executionNew strings appearing in memoryMay reveal configuration or payload contentEstablish when the strings appeared and which region contains themUnexpected thread executionMay indicate control transfer to reconstructed codeExamine thread context and execution historyUnusual child processesMay reveal payload execution or follow-on activityValidate parent process, command line, and process purposeUnexpected outbound connectionsMay indicate command-and-control or payload retrievalInspect destinations, timing, protocol, and content where available

An effective investigation looks for a sequence of related events. For example, memory allocation followed by writes, a protection change, and execution from the affected region is more informative than executable memory alone.

7. Handling Anti-Analysis Behaviour

Some custom-packed malware attempts to detect whether it is running in a virtual machine, debugger, or automated analysis environment.

It may inspect system characteristics, delay execution, check for debugging artefacts, or alter its behaviour when it detects a laboratory environment.

These mechanisms can cause a sample to appear inactive during analysis.

When behaviour is unexpectedly limited, analysts should consider several possibilities: the sample may be benign, its trigger conditions may not have been met, required network resources may be unavailable, or the malware may be deliberately avoiding analysis.

Useful investigative approaches include comparing behaviour across controlled runs, reviewing environmental dependencies, examining available execution traces, and correlating static findings with observed activity.

Changes to the analysis environment should be carefully documented. An observed difference between runs may explain a behaviour change, but it does not automatically establish the sample's intent.

8. Extracting Indicators of Compromise

Dynamic analysis can reveal indicators that are absent from the original packed executable.

Potential findings include:

• File hashes of recovered payloads.

• Domains and IP addresses contacted during execution.

• URLs used for payload retrieval.

• File paths and names created by the sample.

• Registry keys or configuration changes.

• Mutex names and other artefacts, when present.

• Process command lines and parent-child relationships.

These indicators must be validated before being used for detection or blocking.

For example, an observed IP address may belong to shared hosting or a legitimate service. A file path may be common to normal applications. An indicator should be assessed alongside context, reputation, timestamps, and the specific behaviour observed during analysis.

When publishing research, distinguish between indicators observed directly, indicators extracted from a recovered payload, and infrastructure that is only suspected to be related.

Important: Do not invent or assume IOCs. If the sample does not provide a verifiable domain, hash, or other identifier, report that the indicator was not established.

9. Mapping Findings to MITRE ATT&CK

MITRE ATT&CK mappings should reflect the behaviour demonstrated by the sample, not merely the fact that it is packed.

TechniqueIDWhen it may applyObfuscated Files or InformationT1027When evidence supports code or data obfuscation, encryption, or another covered concealment methodDeobfuscate/Decode Files or InformationT1140When the sample decodes or decrypts content to reveal executable functionalityVirtualization/Sandbox EvasionT1497When the sample demonstrates behaviour intended to identify or evade analysis environmentsProcess InjectionT1055When code is injected into another processReflective Code LoadingT1620When code is reflectively loaded into memory

Not every packed sample uses every technique listed above. For example, runtime unpacking within a process does not automatically mean that process injection occurred.

The final mapping should be based on the actual execution flow and supporting evidence.

10. Turning Analysis Findings into Detection

The results of dynamic analysis can improve defensive monitoring in several ways.

Behaviour-based detections

Build detections around suspicious combinations of memory allocation, memory modification, executable protection changes, and subsequent execution.

Where telemetry supports it, correlate the activity with unusual process relationships and unexpected execution locations.

Endpoint monitoring

Use EDR telemetry to investigate unexpected child processes, suspicious module loading, memory-related alerts, and attempts to interfere with security controls.

Ensure that detections account for legitimate applications that use dynamic code generation.

Network detections

Use confirmed network indicators to support investigation and blocking, while avoiding broad rules that may disrupt legitimate shared infrastructure.

Behavioural patterns, such as unexpected connections immediately following suspicious execution, may provide useful context when specific IOCs are unavailable.

Repeatable analysis procedures

Document the environment, tool versions, sample hashes, execution timeline, and collection methods. Repeatable procedures make findings easier to validate and help other analysts distinguish genuine behaviour from environmental effects.

11. Common Mistakes During Dynamic Analysis

Assuming high entropy proves malware. Compression, encryption, and legitimate software protection can all produce high-entropy data.

Executing samples without adequate isolation. A poorly isolated lab can expose systems and networks to risk.

Trusting a single sandbox run. Environment-dependent behaviour and delayed execution may prevent a sample from revealing all its functionality.

Treating unpacking as proof of malicious intent. Legitimate applications may unpack or generate code at runtime.

Ignoring the code after it is reconstructed. The payload's actual behaviour is essential to determining intent and impact.

Publishing unverified indicators. Incorrect IOCs can lead to false positives and unnecessary blocking.

Failing to preserve volatile evidence. Memory-resident code may disappear when a process terminates or the system restarts.

Conclusion

Custom-packed malware challenges conventional analysis because the executable initially available to investigators may conceal much of its functionality. Dynamic analysis helps bridge this gap by revealing how the program behaves during execution and what code or data becomes available in memory.

By combining static inspection, controlled execution, memory analysis, process monitoring, and network investigation, researchers can build a more complete picture of a packed sample. The resulting evidence can support malware classification, incident response, threat hunting, and behavioural detection engineering.

The most important principle is to investigate the full execution chain rather than focus only on the original file. Packing may conceal a payload, but the process of unpacking and executing that payload can create observable behaviour.

For malware analysts, the goal is not simply to unpack the executable. It is to understand what the recovered code does, establish the evidence, and turn those findings into actionable defence.