POS: A Recognition Method for Packed Software in Opened-Set Scenario

Malware plays a critical role in network attacks, making its analysis essential for ensuring network security. To evade detection, malware developers often use packing techniques to hide malicious code, making it difficult for analysts to identify the software’s true behavior. Software that has been packed is referred to as “packed software,” and network security analysts need to employ unpacking strategies to remove these protective layers and restore the software’s actual behavior. This process is crucial in preventing malware from bypassing traditional security mechanisms, as unpacking reveals the underlying code that can be analyzed for malicious intent. However, as malware evolves, packed software can vary greatly in its packing techniques, requiring analysts to stay ahead of emerging trends in obfuscation methods. Furthermore, new packing methods are frequently introduced, posing an ongoing challenge to existing detection systems. Existing packed software identification methods largely rely on known training sets, which can identify known types of packed software but struggle with the opened-set problem, where new or unknown packed software types are encountered. To address this issue, this paper introduces the problem of identifying packed software in both closed-set and opened-set scenarios and proposes an evaluation mechanism using known/unknown recall rates to assess the ability to recognize both types. The known recall rate evaluates the system’s ability to identify known types, while the unknown recall rate measures its ability to recognize new, unknown packed software. This dual approach helps bridge the gap between identifying familiar threats and detecting previously unseen ones, which is increasingly important as malware continues to evolve. Additionally, the paper proposes a strategy that simultaneously addresses both recognition problems, aiming to improve the overall performance of the identification system. Experimental results on a packed software dataset demonstrate that this strategy significantly improves the accuracy and comprehensiveness of identification, validating its effectiveness in practical applications.