A method of self-learning of a cyber defence system based on behavioural invariants of polymorphic and metamorphic malware

New samples of malware, in particular polymorphic and metamorphic, are characterised by increased complexity and the ability to change behavioural characteristics to avoid detection by existing cyber defence systems, which poses a serious threat to the cybersecurity of information and communication systems. The main problem of existing cyber defence systems in this context is the use of databases describing malicious code samples, which lose their relevance over time and become unable to detect new implementations of cyber threats.The article proposes a method of self-learning of a cyber defence system based on behavioural invariants of polymorphic and metamorphic malware. This method involves the use of a two-tier detection system: the first tier analyzes the activity of the information and communication system to detect malware by behavioural features, the values of which are described by fuzzy linguistic terms for each known type of malware; the second tier uses a preformed database of behavioural invariants described by fuzzy product rules to increase the efficiency of detecting polymorphic and metamorphic malware.The key feature of the proposed method is the automatic generation of new fuzzy rules to describe the behavioural characteristics of unknown malware samples, which is achieved by combining the invariant and polymorphic components in the case of detecting the invariant component. Additionally, the system provides for the redefinition of invariant components by re-selecting such features that are naturally repeated in the aggregate for most instances of each type of malware. This process ensures self-learning of the cyber defence system, which makes it possible to more effectively counter polymorphic and metamorphic malware.