Building Adaptive And Generalizable Learning Systems For Dynamic Cybersecurity Environments

Machine learning systems deployed in cybersecurity must operate in environments that change more rapidly than traditional models can accommodate. In this thesis, this challenge is examined across two settings: malware detection and sequential network decision-making. Malware continuously evolves into new families, and network topologies shift in structure and behavior, causing fixed supervised and reinforcement learning models to lose effectiveness. Both settings highlight the need for learning systems that can adapt quickly to new conditions and generalize beyond previously observed data. This thesis investigates how meta-learning can support the development of adaptive and generalizable cybersecurity models that require minimal additional data for effective adaptation. The first part of the thesis focuses on meta-learning for malware detection, addressing the problem of identifying unseen malware families under few-shot conditions. Using three datasets representing raw binary images, static features, and combined static and dynamic features, the study evaluates Model-Agnostic Meta-Learning (MAML), Prototypical Networks, and Relation Networks. Results show that meta-learning significantly improves few-shot detection when feature representations are informative. Feature-based datasets demonstrate strong generalization across malware families, while raw image-based representations exhibit limited adaptability, underscoring the importance of feature quality for generalizable malware detection. The second part applies meta-learning to sequential cybersecurity environments using NetSecGame, a network security simulator with diverse topologies. A MAML-based attacker agent is trained for data-exfiltration tasks and evaluated on its ability to adapt across network structures. Experiments show that the meta-trained agent adapts rapidly to unseen environments using only a small number of support episodes. Training on simpler topologies further improves adaptation speed by providing more successful learning trajectories. In contrast, a Reptile baseline fails to demonstrate meaningful learning under sparse-reward conditions. Overall, the findings demonstrate that meta-learning enables both classifiers and reinforcement learning agents to remain effective in dynamic cybersecurity environments.