A Systematic Review of Kernel-Level Security Mechanisms, Vulnerability Detection and Mitigation in Modern Operating Systems

Kernel attacks are still one of the most severe threats to modern operating systems (OS) due to the kernel's privileged control over hardware, memory, and process management. This study reviews some significant kernel-level security mechanisms regarding vulnerability detection, as well as the prevention and mitigation of exploitation in today's OSs. Using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) methodology, a total of 30 high-quality, peer-reviewed studies were examined and analyzed in detail using the Critical Appraisal Skills Programme (CASP) quality framework. Discussion about the leading research directions emanated from three central questions of this review: What are the predominant kernel attack vectors? How are the techniques for protection and detection that are currently available assessed? What are the emerging research directions? The study identifies the following as the principal sources of kernel compromise: memory corruption, privilege escalation, rootkits, and race condition exploits. It also identifies several techniques for kernel hardening, such as Mandatory Access Control (MAC), the use of SELinux and AppArmor, kernel integrity monitoring, secure and measured boot, fuzz testing, and hardware-assisted protection. Some of these emerged as having a great deal of promise for proactive defense against zero-day vulnerabilities, including machine learning-based detection and live kernel patching. Issues regarding scalability, detection accuracy, and securing containerized and virtualized environments need to be solved. This paper aims to provide relevant, structured, and up-to-date research on kernel security synthesis and offer valuable guidance on the development of robust, adaptive, and novel OS defense mechanisms.