Hybrid Secure Onlooker: Enabling End‐to‐End Security for Cloud Data Center by Hybrid <scp>VM</scp> Segmentation

ABSTRACT Cloud computing is an innovative technology that provides computing services over the internet and replaces the requirement to own physical hardware or software. Security threats present a wide range of risks to cloud computing, and a security threat defense plays a significant role in cloud computing. Virtual machines (VM) serve as the backbone, providing flexible and scalable resources for running and storing data. Moving Target Defense (MTD) and Blockchain enhance security and privacy by reducing the chances of successful attacks and minimizing the impact of security attacks. To address these issues, we propose integrating MTD and blockchain technologies within the cloud computing environment named Hybrid Secure Onlooker (HSO). The proposed work involves several entities, including Cloud Users (CUs), Centralized Subnet Manager (CSM), Distributed Group Manager (DGM), Consortium Block Module (CBM) and Private Block Module (PBM). Initially, we perform Multi‐Factor Authentication (MFA) to establish secure communication and to avoid malicious traffic. Followed by this, we utilize the Komoda Miliphir optimization (KMO) algorithm to perform CUs' task scheduling based upon the task types, task sensitivity, and task size. Entrenched in the scheduled tasks, the CSM performs classification and grouping of cloud VMs, assigning them to their capacity, security protocols, and availability, utilizing the Residual Flowed Capsule Network (RFC‐Net). The grouped subsets are overseen and managed by the DGM, which handles MTD operations such as virtual switch placement and VM migration within the subsets. Finally, the transactions are stored in the hybrid blockchain layer with CBM and PBM to ensure privacy and security. The is the implementation tool for realizing the proposed HSO model. The proposed model can be examined based on several metrics with state‐of‐the‐art work comparisons. The results show that the proposed HSO model outperforms the state‐of‐the‐art models.