A novel hybrid multilayer hyper chaotic and nonlinear modular encryption algorithm for enhancing the security of multimedia information in IoT video codecs H.264 and H.265

Several multimedia encryption algorithms suffer from security and performance limitations, including low resistance to cyberattacks, high computational overhead, potential quality degradation, and codec format incompatibility. To address these challenges, we proposed an encryption framework that integrates hybrid multi-layer hyper chaotic modules. The Cross-Coupled Logistic Confusion Engine provides initial masking and high-entropy diffusion, while the Lorenz Chaotic Fusion Scheme enables dynamic 3Dimensional substitution and cross-dimensional diffusion. The Hybrid hyper-chaotic-Based Confusion Layer introduces dual-map hyper-chaotic substitution with algebraic nonlinearity, and the Dynamic Round Key Mixer ensures Henon-based dynamic key expansion and a large composite key space. The Rule-Based Dynamic Permutation Module, together with iterative chaotic feedback and a dynamic multi-round substitution-permutuation network.The results demonstrate that the proposed encryption algorithm achieves a high level of security statstical test such as entropy (7.9999), a correlation(−0.0003), and a superior number of pixel change rate (99.63). In addition, our proposed encryption scheme achieves a significantly normal effective key (Entropy), which grows exponentially with the configuration size: total effective key entropy approximately $$2^{1024}$$ for the 256-bit configuration, $$2^{2048}$$ for 512-bit, and $$2^{4096}$$ for 1024-bit. This exponential growth demonstrates the system’s scalability and robustness, ensuring that exhaustive search attacks are computationally infeasible. Furhermore, the real-time efficiency ratio results are significantly greater than 1, ranging from 5.53 to 30.30, with an average of 15.89, demonstrating that the proposed encryption scheme achieves high computational efficiency and is suitable for practical real-time IoT video applications.The proposed algorithm also preserves video quality, as the experimental results show an infinite peak signal-to-noise ratio and a structural similarity index measure of one, indicating near-original video quality after post-encryption. Overall, the results demonstrate that the proposed algorithm achieves a high level of security, efficient performance, and quality preservation, while maintaining codec format compliance and supporting real-time multimedia communication in MP4 (H.264/Advanced Video Coding and H.265/High Efficiency Video Coding videos).