An Algebraic Approach of Multilayer Color Image Cryptosystems via Permutation-Substitution Groups and Chaotic Maps

In this manuscript, an algebra-based image encryption scheme is presented, in which the application of an S-box derived from quadratic polynomial mappings over the ring ℤ256, chaotic maps, and an algebraically enriched random matrix affine cipher is included. The algebraic foundation of the utilized RMAC is rooted in the ring of integers modulo n, specifically ℤ256, which represents the pixel intensity space for an 8-bit grayscale image. Our approach is to first encrypt the image using an S-box; further, to enhance security, we apply chaotic maps such as the tent map and logistic map. The final encryption layer utilizes a random matrix affine cipher, framing the encryption process as a sequence of linear and non-linear operations over finite algebraic structures. We have used the high non-linearity property of the S-box and the pseudo-randomness property of chaotic maps in order to make our system robust. To check the robustness of our system, we have run security analysis tests, histogram and chi-square analysis, alongside standard metrics such as MSE (𝓔ms), PSNR (𝓡sn), and SSIM (ηss). We further assessed its robustness against active threats through noise, occlusion, and differential attack simulations. Notably, the SSIM scores between the original and encrypted images approach zero, confirming that the ciphered output retains no structural resemblance to the plaintext—a key indicator of a secure and effective encryption process. Entropy of all the ciphered images is greater than 7.99, and the PSNR values are less than 10 dB. The results prove the efficiency of our proposed approach.