A novel color images security-based on SPN over the residue classes of quaternion integers $$\:\varvec{H}{\left(\mathbb{Z}\right)}_{\varvec{\pi\:}}$$

The exponential growth of multimedia data transmission has intensified the demand for advanced image encryption systems capable of resisting contemporary cryptanalytic attacks while maintaining computational efficiency. Conventional encryption schemes often fail to provide sufficient confusion and diffusion when applied to high-dimensional color images. To overcome these challenges, this paper proposes a novel Substitution-Permutation Network (SPN)-based RGB image encryption algorithm constructed over the residue classes of quaternion integers (RQCI's) [Formula: see text] The method specifically addresses the problem of limited nonlinearity (NL) and weak algebraic complexity in existing S-box designs by introducing quaternion residue-based nonlinear substitution boxes (S-boxes) that exploit the four-dimensional nature of quaternion algebra (QA). The construction begins with quaternion prime (QP) selection and residue class formation, followed by affine mapping and coefficient decoupling to generate bijective and highly nonlinear S-boxes with strong avalanche characteristics. These S-boxes are then integrated into an SPN framework comprising substitution, permutation, and XOR diffusion layers applied independently to the red, green, and blue channels of an image. The use of quaternion arithmetic increases key sensitivity, expands the transformation space, and enhances resistance against differential, linear, and statistical attacks. Experimental evaluations demonstrate superior quantitative performance, with entropy approaching to the ideal value, Number of Pixel Change Rate (NPCR) exceeding 99.6%, Unified Average Changing Intensity (UACI) around 33.4%, and negligible correlation among adjacent pixels. Comparative results confirm that the proposed scheme achieves greater security and efficiency than existing SPN-based image ciphers. Hence, integrating quaternion residue class algebra with SPN architecture offers a mathematically grounded and practically efficient framework for robust color image encryption suitable for secure digital communication systems.