A quadratic integer ring-based approach to S-box construction and its integration into a 4D hyperchaotic RGB image encryption scheme

The rapid growth of digital communication, cloud computing, and internet-based services has intensified the need for secure techniques for protecting digital images against increasingly sophisticated cyberattacks. This paper presents a quadratic integer ring-based approach to the construction of substitution boxes (S-boxes) and their integration into a four-dimensional (4D) hyperchaotic (red green blue) RGB image encryption scheme. A systematic algebraic framework over the quadratic integer ring $$\mathbb {Z}[\sqrt{14}]$$ is developed for generating suitable prime elements and constructing multiple bijective $$8\times 8$$ S-boxes. The cryptographic properties of the constructed S-boxes are comprehensively evaluated using nonlinearity, strict avalanche criterion, bit independence criterion, differential probability, linear approximation probability, algebraic degree, and algebraic immunity. The obtained results demonstrate that the proposed S-boxes possess favorable cryptographic characteristics and provide resistance to linear and differential cryptanalytic attacks. To demonstrate the practical applicability of the constructed S-box, it is integrated into a noval 4D hyperchaotic RGB image encryption framework. The modified discrete 4D hyperchaotic system is rigorously analyzed through bifurcation diagrams, Lyapunov exponent spectra, complete Lyapunov spectra, and three-dimensional phase portraits to verify its complex dynamical behavior. The encryption scheme employs chaotic pixel permutation, channel- and round-dependent shifted S-box substitution, and bidirectional XOR diffusion in a two-round architecture. Extensive experimental and security analyses demonstrate near-ideal ciphertext entropy, negligible adjacent-pixel correlations, high NPCR and UACI values, strong key sensitivity, and robustness against statistical, noise, and cropping attacks. Comparative and ablation analyses further demonstrate the contribution of the proposed S-box to the overall encryption performance. In addition, experiments conducted at multiple image resolutions confirm the computational scalability and efficiency of the proposed scheme. The results demonstrate that the proposed quadratic integer ring-based construction provides a promising algebraic approach for designing cryptographically suitable S-boxes and their application in secure RGB image encryption.