A two-stage algebraic framework for designing cryptographically strong S-boxes for secure image encryption

The security of modern block ciphers heavily relies on the cryptographic strength of Substitution boxes (S-boxes), which introduce non-linearity and confusion. While numerous methods for S-box generation exist, there is a continuous pursuit of designs that offer a superior and balanced profile against various cryptanalytic attacks. This paper presents a new, two-step approach to the construction of highly nonlinear S-boxes, with group-theoretic underpinnings. During the first step, we use a parameterized permutation representation of the triangle group Δ ( 2 , 3 , 4 3 ) on the projective line over the finite field F 2 5 7 , denoted by P 1 ( F 2 5 7 ) , to produce an initial S-box with a nonlinearity of 1 0 6 . The second stage is the main innovation, in which we use a refinement process via the action of the permutation group C 7406532 × C 3 on this original S-box. The refinement increases the average nonlinearity to 112 and yields differential uniformity 4, linear approximation probability 0.0625, strong bit-independence behavior, and strict avalanche characteristics close to the ideal value. The proposed S-box is then incorporated into a multi-round grayscale image-encryption scheme based on fixed S-box substitution, key- and nonce-dependent position permutation, SHA-256-based keystream whitening, and a final bidirectional nonlinear diffusion layer. Experiments on Moon Surface, Airplane, and Clock images produce ciphertext entropies of 7.9968–7.9975, near-zero pixel correlations, NPCR values of 99.7040%–99.7559%, and UACI values of 33.4902%–33.6390%. Cropping and noise experiments further show exact recovery without attack and useful attack-dependent reconstruction under ciphertext degradation, particularly under mild cropping and low-density salt-and-pepper noise. These results demonstrate strong statistical security, differential sensitivity, and practical applicability for grayscale image protection.