Advanced image encryption scheme based on generalized triangle group and neural networks

This work introduces a hybrid image encryption scheme that combines algebraic cryptography and deep learning to improve security and flexibility. The goal is to build a safe and effective image encryption algorithm based on parametrization of Generalized Triangle Groups (GTGs) to create powerful S-Boxes, with the integration of neural networks to add learning-based transformations. The performance is tested through statistical, differential, and structural analysis. The experimental outcomes verify robust cryptographic characteristics, such as a nonlinearity-112, avalanche effect-54.68%, and NPCR-99.36% and UACI-33.41% against differential attacks. In comparison with purely algebraic schemes, the hybrid model provides slightly weaker bit-level diffusion but enhanced pixel-level randomness and adaptability. The originality of this work is in the combination of GTG-based algebraic structures with deep learning models—a field not well researched in previous encryption studies. This combination yields a versatile and secure encryption system that adds new knowledge to hybrid cryptosystem design for image protection.