Hybrid Vigenere and elliptic curve cryptography technique over the finite field $$\mathbb{F}_{256}$$

Data security has become one of the primary concerns, particularly for images, as conventional encryption methods such as the Vigenere cipher or ring [Formula: see text] -based methods are no longer robust against modern types of attacks. To resolve the issue, this paper introduces a novel hybrid image encryption technique that combines elliptic curve cryptography (ECC), finite field extensions integrating the AJ chaotic map for dynamic parameters generation and image-dependent key adaptation. First, the method depends on the pseudorandom selection of an elliptic curve over the field [Formula: see text] designed based on an irreducible polynomial of degree 8. Second, two substitution tables are designed based on discrete logarithm and exponentiation operations to improve the impact of the confusion and diffusion. Numerous in-depth security experiments, including entropy, NPCR, UACI, correlation coefficients, and the NIST statistical test, have been performed, indicating high cryptographic robustness against known attacks. The findings affirm the performance of the proposed method in achieving an average entropy of 7.9998 bits per pixel, a correlation coefficient of adjacent pixels less than 0.001, NPCR of 99.79%, and UACI of 33.59% for the Peppers color image. In addition, our technique achieves a competitive execution time, confirming both its security and efficiency. Therefore, it is evident that the employment of ECC over a finite body along with a dynamically designed SBox is an ideal high-performance method for privacy protection, secure data storage, and trustworthy transmission of sensitive data.