Enhancing RSA with Modified RC4 Using Optimize Symmetric Random Function Generator

Securing data transmission over public networks is increasingly challenging due to RSA's reliance on time-consuming prime number generation, susceptibility to cryptanalytic attacks, and the high cost of managing large key sizes. This study introduces an enhanced approach that integrates a modified RC4 stream cipher with an optimized symmetric random function generator, significantly boosting RSA's security and efficiency. The enhanced RSA demonstrated high throughput, peaking at 0.48 KB/s, and performed exceptionally well with larger files. Encryption and decryption times were efficient, with decryption consistently faster, with entropy values between 6.41 and 6.43 indicating strong data randomness. The P-values are above 0.05 confirmed keystream unpredictability. Validation by NIST SP 800–22 confirms the enhanced RSA perform strong randomness, enhancing security, and building trust in its efficiency. These results reveal that the enhanced RSA with optimized RC4 significantly improves efficiency and strengthens protection against attacks, while optimizing performance for data transmission over public networks.