Learning pentary embedding costs for enhancing image steganography security via curriculum-based GAN

Abstract Embedding-cost design is central to distortion-minimization steganography, yet most GAN-based cost-learning methods are restricted to ternary modifications ( $$\{0,\pm 1\}$$ { 0 , ± 1 } ), which can become limiting at high payloads. We propose Curriculum-based Pentary Embedding GAN (CuPE-GAN) for learning pentary ( $$\{0,\pm 1,\pm 2\}$$ { 0 , ± 1 , ± 2 } ) embedding costs. CuPE-GAN uses a two-layer ternary training surrogate: Stage 1 learns the first-layer $$\pm 1$$ ± 1 allocation, whereas Stage 2 activates the second layer with forced learning-rate decay; a directional-consistency loss discourages canceling layerwise changes; and an Improved Decoupled Ternary Approximation (IDTA) represents the two modification directions with separate one-sided soft indicators and exact zero-probability consistency at finite $$\lambda$$ λ . The learned layer probabilities are converted to a normalized five-symbol distribution and then to pentary costs, while actual message embedding is performed by a standard pentary STC encoder. Trained on CelebA-HQ and ALASKA1 and evaluated on BOSSBase and BOWS2, CuPE-GAN achieves higher average detection error rates than Steg-GMAN, WOW, and S-UNIWARD at medium and high payloads. Relative to Steg-GMAN, the average gains at 0.4 and 0.8 bpp are 6.16 and 5.23 percentage points on BOSSBase, 5.05 and 7.92 percentage points on BOWS2.