Cryptanalytic Extraction of Neural Networks with Various Activation Functions

Originally introduced as a machine learning problem in 1991, model extraction was explicitly cast as a cryptanalytic challenge at CRYPTO 2020 and has since gained increasing prominence in this context. While early work focused on ReLU-based neural networks, recent studies have investigated model extraction in the raw-output setting for PReLU-based models. However, research on other activation functions remains largely unexplored. In modern deep learning, activation functions beyond ReLU are widely used, thereby creating a need for extraction techniques that can accommodate a wider variety of activation functions. This paper broadens the scope of model extraction by introducing a systematic framework for parameter recovery that is specifically tailored to different categories of activation functions. In addition to ReLU and PReLU, we investigate several other activation functions, including Leaky ReLU, HardTanh, ELU, and the Step function. To the best of our knowledge, this is the first study to explore model extraction for these activation functions and for PReLU-based models in the hard-label setting. We provide a detailed theoretical analysis of the properties of each activation function, propose novel attack strategies, and offer new theoretical insights. The effectiveness of our approach is demonstrated through model extraction attacks in both the raw-output and hard-label settings. Moreover, we discuss the security implications of activation functions for neural network design and explore how composite or mixed activation functions may enhance security. This work provides valuable insights into model extraction and introduces a flexible framework that may have meaningful implications for both the cryptographic and machine learning communities.