Large-Scale Recurrent Neural Networks with Fully Homomorphic Encryption for Privacy-Enhanced Speaker Identification

Temporal classification tasks such as speaker identification are often performed by recurrent neural networks (RNNs) that observe potentially private (sensitive) data in order to provide service. Although encrypting this data safeguards it during storage and transit, decryption for computation introduces a potential vulnerability. Fully homomorphic encryption (FHE) is a privacy-enhancing technology that supports computation over encrypted data. A neural network with multiple RNN layers and attention over encrypted data for this task is presented. Using GPU acceleration and novel contributions: (1) a RNN quantization procedure with ternarized parameters and binarized activations, and (2) a ciphertext-ciphertext multiplication method for attention that reduces required computation by 50%, yields the first published multi-layer RNN with attention over encrypted data. This marks a significant step toward practical privacy-enhanced temporal classification.