Challenges with handling keys for secure AI

Secure generative AI using Fully Homomorphic Encryption (FHE) enables the utilization of cutting-edge AI capabilities while preserving the privacy of both the model and user queries. Despite its promise, the development and maintenance of scalable solutions in this area remain obscure with challenges that require further exploration. In this paper, we examine key management challenges that are often overlooked in modern research, particularly in offline flows. We identify gaps in current NIST KMS standardization, introduce the concept of a Hierarchical Key Management System (HKMS) solution, and share insights from a recent demonstration of IBM's He4Cloud design.