Raft-Consistent Metadata with GPU-Accelerated Encryption for Secure HDFS

In today’s cloud computing environments, large-scale data infrastructures like Hadoop are facingsevere challenges in providing efficient and secure static storage; these challenges stem fromlimitations in single algorithm encryption schemes as well as metadata management instability.This paper proposes a co-designed architecture demonstrating that metadata consistency andhigh-throughput encryption can be achieved simultaneously without performance trade-offs. Ourapproach integrates two components: (1) a Raft-consistent metadata service (RCMS) that replaces thesingle-point-of-failure NameNode with distributed consensus-based replication. Empirical evaluationshows that RCMS has a metadata fail-over latency of 1.35(+-0.42) seconds- an improvementof 78 times over ZooKeeper-based HA (105(+-28) seconds). Such fast recovery is essential formetadata-sensitive applications such as real-time analytics, streaming applications, and interactiveSQL engines. Concomitantly, GHEM provides the encryption throughput of 935 MB/s, which is51% uplift from CPU-based HDFS-TDE’s throughput of 620 MB/s. 8.1 times faster comparedto 10 GB files. The key to achieving this architecture is that RCMS and GHEM are executed innon-overlapping critical paths and that metadata transactions and data encryption are performedin parallel with the network I/O, with a total overhead of less than 1%, given all together. Thisco-design confronts this entrenched notion that strong consistency forces performance compromises.Experimental validation for a four-node prototype cluster validates both the fast fail-over of metadataand the high-throughput encryption facilities. Finally, the proposed framework is easily extendable tovarious types of distributed storage systems, including Ceph, Cassandra, and cloud object storage,which makes this framework a solid foundation for building secure, consistent, and high-performancedata storage solutions.