Efficient small file management in Hadoop distributed file system for enhanced e-government services

Purpose This paper introduces the Efficient Small File Management Algorithm (ESFMA) to overcome the challenge of small file inefficiency of Hadoop distributed file system (HDFS) for e-government services. Design/methodology/approach ESFMA is designed with the following features: hierarchical metadata architecture, caching, block aggregation, prefetching and locality-aware data placement. These are intended to optimize NameNode memory usage, metadata handling, data block management, I/O and network performance. The algorithm was implemented in experiments on HDFS with real e-government small files. Findings The experiments showed that ESFMA saves 10% of NameNode memory, 12% of metadata requests, 3.8% of data block use, 15% of read latency, 17% of write latency and 10% of network traffic. Practical implications This study suggests that implementation of ESFMA has the potential to enable better e-government services in HDFS to be run efficiently and effectively. Originality/value This paper presents an algorithm for small file management in HDFS, filling an important need in improving service efficiency and performance in e-government services.