LFI: a communication library for high-performance networks

Abstract Nowadays, high-performance networks in supercomputers have become indispensable for applications such as artificial intelligence, data analysis, and simulation, among others, all of which require high network speeds and minimal latencies. The standard of choice for developing these applications is MPI, as most implementations support such networks. However, some implementations of MPI exhibit notable limitations in error handling and with a rigid initialization model that proves problematic for certain applications. In addition, applications that use POSIX sockets or traditional RPC cannot benefit from these network interfaces. In response to these deficiencies, the Lightweight Fabric Interface (LFI) library has been developed as an alternative for communications, providing a very simplified user interface, error notification based on “is alive” requests, a more flexible initialization model, and is designed for efficient use in multi-threaded applications. LFI offers an interception library that enables applications using POSIX sockets or traditional RPC to benefit from high-performance networks without any modifications. Evaluations demonstrate that LFI delivers superior performance and efficiency compared to established communication libraries across both Omni-Path and InfiniBand architectures. When compared to MPICH, one of the most widely adopted MPI implementations, LFI improves bandwidth performance by up to 36% in real-world applications while substantially reducing resource overhead in multi-threaded environments by up to 97%. Furthermore, LFI consistently outperforms MPICH, OpenMPI, Mercury, and POSIX sockets (IPoIB) in terms of both raw throughput and operational efficiency, establishing it as a highly scalable solution for high-performance interconnects.