FLUEnT: Transformer for detecting lung consolidations in videos using fused lung ultrasound encodings

Pneumonia is the leading cause of death among children around the world. According to WHO, a total of 740,180 lives under the age of five were lost due to pneumonia in 2019. Lung ultrasound (LUS) has been shown to be particularly useful for supporting the diagnosis of pneumonia in children and reducing mortality in resource-limited settings. The wide application of point-of-care ultrasound at the bedside is limited mainly due to a lack of training for data acquisition and interpretation. Artificial Intelligence can serve as a potential tool to automate and improve the LUS data interpretation process, which mainly involves analysis of hyper-echoic horizontal and vertical artifacts, and hypo-echoic small to large consolidations. This paper presents, Fused Lung Ultrasound Encoding-based Transformer (FLUEnT), a novel pediatric LUS video scoring framework for detecting lung consolidations using fused LUS encodings. Frame-level embeddings from a variational autoencoder, features from a spatially attentive ResNet-18, and encoded patient information as metadata combiningly form the fused encodings. These encodings are then passed on to the transformer for binary classification of the presence or absence of consolidations in the video. The video-level analysis using fused encodings resulted in a mean balanced accuracy of 89.3 %, giving an average improvement of 4.7 % points in comparison to when using these encodings individually. In conclusion, outperforming the state-of-the-art models by an average margin of 8 % points, our proposed FLUEnT framework serves as a benchmark for detecting lung consolidations in LUS videos from pediatric pneumonia patients.