Freight rail activity inventory system using a vision‐based deep learning framework

Rail freight serves as a reliable cost-effective and fuel-efficient mode for long-distance ground freight transportation. Existing rail data sources rely heavily on aggregate reports that lead to significant spatiotemporal data gaps for infrastructure planning and regulatory evaluation. This paper presents RailVM—a vision-based deep learning framework for freight rail monitoring using infrared-enabled side-fire cameras. RailVM can accurately identify railcar and locomotive classes and extract unique locomotive tag identifiers for continuous 24/7 monitoring. It introduces three key innovations. The first is a depth-aware background modeling module that incorporates depth information to improve foreground extraction across diverse environments. The second is an advanced you only look once (YOLO)-based object-detection model—rail-specific-YOLO—that integrates a triplet attention mechanism and a Rail-intersection over union loss function to improve the identification of low-profile railcars. The third is that RailVM enables continuous day–night monitoring using infrared imaging to ensure accurate performance even in low-visibility conditions. RailVM was designed and validated for independent transferability at major rail freight gateways in California. Remarkably, it reduced gondola count errors from 41% to 2% and achieved under 5% mean error across 14 railcar classes in red, green, and blue as well as infrared modes of operation, outperforming baselines and demonstrating potential for robust real-world generalization.