A multimodal learning and simulation approach for perception in autonomous driving systems

Autonomous driving has witnessed substantial advancements, yet achieving reliable and intelligent decision-making in diverse, real-world scenarios remains a significant challenge. This paper proposes a deep learning-based framework that integrates multimodal sensor fusion, advanced 3D object detection, digital twin simulation, and explainable AI to enhance autonomous vehicle (AV) perception and reasoning. The framework combines data from LiDAR, radar, and RGB cameras through multimodal fusion to capture a comprehensive understanding of the driving environment. A deep convolutional backbone, ResNet-50, is utilized to extract rich spatial features, while a Transformer-based architecture incorporates temporal context to improve trajectory prediction and decision-making. Experimental evaluations are conducted using the nuScenes dataset (v1.0-trainval split, comprising 850 scenes), which offers diverse and synchronized multimodal sensor data. Ablation studies validate the superiority of ResNet-50 over variants like ResNet-18 and 34, as well as EfficientNet-B0, achieving the lowest validation loss (0.43) for visual feature extraction, with an optimal PointPillars backbone depth of 2 for 3D detection. Extended training over 80 epochs with cosine-annealing learning rate scheduling and enriched inputs (ego-velocity, heading) further enhances convergence, as evidenced by stable loss curves below 1.0. The digital twin component simulates real-time vehicle behavior and environmental dynamics with latency under 50 ms, facilitating safe testing and validation in a virtual environment. Furthermore, explainability is achieved through Grad-CAM, which provides visual insight into model decisions, enhancing transparency and trust in AV systems. Results demonstrate competitive trajectory forecasting performance with average displacement error (ADE) of 2.82 meters (m) and final displacement error (FDE) of 6.17 m for the proposed Transformer model, outperforming physics-based baselines (e.g., Constant-Velocity: ADE 4.27 m, FDE 7.27 m; Constant-Acceleration: ADE 5.10 m, FDE 10.46 m) and showing near-competitive results against an LSTM baseline (ADE 2.58 m, FDE 5.44 m). The framework achieves a mean Average Precision (mAP) of 72.8 in 3D detection, confirming enhanced perception, decision reliability, and overall autonomy. This integrated approach represents a significant step toward the development of safe, explainable, and intelligent autonomous mobility systems.