A Collaborative Error-Tolerant Synchronization Framework With its Application in Multimodal Light Field System Development

Multimodal systems integrating cameras, LiDARs, radars, and other sensors are increasingly essential for intelligent transportation due to their superior environmental perception. However, the lack of a unified synchronization mechanism across diverse sensor modalities poses challenges for system-wide synchronization, causing data misalignment, reduced acquisition frequency, and degraded perception accuracy. This paper proposes a novel multimodal synchronization framework that enables high-frequency and low-latency data acquisition to improve perception accuracy. The framework integrates multi-frequency hardware synchronization with a temporal-spatial error-tolerant model to minimize data misalignment while maximizing effective frame rates. A frame header-based time synchronizer generates corrected multi-frequency trigger signals, achieving sub-4ms synchronization across 12 sensors and 4 modalities, improving the mean Intersection over Union (mIOU) of dynamic objects by an average of 4.97% in point cloud projection and 15.50% in image registration compared to existing methods. It enables us to build a multimodal system with the highest modality count (a 3×3 camera array, a LiDAR, a radar, and a long-wave infrared camera) and frame rate, capable of capturing novel light field data, outperforming existing multimodal setups. Evaluations on our real-world dataset show Average Precision (AP) gains of 49.35% for cyclists and 39.53% for pedestrians in object detection, along with significant improvements in image translation.