A monitoring system to improve fault diagnosis in telescope arrays

The Ground-based Wide-Angle Cameras array necessitates the integration of more than 100 hardware devices, 100 servers, and <styled-content style-type="number">2500</styled-content> software modules that must be synchronized within a 3-second imaging cycle. However, the complexity of real-time, high-concurrency processing of large datasets has historically resulted in substantial failure rates, with an observation efficiency estimated at less than 50% in 2023. To mitigate these challenges, we developed a monitoring system designed to improve fault diagnosis efficiency. It includes two innovative monitoring views for state evolution" and “transient lifecycle”. Combining these with “instantaneous state” and “key parameter” monitoring views, the system represents a comprehensive monitoring strategy. Here we detail system architecture, data collection methods, and design philosophy of the monitoring views. During one year of fault diagnosis experimental practice, the proposed system demonstrated its ability to identify and localize faults within minutes, achieving fault localization nearly ten times faster than traditional methods. Additionally, the system design exhibited high generalizability, with possible applicability to other telescope array systems.