Prometheus system monitoring stack

The MeerKAT radio telescope, a 64-dish instrument located in South Africa, represents a significant leap in Southern Hemisphere radio astronomy, providing unprecedented sensitivity prior to its integration with the Square Kilometer Array (SKA). The operational efficiency of complex projects like MeerKAT relies heavily on robust Control and Monitoring (CAM) systems, which is underpinned by over fifty Linux server infrastructures. Ensuring the stability and performance of these servers is paramount to maximizing scientific output. This paper details the implementation and benefits of a Prometheus-based monitoring stack, designed to provide comprehensive surveillance of the MeerKAT CAM system’s hardware, operating systems and CAM services. The system proactively detects issues , triggers alerts, speeds remediation, and improves troubleshooting, ultimately minimizing downtime and protecting critical data. The monitoring stack for CAM was deployed with Ansible. The Node Exporter application extracts the operating system metrics such as node_memory_MemFree_bytes, node_filesystem_free_bytes as well as our custom CAM services metrics such as karoo_camlog_status ,karoo_vault_status and feeds them into Prometheus. Prometheus processes these metrics and generates alerts using python scripts that the Alertmanager application then uses to generate notifications and alarms via the Mattermost messaging application. Grafana transforms stored metrics in Prometheus into interactive dashboard.