HDB++, a retrospective on 5+ years using TimescaleDB

The Tango HDB++ project is a high-performance, event-driven archiving system that stores data with microsecond resolution timestamps. HDB++ supports various backend databases to accommodate any infrastructure choice, with Timescale as the default option. Timescale, an extension of PostgreSQL, is selected for its exceptional performance, reliability, and open-source license. After more than five years of using the system in production at major facilities such as the ESRF, MAX IV and SKAO, this paper presents the insights gained from operating HDB++ with Timescale in a large research facility. Results are presented considering various perspectives. From a performance standpoint, the paper examines how the scalability features have maintained low query response times despite the continuous growth in data volume over the years. From the system administration perspective, findings show that standardized and proven technologies have consistently supported high-quality service delivery. Lastly, from the user perspective, we analyze how users can query data stored from the inception of the project up to last week within seconds, either from the python API or from clients like grafana. This capability is also enabled by the successful migration and integration of archived data from older or different systems into the database in full compliance with HDB++ standards.