Design of a graph-oriented database for data management in statistical data infrastructure

• Graph-based approach for master data management in official statistics. • Integration of administrative data from multiple institutional silos. • Experimental comparison of relational, graph, and hybrid database systems. • Evaluation with real production data for generating statistical outputs. • Guidelines for migrating from relational to graph-based master data systems. In recent years, official statistical agencies have embraced the development of public statistics based on administrative records rather than surveys. This solution necessitates connecting numerous sources and silos to obtain not only the data but also their relationships, generating significant complexity in managing schemas and scalability. One of ISTAC’s objectives (Canary Islands Institute of Statistics) is to promote an Integrated Data System (iDatos) to produce multi-source statistics. The large volume of data and its continuous growth requires the use of Big Data technology to ensure its efficiency and performance. The current solution stores information in a relational database with a design based on master data management. However, in recent years, the use of graph-based databases for master data management has increased in different contexts, where relationships constitute the central element of the data model. This paper presents the transformation of the current system to a design based on graph-based databases. This paper presents the analysis of the problem, the design of the graph-based database schema, and the implementation of a computational experiment to compare performance.