Leveraging on common information model (CIM) for big data analytics

Enedis has successfully developed numerous Big Data analytics use cases, such as targeting network renewals, improving network resilience against flooding, and predictive maintenance on overhead medium-voltage (MV) networks. These use cases were implemented on an in-house framework using Spark and Scala. As the need to minimize software running costs and reuse existing code for emerging use cases grows, a more standardized approach to data management has become crucial. In this paper, we describe how we have integrated Common Information Model (CIM) objects directly into the Spark Dataset API. This model-oriented approach standardizes network representations across different use cases, thereby simplifying data processing and improving overall data management efficiency. To enable topological calculations, specific extensions to the CIM standard have been implemented, allowing integration with graph libraries. Our results demonstrate that leveraging CIM in conjunction with Spark/Scala significantly improves co de robustness. The model-oriented approach improves business specifications, testability, and code sharing across use cases. Additionally, this approach facilitates the integration of new data sources, offering a scalable and robust solution for DSO aiming to optimize their Big Data analytics workflows.