Plan for a renewed PaaS Orchestration solution in the DataCloud Project at INFN

In 2021, the National Institute for Nuclear Physics (INFN) launched the INFN Cloud Orchestration system to support Italy’s largest research and academic distributed infrastructure. The INFN Cloud orchestration system is an open-source middleware designed to seamlessly federate heterogeneous computing environments, including public and private resource providers, container platforms, and more. It provides a customizable service portfolio, crafted to suit the distinct needs of specific communities. It supports standard Infrastructure as a Service (IaaS) options, advanced Platform as a Service (PaaS) configurations and useful Software as a Service (SaaS) solutions, such as Jupyter Hub, Kubernetes, Spark, and HTCondor clusters. Its primary function resides in orchestrating the deployment of virtual infrastructures, ranging from simple to intricate setups, providing users with convenient access and operational control. The INFN Cloud platform’s federation middleware is composed of several interconnected open-source microservices. At the core of the orchestration system lies the INDIGO PaaS Orchestrator a Java-based REST API that handles high-level deployment requests across federated cloud providers. The other microservices of the federation middleware play an essential role in the selection of the most suitable federated provider and managing communication within the distributed environment. The most recent software upgrades can be understood as the first steps toward the definition of a new architecture based on message exchange between micro-services and exploiting Machine Learning for the optimal resource provider selection. In this context, a plan to replace the existing micro-services, exploiting newer and modern technologies, is in the making and will be adopted in the next period. In particular, the AI-ranker, devoted to the smart choice of the best provider, and the Federation-Registry, devoted to collect different information from the federated providers, will replace and evolve the in-use services offering limited features. Following a similar approach, new components will be introduced to enable advanced features, including the use of open-source Infrastructure as Code (IaC) tools to support interaction with containerized platforms such as Kubernetes. Additionally, as part of the system renovation, the Kafka queue mechanism will be adopted to manage PaaS deployments and to deliver deployment details to the INDIGO PaaS Orchestrator microservices. Finally, with the continuous growth of the number and quality of micro-services, a reliable and automated procedure aimed at securing and simplifying the deployment procedure of the core services is under definition.