KEENsight: Cloud based Collaborative Environment for Machine Learning Development

We are now in an inevitable phase where interdisciplinary projects—bringing together domain experts from various disciplines—are on the rise, driven by the powerful synergy they create. This collaborative approach aims to deliver innovative solutions that transcend the limitations of conventional methods. Similarly, in the field of machine learning, the approach has evolved from an individualistic attempt to a collaborative effort, harnessing collective expertise to tackle increasingly complex challenges as they emerge.However, increased collaboration introduces a diverse landscape of data types, formats, and sizes, making the understanding and management of such data progressively complex. Over time, the contextual integrity of this data may diminish due to factors such as the loss of individual or collective memories or the departure of key contributors, such as domain experts. To address these challenges, it becomes essential to establish a robust research data management system. Specifically, storing data using a well-defined schema within such a system ensures long-term usability while adhering to the FAIR principles: Findable, Accessible, Interoperable, and Reusable [1 ,2].Another significant challenge is effectively managing the collaborative development process in machine learning projects.The poster highlights KEENsight, a scalable, adaptive, and interoperable environment specifically designed to fulfill all DevOps requirements, with a particular focus on machine learning applications. It includes a collaborative environment designed to harness the advantages of a team-driven approach by integrating various Open Source tools. Its framework can be summarized into three core components: a research data management system, a collaborative environment, and computing resources. This poster showcases the establishment of an optimized setting for code collaboration, model development, data sharing, and efficient utilization of computational resources, with a special emphasis on research data management [3]. Reference:[1] L. Sherpa, R. Müller-Pfefferkorn, G. Tolksdorf, V. Khaydarov, M. Wiedau, and L. Urbas, “Prometas – a metadataschema for process engineering and industry,” Chemie Ingenieur Technik, vol. 95, no. 7, pp. 1041–1048, 2023.[2] Sherpa, L., Khaydarov, V. and Müller-Pfefferkorn, R. (2024) ‘FAIRness Along the Machine Learning Lifecycle Using Dataverse in Combination with MLflow’, Data Science Journal, 23(1), p. 55. doi : 10.5334/dsj-2024-055.[3] L. Sherpa, S. Attar-Khorasani, R. Sankar, R. Müller-Pfefferkorn and S. Ghiasvand, "KEENsight: Cloud BasedCollaborative Environment for Streamlining Machine Learning Development," 2024 15th International Conferenceon Information, Intelligence, Systems & Applications (IISA), Chania Crete, Greece, 2024, pp. 1-8, doi:10.1109/IISA62523.2024.10786688. Notes:Funded by: The German Federal Ministry of Education and Research (BMBF, SCADS22B) and the Saxon State Ministry for Science, Culture and Tourism (SMWK)