The Importance of System Engineering Competences and Knowledge for Big Data Science and Research Infrastructure Projects

Big Data Science projects are an essential part of the modern data driven science and require extensive use of digital technologies and are typically built using large scale Research Infrastructures (RI) that combine computational, storage and data management facilities to support data collection, processing and scientific workflow management. The success of realising Big Data Science projects and effective use of Big Data Infrastructures (BDI) will depend on adopting modern technologies and well-defined architectures that support the sustainability and long term evolution of technical solutions. Building and operating modern BDIs and data driven scientific instruments require a wide spectrum of competences and knowledge related to modern technologies, system and software engineering, including also Big Data infrastructure engineering. The paper provides background information on large European RI and e-Infrastructure projects and analyses different cases/scenarios where the availability of necessary system and software engineering competences and knowledge are critical for the success of RI. The paper summarises the Sustainable Architecture Design Principles (SADP) that can provide both guidance for the technical design of RI/BDI and a basis for targeted training for engineering and scientific personnel. The paper also refers to ongoing research and developments on ensuring the environmental sustainability of RIs where the proposed SADP plays an important role. The presented work is based on the author’s experiences of developing and teaching courses on Big Data Infrastructure Technologies for Data Analytics (BDIT4DA) and DevOps for Software Engineering (DevOps4SE) that include necessary competences and learning outcomes required for building modern data-driven infrastructures and applications. The paper shares the experience of how both courses have evolved and been adapted to the continuous technology development and specific needs of the developer teams.