Reinventing CI/CD for Collaborative Sciences: A Blockchain-Integrated Decentralized Middleware for Scalable and Fault-Tolerant Workflows

The expansion in the scale of collaborative scientific efforts has brought about numerous challenges in continuous integration/deployment (CI/CD). Current CI/CD tools solely depend on centralized data management, resulting in reliability and scalability concerns. To address the issue, this paper advocates for and demonstrates a decentralized approach that leverages lightweight data management techniques and distributed computing principles. Our middleware solution, DeQL, transforms the current CI/CD workflows, and uses decentralized logic for automated code analysis before GitHub commits. The involvement of distributed participants (i.e., GitHub contributors), together as a part of the decentralized network, ensures code integrity, security and reliable integration processes. For ensuring low overheads and scalability, this paper introduces lightweight consensus protocols designed for efficient smart contract execution, significantly reducing overhead compared to traditional decentralized data management solutions. We incorporate a fine-grained CI/CD commit dependency graph that effectively manages CI/CD commit complexity, facilitating faster (and more reliable) processing. Experimental results show that the proposed decentralized protocol outperforms the traditional centralized CI/CD system and state-of-the-art blockchain protocols, achieving up to 80% faster response times and handling 7× more commits in a given amount of time.