Applying Large-Scale Distributed Computing to Structural Bioinformatics – Bridging Legacy HPC Clusters with Big Data Technologies using kafka-slurm-agent

This paper presents the Kafka Slurm Agent (KSA), an open source (Apache 2.0 license) distributed computing and stream processing engine designed to help researchers distribute Python-based computational tasks across multiple Slurm-managed HPC clusters and workstations. Written entirely in Python, this extensible framework utilizes an Apache Kafka broker for asynchronous communication between its components. It is intended for non-expert users and does not require administrative privileges or additional libraries to run on Slurm. The framework's development was driven by the introduction of the AlphaFold protein structure prediction model, specifically, it was first created to facilitate the detection of knots in protein chains within structures predicted by AlphaFold. KSA has since been applied to several structural bioinformatics research projects, among others, leading to the discovery of new knotted proteins with previously unknown knot types. These knotted structures are now part of the AlphaKnot 2.0 web server and database, where KSA is applied to manage the knot detection process for user-uploaded structures.