DatApollo: Orchestration of Serverless Functions for Scalable Data Mining

The exponential growth of data generated from enterprise systems, social networks, and Internet of Things (IoT) devices presents major scalability and efficiency challenges for traditional data mining techniques. Among these, Association Rule Mining (ARM) – a foundational unsupervised learning method for discovering frequent patterns in transactional datasets – frequently encounters performance bottlenecks in distributed environments due to excessive memory consumption, static resource provisioning, and costly data shuffling. This paper introduces <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">DatApollo</i>, a novel serverless orchestration framework designed to support scalable and efficient execution of distributed ARM workflows. Built on the Apollo orchestration engine, DatApollo employs stateless cloud functions with dynamic scheduling, intermediate state persistence, and fine-grained fault-tolerant coordination to address the limitations of both traditional cluster-based architectures and existing Function-as-a-Service (FaaS) models. The framework decomposes Apriori-style ARM pipelines into orchestrated micro-functions, enabling elastic, cloud-native execution with minimal idle overhead. We detail the architectural design, algorithmic components, and computational complexity of DatApollo, followed by a thorough experimental evaluation using real-world healthcare and meteorological datasets. Compared to Apache Spark, DatApollo demonstrates up to a 5-times speedup in execution time and significantly lowers infrastructure costs by leveraging elastic scaling and event-driven function invocation. These results position DatApollo as a robust, cost-effective, and high-performance alternative for ARM in dynamic, large-scale data environments.