Optimizing Real-Time Data Processing in Resource-Constrained Environments: a Spark and Gpu-Driven Workflow for Large Language Models

Resource limitations on computational infrastructure reduce the effective use of Large Language Models (LLMs) for processing streaming data, especially in research. The combination of Apache Spark and Graphics Processing Units (GPUs) has revolutionized large-scale data processing but benefits primarily resource-rich computational clusters. This research explores the potential of an advanced real-time data processing pipeline built on Apache Spark Streaming, efficiently managing data flow across streaming channels. This enhanced infrastructure significantly improves the real-time analytical capabilities of LLMs, enabling them to perform scalable and context-aware batch tasks more efficiently. Our proposed pipeline expands the functionality of the Spark ecosystem, particularly in GPU-accelerated clusters, which solves the resource limitation of low computational power machines. Deploying LLMs across Spark's distributed nodes ensures their readiness for various analytical tasks. In a practical scenario, the system's performance is assessed by evaluating student programming assignments and giving feedback through LLMs. We tested the system across multiple tasks, including code generation from questions, and compared single, dual, and triple-worker configurations, focusing on time efficiency and resilience. Our results show that the pipeline achieves up to a 50% and <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$70-80\%$</tex> reduction in inference time with dual and three workers, respectively, without compromising result accuracy. The system also demonstrates resilience through a failover strategy that ensures seamless task redirection to alternate nodes in case of failure, maintaining continuous operation. Including additional workers significantly enhances data processing capacity and improves load distribution, making the system scalable for more resource-intensive tasks. This adaptability benefits researchers with different resource capabilities, optimizing the performance of LLMs in diverse programming and code generation scenarios.