A Scalable Framework for Big Data Analytics in Psychological Research: Leveraging Distributed Systems and Cluster Management

Anxiety and depression are prevalent psychological disorders that can occur throughout life, with a notably higher prevalence among women during the perinatal period, encompassing pregnancy and the postpartum phase. The early detection and monitoring of these conditions are crucial for timely intervention and improved patient outcomes. Although healthcare analytics has progressed considerably, the extraction of actionable insights from large-scale patient data remains computationally intensive, especially under instant processing constraints. Furthermore, conventional healthcare infrastructures frequently lack the scalability, computational efficiency, and architectural flexibility required to integrate machine learning models into clinical workflows effectively. To address these challenges, the proposed distributed computing framework employs Apache Kafka for instant data streaming, Apache Spark for efficient in-memory machine learning–based analytics, and Kubernetes for orchestrating scalable, fault-tolerant deployment. This architectural configuration facilitates continuous data ingestion, accelerates analytical processing, and ensures system resilience, thereby enabling the timely identification of psychological conditions such as anxiety and depression during the perinatal period. Unlike schematic or batch-only prior work, we provide a production-ready, streaming-first clinical deployment and an empirical scaling analysis linking executors to end-to-end diagnostic latency and resource efficiency. Performance evaluations demonstrate the efficiency and scalability of the proposed system, highlighting its potential for real-world applications in healthcare analytics.