Autism Insight: Harnessing PySpark for Early Spectrum Detection

Autism Spectrum Disorder (ASD) early detection is crucial for effective intervention, but traditional diagnostic methods struggle with complex, voluminous data. This study explores PySpark’s potential in ASD prediction, leveraging its distributed computing capabilities for efficient large-scale data processing. We implemented and compared multiple machine learning models-Logistic Regression, Random Forest, Decision Tree, and Support Vector Machine (SVM)-using PySpark’s MLlib. The models were evaluated on a dataset of 600 entries with 16 behavioral features. Results showed that SVM outperformed other models, achieving 96.5% accuracy and ${9 7. 6 \%}$ F1 score. Random Forest followed with ${9 0. 8 \%}$ accuracy and $93.8 \%$ F1 score. Decision Tree and Logistic Regression achieved $84.4 \%$ and $73 \%$ accuracy, respectively. This research demonstrates PySpark’s ability to enhance both speed and accuracy in ASD prediction, particularly with extensive datasets. The findings contribute to advancing big data analytics in healthcare, offering potential improvements in early ASD detection and intervention strategies.