Real-Time Stress Detection on Social Network Posts Using Big Data Technology

In this work, we built a real-time system for detecting stress from online posts based on the dataset “Dreaddit: A Reddit Dataset for Stress Analysis in Social Media”, collected from 187,444 posts in five different domains on Reddit. Each domain includes texts containing stressful or non-stressful content as well as different ways of expressing stress in the post, of which there are 3,553 labeled lines to form the training dataset. We used Apache Kafka, PySpark combined with AirFlow to build the model and deploy the system. LogisticRegression gives the highest results for new streaming data, reaching 69,39% for measuring accuracy and 68,97 for measuring F1-scores.