Mental Level Prediction using Long Sentence Discriminators and Language Generation Models

In this paper, I describe a method for mental level prediction using long sentence discriminators and language generation models in the IEEE BigData 2024 Cup: Detection of Suicide Risk on Social Media. The approach consists of five steps. First, complex preprocessing is performed on the labeled text data, which is then passed to MentaLLaMa, a language generation model, to generate paraphrased text. Second, pseudo-labels are assigned to the unlabeled text using a fine-tuned mental-longformer, one of the long sentence discriminators, with the generated text and the original labeled text. At this time, two methods are implemented for data validation: the Hold-Out method and the Stratified K-Fold method, respectively. Third, perform complex and simple preprocessing on all text and fine-tune mental-longformer for each type. At this time, cross-validation by stratified K-Fold is performed. In addition, since there are two types of pseudo-labeled text, the same content is implemented twice. Fourth, a simple preprocessing is performed on the text that implements the Hold-Out method when assigning pseudo-labels, and the predicted probabilities output by mental-longformer for that text are passed to TabPFN, a transducer-based classifier, to obtain prediction results. Finally, I compute the final result by combining the mental-longformer predictions for the complex preprocessed text with the TabPFN predictions obtained above. For fine-tuning, a combination of Label Smoothing, Focal Loss, and Class Balanced Loss was used as the loss function. The method achieved an intermediate score of 0.7657 and a final score of 0.7505, proving that it is effective in predicting for small amounts of long sentences.