Optimal Preprocessing and Feature Engineering Combination for DANA App Sentiment Classification: A Full Factorial Study

Purpose: Selecting optimal text-processing methods for sentiment analysis of digital financial application reviews, especially in Indonesian, is challenging. Previous studies have typically evaluated preprocessing techniques, feature engineering methods, and classification models in isolation, leaving the question of their optimal combination unresolved. This study aims to systematically determine the best pipeline for sentiment classification of DANA application reviews on the Google Play Store. Methods: A full factorial experimental design was employed, generating 24 combinations across four preprocessing configurations (no preprocessing, stopword removal, stemming, and stopword removal combined with stemming), three feature extraction methods (Bag of Words, TF-IDF, and N-gram), and two classifiers (Naive Bayes and Logistic Regression). A dataset of 57,115 Indonesian reviews, comprising 34,435 positive (60.3%) and 22,680 negative (39.7%) instances, was used, with labels based on user star ratings. Result: From the experimental results, it was found that the best setting is one that does not involve any pre-processing, the use of N-grams, and Logistic Regression, giving an F1-Score of 0.8772 and surpassing the baseline F1-Score by 1.82 percent. Logistic Regression has been found to be significantly better than Naive Bayes using the Mann-Whitney U test (p=0.0001). Preprocessing and feature extraction independently had significant effects on model performance, as indicated by the Kruskal-Wallis test (p=0.0274 for preprocessing and p=0.0269 for feature extraction). N-gram was proven superior to TF-IDF in classifying the negative class based on the Wilcoxon Signed-Rank test (p=0.0039). However, preprocessing and feature extraction did not jointly exert a significant effect on model performance based on the Scheirer-Ray-Hare test (H=6.0117, p=0.3051), and no significant interaction effect was found between both factors (H=0.3683, p=0.9991), implying that both factors can be independently optimized. Novelty: This study contributes a systematic full factorial evaluation framework for Indonesian NLP pipelines in the fintech domain. Unlike previous studies that tested variables in isolation, this approach enables a fair and comprehensive comparison across method combinations, offering practitioners evidence-based guidance for sentiment analysis method selection. Future research may explore deep learning or large language model-based approaches to further improve classification performance.