Refining Words for Improved BERT Model Performance in User Review Classification

Understanding user needs and wants for an app is an important aspect. An app launched will be continuously improved to satisfy user needs and wants. App user reviews are an important source of information for developers to understand user needs and improve app quality. However, manual analysis of several reviews that use informal language, slang, and abbreviations is time-consuming and inefficient. Reviews can be technical (complaints about app performance), nontechnical (complaints about services, business processes), or general (no specific focus). Therefore, automatic classification is required to direct reviews to the right team to follow up on the complaints. This paper proposes a BERT-based automatic classification method to categorize reviews based on technical, nontechnical, and general aspects. The Indonesian dataset was taken from the Google Play Store, and initial labeling was performed using the Snorkel method based on keywords to reduce time and cost. Word refinement by changing slang words and spelling correction using SymSpell were applied to handle informal language and address out-of-vocabulary words in the reviews, thus improving the performance of the analysis method. This study shows that manual word refinement has the highest F1 score of 0.8924, followed by automatic refinement with 0.8681, whereas no refinement only achieves an F1 score of 0.8564.