(Old Version) Offensive Language Detection: A Comparative Study of Classical and Transformer-Based Approaches on Imbalanced Twitter Data

This is an old version. The latest version is available at https://doi.org/10.5281/zenodo.14643089. Abstract This study tackles the automatic detection of harmful comments and cyberbullying on social media, focusing on the “tdavidson/hate_speech_offensive” dataset of 24,783 English tweets. Each tweet was originally annotated with hate speech, offensive language, or neither; for this project, hate speech and offensive language were combined into a single “offensive” class, with all other content labeled as “non-offensive.” Because the dataset contains far more offensive (20,608) than non-offensive (4,158) tweets, class-weight balancing and oversampling via ADASYN were employed. Transformer-based models (BERT, RoBERTa, and an 8-bit quantized LlamaForSequenceClassification) fine-tuned via LoRA and hyperparameter-optimized with Optuna achieved 96% accuracy on Twitter test data but misclassified over 40% of samples in a separate dataset of 100 everyday usage examples. In comparison, TF-IDF-based models including Naive Bayes, Logistic Regression, SVM, and XGBoost achieved up to 94% accuracy on Twitter data but showed better generalization, correctly classifying over 70% of the everyday usage samples. I concluded that this performance disparity primarily stems from transformer models overfitting to Twitter-specific patterns and the inherent limitations of smaller model sizes in contextual understanding, while classical keyword-based approaches like TF-IDF prove more effective at identifying explicit offensive language by directly leveraging the presence of offensive words as classification features. For future work, expanding the training data to incorporate diverse language styles beyond Twitter would enhance model generalization. Furthermore, fine-tuning larger language models with tens or hundreds of billions of parameters could leverage their superior contextual comprehension capabilities, potentially leading to more robust and reliable offensive language detection systems. Code.zip Contains the source code for all models and experiments, including: A BERT baseline (fine-tuned via LoRA and hyperparameter-optimized with Optuna) RoBERTa Large (fine-tuned via LoRA and hyperparameter-optimized with Optuna) LlamaForSequenceClassification (8-bit quantized, fine-tuned via LoRA and hyperparameter-optimized with Optuna) Naive Bayes, Logistic Regression, SVM, and XGBoost (using TF-IDF) Report.pdf A PDF file containing the entire study, including the abstract, problem statement, dataset description, methodology, results, discussion, conclusion, and references. processed_data.csv A CSV file containing the preprocessed dataset used for training and evaluating the models.