Evaluating Large Language Models vs Traditional Machine Learning Models in Classifying Automotive Reports

Background. Efficient classification of mixed-format automotive reports is essential for improving scalability and operational efficiency in industries that rely heavily on diverse datasets. The complexity arises from handling structured numerical and categorical data along with unstructured text, necessitating robust and adaptable AI methods. Objectives. This study aims to evaluate and compare traditional machine learning models with large language models for classifying automotive reports, particularly addressing the challenge of data imbalance and mixed-format data types. Methods. A structured experimental design was utilized, incorporating traditional machine learning algorithms such as logistic regression, random forest, and LinearSVC, as well as advanced large language models with various prompting strategies, including retrieval-augmented generation. Data preprocessing involved text cleaning, translation, lemmatization, term frequency–inverse document frequency vectorization, and several balancing techniques, including synonym replacement methods (basic, embedding-based, and few-shot). Results. Results indicate that large language models, particularly with retrieval-augmented generation-based prompting and clear target descriptions, achieved significant improvements in classification accuracy over baseline traditional machine learning models. The best-performing scenario combined detailed label descriptions with recent retrieval-augmented generation examples, achieving up to 87% accuracy. Traditional machine learning models showed consistent performance but required extensive preprocessing and careful feature engineering. Conclusions. Large language models demonstrate substantial potential in accurately classifying automotive reports when enhanced with contextually relevant examples through retrieval-augmented generation. However, careful management of prompt complexity and the quality of historical data used for examples is essential. Future research should further investigate embedding quality and extend evaluation across different industries.