Pushing The Limit of LLM Capacity for Text Classification

In this era of open-ended language modeling, where task boundaries are gradually fading, an urgent question emerges: have we made significant progress in text classification with the full benefit of LLMs? To answer this question, we propose RGPT, an adaptive boosting framework tailored to produce a specialized text classification LLM by recurrently ensembling a pool of base learners. The base learners are constructed by adaptively adjusting the distribution of training samples and iteratively fine-tuning LLMs with them. Such base learners are then ensembled to be a specialized text classification LLM, by recurrently incorporating the historical predictions from the previous learners. Through a comprehensive empirical comparison, we show that RGPT significantly outperforms 8 state-of-the-art (SoTA) PLMs and 7 SoTA LLMs on four benchmarks by 2.90% on average. Further evaluation experiments reveal a clear superiority of RGPT over average human classification performance.