Fine-Tuning TinyLLaMA and GPT-2 with Proximal Policy Optimization for Sentiment Enhancement and Toxicity Reduction

Large language models (LLMs) while incredibly advanced, have the potential to produce toxic or biased outputs, making it crucial to address these issues for their safe and ethical deployment. Proximal Policy Optimization (PPO) is a highly efficient reinforcement learning algorithm that balances the exploration and exploitation of the data. It is particularly suitable for fine-tuning LLMs to improve their alignment, safety, and task-specific performance. This research evaluates the effectiveness of PPO in fine-tuning GPT-2 and TinyLLaMA models using the IMDB (Internet Movie Database) dataset, a widely recognized benchmark for sentiment analysis containing labelled movie reviews. The study examines the models’ ability to generate sentiment-aligned text while maintaining linguistic quality, computational efficiency, and resilience to adversarial prompts. The results demonstrate that TinyLLaMA outperforms GPT-2 in several key areas, including sentiment enhancement, rollout scores, and consistency of outputs. TinyLLaMA also exhibits faster convergence and greater stability during training, as reflected in lower policy loss and reduced KL (Kullback-Leibler) divergence. While GPT-2 shows advantages in computational efficiency, its outputs are less consistent and diverse compared to TinyLLaMA. These findings highlight TinyLLaMA’s potential for real-world applications that demand precise sentiment control and high-quality text generation, providing valuable insights into optimizing LLM reinforcement learning.