Advancing LLM Capabilities in Physics: A Reinforcement Learning Approach with Human and AI Feedback

Currently, Large Language Models (LLMs) have shown great performance in general tasks like text summarization.However, they often struggle with complex arithmetic questions and mathematical reasoning tasks.While simple approaches, such as fine-tuning LLMs for specific mathematical problem-solving tasks, have improved their reasoning abilities, they still face challengeswhenencounteringnewquestionsorvariationsinproblem-solvingapproaches. Thisstudy aims to enhance LLMs to effectively handle physics-related questions extracted from the PhyQA dataset, which consists of problem sets from Indian NCERT textbooks for grades 11 and 12. We employReinforcementLearningtoimprovetheefficacyandaccuracyofourmodelsinarithmetic reasoningforthePhyQAdataset. Variousreinforcementlearningmethods,includingDPO,ReMax, andPPOoptimization,areexploredtoassesstheirperformanceinphysicsproblem-solvingacross different scenarios.A crucial aspect of our approach involves integrating human and artificial intelligencefeedback,referredtoasReinforcementLearningwithHumanandArtificialIntelligence Feedback.Thisinnovativeapproachhelpstrainourmodelstogeneratemorelogicalandreasonable solutionstophysicsproblems. Inevaluations,theMISTRAL-PPOmodelstandsoutforitsability toproducereasonablesolutions,achievingcommendablescoressuchasa58.67%METEORScore, an80.39%ReasoningScore,and38.0%accuracyinamanualevaluationof100randomsamples, quantitatively measuring the model’s reasoning abilities.