Using Explainable Techniques to Enhance Chain of Thoughts in LLMs

With the recent advancements in reasoning capabilities with semantic understanding, Large Language Models (LLMs) are increasingly mimicking human reasoning processes and aligning more closely with human values. Yet, unlike how humans sometimes make decisions intuitively by reasoning or understanding the underlying logic, LLMs struggle with transparency in their internal reasoning. Similar to how humans often pause to analyze their reasoning step-by-step to understand their own decisions better, we propose enabling LLMs to perform analogous introspective reasoning using a Chain-of-Thought (CoT) framework powered by SHAP-based explainability. CoT method guides LLMs to explicitly render intermediate reasoning steps, effectively simulating a human’s reflective thought process. Meanwhile, SHAP explainability provides context or logic by quantifying how each semantic token or feature contributes to the reasoning, analogous to a human examining which factors most influenced their choices. Our analysis explores how various training methods influence their semantic comprehension and, consequently, their ability to communicate their reasoning transparently. We apply this approach to a specialized environmental and Earth science ranking dataset—characterized by user feedback and sparse annotations—and we assess whether LLM-generated rankings can not only reflect accurate semantic understanding but also transparently emulate human-like introspection.