An LLM-Based Multi-Path Question Answering System with XGBoost Routing and Threshold-Based Refusal

In conventional question answering systems, general-purpose large language models (LLMs), despite their strong capabilities in language understanding and generation, exhibit notable limitations in scenarios with stringent factuality requirements. Their outputs often lack explicit evidential grounding, making them prone to hallucinations and inconsistent responses. Moreover, LLMs do not inherently guarantee determinism when performing operations over structured data—such as aggregation, conditional filtering, and cross-field constraints—thereby undermining result consistency and reliability. To address these issues, we propose an internal-data-first framework for controllable question answering in high-risk scenarios. The framework categorizes knowledge sources into unstructured documents and structured data, enabling evidence-constrained generation via retrieval-augmented generation (RAG) and database-backed, verifiable query execution via restricted, read-only structured queries. In addition, an UNK branch is introduced as a safe degradation mechanism that triggers refusal when inputs lack sufficient evidence, exceed system capability boundaries, or fail to meet confidence requirements, thereby suppressing hallucinations and unauthorized generation. To enable controlled selection among the two execution pathways (RAG/SQL) and the safety degradation branch (UNK) at the system level, we design a learned router based on XGBoost with confidence-thresholded selective prediction, which preferentially activates UNK refusals for low-confidence or out-of-distribution inputs. We validate the proposed framework using a graduate admissions consultation system as an exemplar application, constructing both a document knowledge base and structured score tables, and conducting controlled comparisons across multiple system variants with multi-metric evaluations. Experimental results indicate that, under the current controlled evaluation setting, the proposed framework exhibits relatively stable behavior under complex query formulations and demonstrates practical engineering potential in high-risk vertical-domain question answering scenarios.