On the Uncertainty of LLMs over Heterogeneous Many-Core Systems

The deployment of large language models (LLMs) on heterogeneous many-core systems has the potential to significantly enhance computational efficiency and model performance. However, the inherent complexity of these systems introduces additional sources of uncertainty that can affect the reliability and accuracy of model predictions. This paper investigates the quantification of uncertainty in LLMs operating on heterogeneous many-core architectures. We propose a novel framework that leverages the parallel processing capabilities of many-core systems to systematically assess and mitigate uncertainty in model outputs. We first identify key tokens as those that carry significant attention weights and are linked to the hallucinations and repetitions commonly observed in LLMs. Based on this observation, we propose the Key Token-based Negative Log-Likelihood (KT-NLL) algorithm to compute uncertainty scores for LLM responses. This approach yields a notable enhancement of up to 22.4% in AUROC and 6.1% in PRAUC compared to existing baseline methods. Furthermore, we develop a Key Token-driven context refinement technique to address repetition issues, resulting in a performance boost of up to 50.3%.