The Wisdom of Artificial Crowds: Testing Self-Consistency Aggregation using Distributional Analysis on Output Samples

Self-consistency applied to Large Language Models has demonstrated significant improvement in LLM reasoning. A number of techniques currently exist to expand upon the improvement of self-consistency, including variation of temperature and Top-k sampling. However, utilizing distributional patterns among samples to perform self-consistency has not yet been explored. In this work, we use a variety of statistical aggregation techniques to perform self-consistency on samples of varying temperature and top-k and evaluate performance on an LLM estimation benchmark. We found that comparable performance to standard “majority vote” self-consistency can be achieved using clustering, deduplication, offsetting predicted error, and even weighting samples to break ties. This demonstrates potential avenues for further improvement in self-consistency and is likely of interest to groups improving LLM reasoning.