Benchmarking Local Large Language Models for Predicting Vestibular Dysfunction Based on Caloric Testing: A Comparison with Conventional Machine Learning on Stabilometry Data

Background/Objectives: In-context learning enables large language models (LLMs) to perform clinical prediction without task-specific training; however, their performance on structured tabular data relative to conventional machine learning remains unclear. This exploratory study evaluated the performance, scalability, and efficiency of local LLMs for predicting caloric-test-defined vestibular dysfunction from stabilometry data. Methods: We analyzed stabilometric data from 238 participants (75 with vestibular dysfunction, 163 controls). Forty-three local LLMs spanning multiple families were evaluated using in-context learning on a local setup using CPU-dominant inference with partial GPU offloading, without external data transmission. Performance and inference time were measured via repeated hold-out validation and compared with logistic regression (LR) and tree-based ensemble methods. Results: The tree-based ensemble models showed the highest predictive performance. LLM performance was heterogeneous and varied by model family and size. Mid-sized models achieved accuracy comparable to LR, although their performance was not statistically distinguishable from either LR or the majority-class baseline, whereas smaller models frequently failed to produce usable binary outputs. Larger models did not consistently provide proportional gains relative to their computational cost. Conclusions: Local LLMs did not outperform conventional machine learning for maximizing predictive accuracy on this small tabular dataset. However, they may serve as a complementary, privacy-preserving approach that enables deployment without task-specific training. Future work should validate these findings with larger datasets, more robust validation, standardized outcome definitions, and constrained decoding.