Numerical Consistency and Accuracy of Large Language Models in Risk Communication for Cardiovascular Prevention in Older Adults: A Cross-Sectional Multi-Model Paired Study

Abstract Background: Cardiovascular disease remains the leading cause of morbidity and mortality worldwide, with its burden concentrated in older populations. In primary prevention, decisions to initiate pharmacological interventions such as statins depend on a shared understanding of benefits and harms. Large language models (LLMs) are increasingly used to convey such quantitative information, yet existing evaluations have focused almost exclusively on accuracy. Objective: To evaluate the run-to-run consistency, accuracy, and cross-language consistency of mainstream LLMs when converting absolute benefit across different risk presentations in cardiovascular prevention scenarios for older adults. Methods: In this cross-sectional multi-model paired study, we constructed 80 synthetic scenarios of cardiovascular primary prevention. Each scenario was presented in five risk formats (relative risk reduction, pre- and post-treatment event rates, natural frequencies, risk ratio, and a direct-read positive control). Three LLMs (deepseek-v4-flash, doubao-seed-2.1-turbo, and kimi-k2.6) answered each item independently three times in both Chinese and English, yielding 7,200 responses collected through official application programming interfaces on 3–4 September 2026. Two independently implemented automated scorers (Python and R) graded all responses against a locked gold standard using precision-aware tolerance rules, and the two implementations agreed on every item. The primary outcome was the run-to-run complete agreement rate, defined as identical values for all four numerical outputs across three repeated runs of the same item group (positive controls excluded). Item groups answered correctly but inconsistently were analysed for the source of disagreement, and accuracy was reported as a secondary outcome. Between-model differences were compared using mixed-effects logistic regression with scenario as a random intercept. Results: Complete agreement rates were 88.4% for deepseek, 98.4% for doubao, and 98.6% for kimi. Adjusted marginal agreement rates were 93.7% (95% CI: 90.1%–96.0%), 99.4% (98.6%–99.8%), and 99.5% (98.8%–99.8%), respectively. deepseek was significantly less consistent than both comparators (OR = 0.086 and 0.072; both P < 0.001), whereas doubao and kimi did not differ (P = 0.946). Consistency was higher in the English arm than in the Chinese arm (OR ≈ 4.2, P < 0.001). Of the 89 correct-but-inconsistent item groups, 88 differed only in NNT or NNH by one (rounding instability), and one additionally showed an ARR tail difference of 0.01 percentage points. Malignant inconsistency (materially different values, fluctuating correctness, or runs with no extractable answer) accounted for 0.16% of item groups (3/1,920). Cross-language numerical agreement ranged from 92.2% to 99.1%, and McNemar tests detected no asymmetry in accuracy between languages (P = 0.48–1.00). The four-value fully correct rate ranged from 99.8% to 100%, indicating a ceiling effect. Conclusion: On this structured risk-communication task, the numerical conversion accuracy of current mainstream LLMs has approached the ceiling, and between-model differences lie primarily in run-to-run consistency. Correct-but-inconsistent responses arose almost entirely from rounding instability that does not alter clinical conclusions, and Chinese and English outputs were highly concordant. Evaluations of medical AI should incorporate run-to-run consistency, particularly the malignant inconsistency rate, alongside accuracy.