Can an LLM Agree with Itself? Repeatability and Reproducibility in Grant Screening

Large language models are increasingly being considered for grant proposal screening, yet their reliability has received less attention than their agreement with human reviewers. This case study evaluates the repeatability and reproducibility of direct textual interpretation by DeepSeek-V4-Flash-0731 on 1,026 applications to a 2026 science and technology project program. Seven categorical questions were evaluated under five conditions spanning three temperatures and three prompt versions, with ten runs per application and 51,300 model calls in total. Reliability was measured as pairwise response disagreement at the application level. Cross-condition drift was adjusted for each condition's intrinsic stochastic drift. At temperature 0, mean intrinsic drift was 8.87% across questions, and only 15.8% of applications were fully stable on all seven questions. Varying temperature from 0 to 0.6 added no detectable drift beyond the model's intrinsic behavior. Semantic prompt reformulation produced effects more than an order of magnitude larger and shifted response probabilities by over 50 percentage points for some questions. In one limiting case, adding a single comma changed the share of affirmative responses from 31.8% to 48.8% without changing any words. These findings suggest that even direct textual interpretation should be delegated to an LLM only after questions have been decomposed, carefully worded, and empirically tested for reproducibility.