Evidence-grounded biomedical classification via retrieval-augmented BioGPT with LoRA fine-tuning and post-hoc calibration

Abstract Background Answering yes/no/maybe research questions from biomedical literature is difficult for contemporary language models, which tend toward overconfidence on inconclusive evidence and toward majority-class predictions on imbalanced data. Methods We present a retrieval-augmented pipeline for tri-class (yes/no/maybe) biomedical question answering on the full PubMedQA PQA-Labeled dataset (1,000 expert-annotated questions), combining PubMedBERT dense retrieval, LoRA fine-tuning of BioGPT-Large with focal loss and per-class weighting, two-dimensional post-hoc log-probability calibration, and a five-layer clinical safety framework. We evaluate under two evidence conditions: an oracle-context setting in which the model receives each question's true source abstract, and a retrieved-context setting in which the model receives only what the retriever actually returns from a 20,000-passage corpus, reflecting real deployment. We further validate with 5-fold cross-validation across all 1,000 labeled records, bootstrap confidence intervals, McNemar's test, and an ablation study retrained under identical hyperparameters. Results Five-fold stratified cross-validation across all 1,000 PQA-Labeled records, the primary robustness analysis in this study, gave 58.9% +/- 8.1% raw accuracy, macro-F1 of 0.478 +/- 0.080, and MCC of 0.319 +/- 0.123; accuracy-oriented calibration raised mean accuracy to 63.0% but did not consistently improve macro-F1 or MCC across folds. As a secondary analysis on a fixed 151-sample oracle-context held-out split, the best single-split configuration reached 73.51% accuracy (95% CI 66.2-80.1%), macro-F1 of 0.549, and MCC of 0.509, illustrating that a favourable single split can look substantially stronger than the cross-validated estimate. With retrieved context, raw accuracy is 54.3% (95% CI 46.4-61.6%) and MCC is 0.144, approximately 13 points of accuracy below the oracle-context ceiling; because a repeat run of the identical retrieved-context configuration reached 68.21% accuracy, this gap should be read as an association with the evidence condition rather than a precise, causally-isolated estimate of retrieval loss. Ablations trained under identical conditions nonetheless show removing retrieved/gold context entirely causes the largest single drop in performance of any component tested, which is a more robust signal that evidence quality matters than the single oracle-vs-retrieved point estimate. Conclusions Reporting oracle-context and retrieved-context performance separately, together with cross-validated variance and the observed run-to-run training variability, gives a materially more honest picture of system performance than a single held-out split. The results indicate that evidence quality is an important factor in this pipeline's performance, although substantial run-to-run training variability prevents us from attributing the oracle-versus-retrieved performance gap exclusively to retrieval quality.