From discharge alerts to follow-up work: design and retrospective evaluation of a local, human-in-the-loop AI agent for primary care

Objective: Primary care practices learn about their patients' hospital discharges from payer alerts, but turning each alert into timely follow-up is manual work, and the hospital's discharge summary often does not reach the practice. We designed an AI agent that runs on a practice's own hardware and evaluated its components retrospectively on de-identified hospital data. Materials and Methods: For each discharge, deterministic code scores readmission risk (LACE) and sets the follow-up dates. A local language model extracts structured facts from the discharge summary, when one exists, and drafts a coordinator call script and a provider brief; by design a person approves them, and this evaluation stopped before that step. On MIMIC-IV, we evaluated the risk tiers against 30-day unplanned readmission (an admission-type proxy) in 64,036 stays; extraction in 200 discharge summaries, comparing prompted Qwen3 8B and 14B models with QLoRA fine-tunes trained on rule-generated or teacher labels; and drafting in 195 discharges, with automated checks. Both extraction references started from the rule-generated labels: an author-corrected diagnosis reference (51 notes) and a language-model-corrected reference for all fields (200 notes). Results: High-risk discharges were readmitted about twice as often as the rest (28.3% against 14.3%; area under the receiver operating characteristic curve [AUROC] for LACE, 0.638). Against the diagnosis reference, the fine-tuned models scored highest (micro F1 0.93-0.94, against 0.69-0.72 for prompted models zero-shot). Against the all-field reference, prompted models zero-shot scored 0.70-0.72 and the fine-tunes 0.58-0.60, similar to the rule-generated labels themselves (0.58), and the fine-tunes matched 243-299 of the rules' 301 potentially false "started" medications (by note and drug-name key), against 22-46 for prompted models. Drafting took 16.8 s per discharge on one consumer GPU; every medication name in the drafts matched a record stand-in that included the model's own extraction, and 98.4% also appeared in the note text. Discussion: Deterministic rules can carry the time-critical scheduling while small local models handle text. Fine-tuning on rule-generated labels reproduced the labels' gaps and potential errors, and evaluation against those same labels concealed it. Conclusion: A local, human-in-the-loop follow-up agent is computationally feasible on commodity hardware. On these benchmarks a prompted base model was the better default extractor; fine-tuning should be limited to fields whose labels have been checked against a reference other than the training labels. Human review, the clinical accuracy of the drafts and operational benefit remain to be evaluated, and the agent has not been used in patient care.