Fine-tuning LLMs to Extract Epilepsy Seizure Frequency Data from Health Records

We developed a new methodology of extracting the frequency of a patient's epilepsy seizures from unstructured, freetext outpatient clinic letters by: first, devising a singular unit of measurement for seizure frequency; and second, fine-tuning a generative Large Language Model (LLM) on our bespoke annotated dataset.We measured frequency by the number of seizures per month: one seizure or more requires an integer; and less than one a decimal.This approach enables us to track whether a patient's seizures are improving or not over time.We found fine-tuning improves the F1 score of our bestperforming LLM, Ministral-8B-Instruct-2410, by around three times compared to an untrained model.We also found Ministral demonstrated an impressive ability for mathematical reasoning.