Creating A Gold Standard Annotation Guideline for Electronic Health Records

Abstract Purpose: Large Language Models have captured the research environment with their potential to aid in understanding a multitude of natural language processing tasks. Often they require fine-tuning with corpora of relevant information. Electronic health records have an abundance of free-text that is dense with information and could be used for this purpose. This work aims to create a gold-standard annotated dataset with an associated set of guidelines for the annotation process. Methods: Using a dataset of 1500 electronic health records from epilepsy clinics, we created a set of annotation guidelines and a team of data scientists and clinicians annotated these accordingly. Results: We find an overall inter-annotator agreement of 0.769, as measured with Krippendorff's Alpha, and showed that the most annotated and agreed upon annotations are those related to anti-seizure medications and diagnoses of epilepsy and other comorbidities. Conclusion: This work provides a valuable resource for the development and evaluation of NLP systems for epilepsy and electronic health records.