Identifying prevalence of fertility discussions using a natural language processing like key term indicator.

625 Background: Clinical guidelines recommend that providers discuss risks to fertility before initiating cancer treatment. Unfortunately, it remains challenging to determine the prevalence of fertility discussions because they are not included as discrete data in the electronic health record (EHR). To inform ongoing quality improvement efforts, this study aimed to evaluate the accuracy of a natural language processing (NLP)-like key term indicator (KTI) to identify the prevalence of fertility discussions and compared it to three discrete EHR data elements: fertility preservation counseling billing code (ICD-10-CM Z31.62), oocyte retrieval procedure code (CPT 58970), and referral to a fertility specialist. Methods: We developed an NLP-like KTI to identify clinical notes that include key terms related to fertility (“egg”, “embryo”, “fertility”, “oocyte”, “rei”, “reproductive endocrinology”, “sperm”). The KTI was applied to all notes within 1 month of first pediatric or adult outpatient cancer visit at the UCLA Jonsson Comprehensive Cancer Center for patients of reproductive age (females age 12-45, males age 12-55) over 1-year (4/1/2023-3/31/2024). True prevalence was determined by chart review of a stratified random sample of eligible patients. Sensitivity and specificity were compared across all measures. Results: 9041 patients met eligibility criteria. A stratified random sample (by sex and visit month) of 240 patients were identified; 3 were excluded (2 test accounts and 1 who was never seen.) 98/237 (41.4%) patients had fertility discussions confirmed by chart abstraction. Only 62/98 (63.3%) had documented discussions before receiving systemic therapy. The KTI identified 56/237 (23.6%) patients. Fertility counseling billing code, oocyte retrieval procedure code, and fertility specialist referral were identified in 6/237 (2.5%), 2/237 (0.8%), and 9/237 (3.8%) patients. Sensitivity and specificity of the KTI were 46% and 92%, which were higher and lower respectively than the other EHR-derived metrics (Table 1). Conclusions: Our KTI identified fertility discussions with low sensitivity (46%) and high specificity (92%). While the KTI performed better than other EHR-derived measures, we anticipate further iteration with the inclusion of additional fertility terms and structured data elements will improve the sensitivity of this surrogate marker. Sensitivity and specificity of surrogate measures for fertility discussion documentation. KTI Billing Code Procedure Code Specialist Referral Sensitivity 0.46 0.06 0.03 0.06 Specificity 0.92 1.00 1.00 0.98