Sticks And Stones... Gender Bias In Large Language Model-generated Patient Education For Anterior Cruciate Injury

As popularity of large language models (LLM) increases, patients may seek patient education materials (PEM) from these sources. Anterior cruciate ligament (ACL) injury predominantly affects young individuals who participate in soccer. PURPOSE: The purpose of this study is 1) to explore gender bias in LLM-generated ACL PEMs, and 2) to assess the accuracy and consistency of the information provided. METHODS: Four LLMs provided PEMs on ACL injury. Readability was assessed via the Flesch-Kincaid Grade Level. The Patient Education Materials Assessment Tool for Printable Materials (PEMAT-P) assessed understandability and actionability. Further analysis using the Linguistic Inquiry and Word Count dictionary assessed gendered-language differences. Finally, a qualitative analysis assessed accuracy and robustness of responses. RESULTS: Sixteen LLM-generated responses were analyzed. The mean Flesch-Kincaid Grade Level was 10.9 ± 1.32 for the Girl persona, 11.35 ± 1.52 for the Boy persona, 10.03 ± 1.04 for the Gender-fluid persona, and 10.13 ± 0.43 for the Control persona. Sixteen (100%) and 14 (88%) responses scored above the 70% adequate PEMAT-P threshold for understandability and actionability, respectively. Mean understandability and actionability scores were 80% ± 8.16% (range 70-90) and 68.8% ± 12.5% (range 50-75) for the Girl persona, 85% ± 5.77% (range 80-90) and 75% ± 0% for the Boy persona, 82.23% ± 9.30% (range 70-90) and 75% ± 0% for the Gender-fluid persona, and 82.22% ± 9.30% (range 70-90) and 72.9% ± 4.15% (range 66.7-75) for the Control persona, respectively. Kruskal-Wallis test revealed no statistically significant differences in gendered-language between personas across the four LLMs (p > .05). Linguistic analysis revealed 75% of the Girl persona and 100% of the Boy persona responses had a negative tone. Extrinsic risk factors were included in only seven (44%) responses and sociocultural risk factors or diagnostic details were included in no responses. CONCLUSION: While LLM-generated PEMs were adequately understandable, showed no gender bias, and provided actionable information, the high readability level may limit generalizability. Furthermore, crucial information was lacking from the PEMs and the overall negative tone may unintentionally increase patient fear.