Extracting social support and social isolation information from clinical psychiatry notes: comparing a rule-based natural language processing system and a large language model
Unexpectedly, the RBS outperformed the LLMs across all metrics. An intensive review demonstrates that this finding is due to the divergent approach taken by the RBS and LLM. The RBS was designed and refined to follow the same specific rules as the gold-standard annotations. Conversely, the LLM was more inclusive with categorization and conformed to common English-language understanding. Both approaches offer advantages, although additional replication studies are warranted.
Authors
- Praneet Kasi Reddy Jagadeesh Kumar
- Myrna M. Weissman (ORCID: https://orcid.org/0000-0003-3490-3075)
- Mark Olfson (ORCID: https://orcid.org/0000-0002-3958-5662)
- Mohit Sharma (ORCID: https://orcid.org/0000-0003-0091-5510)
- Lauren Lepow (ORCID: https://orcid.org/0000-0002-1919-6897)
- Braja Gopal Patra (ORCID: https://orcid.org/0000-0003-2997-5314)
- Ardesheer Talati (ORCID: https://orcid.org/0000-0001-5708-1372)
- Jorge A. Sánchez-Ruiz (ORCID: https://orcid.org/0000-0003-4282-7359)
- Girish N. Nadkarni (ORCID: https://orcid.org/0000-0001-6319-4314)
- Joanna M. Biernacka (ORCID: https://orcid.org/0000-0001-9350-4440)
- Jyotishman Pathak (ORCID: https://orcid.org/0000-0002-4856-410X)
- Euijung Ryu (ORCID: https://orcid.org/0000-0001-6281-8738)
- J. John Mann (ORCID: https://orcid.org/0000-0002-0265-6586)
- Veer Vekaria (ORCID: https://orcid.org/0000-0001-9801-2250)
- Isotta Landi (ORCID: https://orcid.org/0000-0003-4212-4709)
- Brian Fennessy (ORCID: https://orcid.org/0000-0002-6222-6190)
- Alexander W. Charney (ORCID: https://orcid.org/0000-0001-8135-6858)
- Prakash Adekkanattu (ORCID: https://orcid.org/0000-0003-1125-1449)
- Yiye Zhang (ORCID: https://orcid.org/0000-0003-3494-2699)
- Gavin Hynes
Institutions
- Mayo Clinic (US)
- Cornell University (US)
- Columbia University Irving Medical Center (US)
- WinnMed (US)
- Mayo Clinic in Florida (US)
- Weill Cornell Medicine (US)
- New York State Psychiatric Institute
- Columbia University (US)
- Icahn School of Medicine at Mount Sinai (US)
Publication Details
- Journal
- Journal of the American Medical Informatics Association
- Published
- 2024-10-18
- DOI
- https://doi.org/10.1093/jamia/ocae260
- Citations
- 20
- Primary Topic
- Health disparities and outcomes
- Type
- article
- Field-Weighted Citation Impact
- 10.56
Funders
- National Institutes of Health