Improving predictive accuracy for clinical outcomes with LLM text embeddings for open-ended free-text responses: A case study on reports of psychedelic experiences
Clinical studies regularly ask participants to describe their experiences in their own words. These open-ended questions are included so that participants can report what they themselves consider relevant, and ideally researchers want to harvest this information. In practice however, these responses are rarely analyzed and often effectively discarded, since translating them into a useful quantitative format for further analysis traditionally required trained coders or advanced technical skills. Current transformer-based language models offer an appealing alternative option to quantify text: they internally represent any text input as high-dimensional numerical representations called text embeddings. Fromavailable open-weight models, these text embeddings can be freely extracted on a local machine. As text embeddings are just vectors of numbers, they can be plugged as predictors into standard statistical analyses. Here, we applied this to three large datasets in which participants described their experiences in the context of psychedelic substance use. We investigated whether text embeddings of these free-text descriptions improve cross-validated, held-out predictions of clinically relevant outcomes (well-being, change in well-being, change in psychological flexibility, post-traumatic symptom severity, post-traumatic growth) after we control for a range of other predictors capturingdemographic information, the setting of the experience, and a series of questionnaire data. We found reliable increases in predictive accuracy for three of the five outcomes (change in well-being and psychological flexibility, post-traumatic symptom severity), in two of the three datasets. Some of these increases are substantial. We therefore recommend (where possible) the inclusion of open-ended questions in any questionnaire, and to actually pass this information to analysis, to efficiently capture valuable information that would otherwise be lost.
Authors
- Ulrike Lueken (ORCID: https://orcid.org/0000-0003-1564-4012)
- Abdo Uyar
- Ricarda Evens (ORCID: https://orcid.org/0000-0002-2834-2171)
- Fritz Guenther
- Max Wolff (ORCID: https://orcid.org/0000-0001-6896-9633)
- Mathis Parent-Couture
- Klara Wieczorke (ORCID: https://orcid.org/0009-0006-8169-9424)
Publication Details
- Journal
- PsyArXiv (OSF Preprints)
- Published
- 2026-11-18
- DOI
- https://doi.org/10.31234/osf.io/e9wzk_v1
- Primary Topic
- Psychedelics and Drug Studies
- Type
- preprint