Improving predictive accuracy for clinical outcomes with LLM text embeddings for open-ended free-text responses: A case study on reports of psychedelic experiences

Clinical studies regularly ask participants to describe their experiences in their own words. These open-ended questions are included so that participants can report what they themselves consider relevant, and ideally researchers want to harvest this information. In practice however, these responses are rarely analyzed and often effectively discarded, since translating them into a useful quantitative format for further analysis traditionally required trained coders or advanced technical skills. Current transformer-based language models offer an appealing alternative option to quantify text: they internally represent any text input as high-dimensional numerical representations called text embeddings. Fromavailable open-weight models, these text embeddings can be freely extracted on a local machine. As text embeddings are just vectors of numbers, they can be plugged as predictors into standard statistical analyses. Here, we applied this to three large datasets in which participants described their experiences in the context of psychedelic substance use. We investigated whether text embeddings of these free-text descriptions improve cross-validated, held-out predictions of clinically relevant outcomes (well-being, change in well-being, change in psychological flexibility, post-traumatic symptom severity, post-traumatic growth) after we control for a range of other predictors capturingdemographic information, the setting of the experience, and a series of questionnaire data. We found reliable increases in predictive accuracy for three of the five outcomes (change in well-being and psychological flexibility, post-traumatic symptom severity), in two of the three datasets. Some of these increases are substantial. We therefore recommend (where possible) the inclusion of open-ended questions in any questionnaire, and to actually pass this information to analysis, to efficiently capture valuable information that would otherwise be lost.

Authors

Publication Details

Journal
PsyArXiv (OSF Preprints)
Published
2026-11-18
DOI
https://doi.org/10.31234/osf.io/e9wzk_v1
Primary Topic
Psychedelics and Drug Studies
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Improving predictive accuracy for clinical outcomes with LLM text embeddings for open-ended free-text responses: A case study on reports of psychedelic experiences

Ulrike Lueken, Abdo Uyar, Ricarda Evens, Fritz Guenther et al.
2 links
PsyArXiv (OSF Preprints)
Psychedelics and Drug Studies
preprint

Improving predictive accuracy for clinical outcomes with LLM text embeddings for open-ended free-text responses: A case study on reports of psychedelic experiences

Ulrike Lueken, Abdo Uyar, Ricarda Evens, Fritz Guenther, Max Wolff, Mathis Parent-Couture, Klara Wieczorke
preprint en

Abstract

Clinical studies regularly ask participants to describe their experiences in their own words. These open-ended questions are included so that participants can report what they themselves consider relevant, and ideally researchers want to harvest this information. In practice however, these responses are rarely analyzed and often effectively discarded, since translating them into a useful quantitative format for further analysis traditionally required trained coders or advanced technical skills. Current transformer-based language models offer an appealing alternative option to quantify text: they internally represent any text input as high-dimensional numerical representations called text embeddings. Fromavailable open-weight models, these text embeddings can be freely extracted on a local machine. As text embeddings are just vectors of numbers, they can be plugged as predictors into standard statistical analyses. Here, we applied this to three large datasets in which participants described their experiences in the context of psychedelic substance use. We investigated whether text embeddings of these free-text descriptions improve cross-validated, held-out predictions of clinically relevant outcomes (well-being, change in well-being, change in psychological flexibility, post-traumatic symptom severity, post-traumatic growth) after we control for a range of other predictors capturingdemographic information, the setting of the experience, and a series of questionnaire data. We found reliable increases in predictive accuracy for three of the five outcomes (change in well-being and psychological flexibility, post-traumatic symptom severity), in two of the three datasets. Some of these increases are substantial. We therefore recommend (where possible) the inclusion of open-ended questions in any questionnaire, and to actually pass this information to analysis, to efficiently capture valuable information that would otherwise be lost.

PsyArXiv (OSF Preprints)
Good health and well-being
Psychedelics and Drug Studies
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon