Mindbench.ai: an actionable platform to evaluate the profile and performance of large language models in a mental healthcare context
Individuals are increasingly utilizing large language model (LLM)-based tools for mental health guidance and crisis support in place of human experts. While AI technology has great potential to improve health outcomes, insufficient empirical evidence exists to suggest that AI technology can be deployed as a clinical replacement; thus, there is an urgent need to assess and regulate such tools. Regulatory efforts have been made and multiple evaluation frameworks have been proposed, however,field-wide assessment metrics have yet to be formally integrated. In this paper, we introduce a comprehensive online platform that aggregates evaluation approaches and serves as a dynamic online resource to simplify LLM and LLM-based tool assessment: MindBench.ai. At its core, MindBench.ai is designed to provide easily accessible/interpretable information for diverse stakeholders (patients, clinicians, developers, regulators, etc.). To create MindBench.ai, we built off our work developing MINDapps.org to support informed decision-making around smartphone app use for mental health, and expanded the technical MINDapps.org framework to encompass novel large language model (LLM) functionalities through benchmarking approaches. The MindBench.ai platform is designed as a partnership with the National Alliance on Mental Illness (NAMI) to provide assessment tools that systematically evaluate LLMs and LLM-based tools with objective and transparent criteria from a healthcare standpoint, assessing both profile (i.e. technical features, privacy protections, and conversational style) and performance characteristics (i.e. clinical reasoning skills). With infrastructure designed to scale through community and expert contributions, along with adapting to technological advances, this platform establishes a critical foundation for the dynamic, empirical evaluation of LLM-based mental health tools-transforming assessment into a living, continuously evolving resource rather than a static snapshot.
Authors
- Jake Linardon (ORCID: https://orcid.org/0000-0003-4475-7139)
- Andrea Cipriani (ORCID: https://orcid.org/0000-0001-5179-8321)
- Asim H. Gazi (ORCID: https://orcid.org/0000-0003-4698-2823)
- Julian Herpertz (ORCID: https://orcid.org/0009-0003-8416-2970)
- Jason Bantjes (ORCID: https://orcid.org/0000-0002-3626-9883)
- Khai N. Truong (ORCID: https://orcid.org/0000-0003-0774-5964)
- Joseph Firth (ORCID: https://orcid.org/0000-0002-0618-2752)
- John Blake Torous (ORCID: https://orcid.org/0000-0002-5362-7937)
- Carolyn I. Rodríguez (ORCID: https://orcid.org/0000-0001-6697-1692)
- Laura Ospina‐Pinillos (ORCID: https://orcid.org/0000-0002-7929-1511)
- Jiaee Cheong
- Soumya Choudhary (ORCID: https://orcid.org/0000-0002-7522-6923)
- Margaret R. Emerson (ORCID: https://orcid.org/0000-0002-1103-8800)
- Ken Duckworth (ORCID: https://orcid.org/0000-0002-8303-0058)
- Akane Sano (ORCID: https://orcid.org/0000-0003-4484-8946)
- Allison G. Dempsey (ORCID: https://orcid.org/0000-0003-2331-9455)
- Xuhai Xu (ORCID: https://orcid.org/0000-0001-5930-3899)
- Bryce Hill
- Julian Schwarz (ORCID: https://orcid.org/0000-0001-7306-7909)
- Michelle A. Patriquin (ORCID: https://orcid.org/0000-0002-1910-6724)
- Martin P. Paulus (ORCID: https://orcid.org/0000-0002-0825-3606)
- Darlene King
- Yining Hua (ORCID: https://orcid.org/0000-0002-3377-6943)
- Gillian Strudwick (ORCID: https://orcid.org/0000-0002-1080-7372)
- C.F. Stromeyer (ORCID: https://orcid.org/0009-0006-5957-9258)
- Carla Gorban (ORCID: https://orcid.org/0009-0003-9568-1792)
- Matthew Flathers (ORCID: https://orcid.org/0009-0004-3351-0688)
- Steven Siddals (ORCID: https://orcid.org/0009-0007-0514-8978)
- Bridget Dwyer
- Michael B. Wood
- Eden Rozenblit (ORCID: https://orcid.org/0009-0000-0487-5216)
- Daniel H. Gillison
- Steven Scheuller
Institutions
- The University of Sydney (AU)
- Beth Israel Deaconess Medical Center (US)
- Centre for Addiction and Mental Health (CA)
- Harvard University (US)
- Deakin University (AU)
- Baylor College of Medicine (US)
- University of California, Irvine (US)
- Pontificia Universidad Javeriana (CO)
- Stellenbosch University (ZA)
- Manchester Academic Health Science Centre (GB)
- University of Manchester (GB)
- National Alliance on Mental Illness (US)
- National Institute for Health and Care Research (GB)
- University of Oxford (GB)
- Native Health (US)
- Oxford BioMedica (United Kingdom) (GB)
- Laureate Institute for Brain Research (US)
- Franklin University (US)
- University of Nebraska Medical Center (US)
- National Institute of Mental Health and Neurosciences (IN)
- Rice University (US)
- Columbia University (US)
- Charité - Universitätsmedizin Berlin (DE)
- The University of Texas Southwestern Medical Center (US)
- Stanford University (US)
Publication Details
- Journal
- NPP—Digital Psychiatry and Neuroscience
- Published
- 2025-11-14
- DOI
- https://doi.org/10.1038/s44277-025-00049-6
- Citations
- 2
- Primary Topic
- Digital Mental Health Interventions
- Type
- article
- Field-Weighted Citation Impact
- 3.09
Funders
- Argosy Foundation
- Wellcome Trust
- National Institute for Health and Care Research
- Department of Health and Social Care
- National Institutes of Health