ProactiMate: evaluating LLM-based chatbots for behavior change interventions

This paper explores the role of Large Language Models (LLMs) in promoting sustainable behavior, specifically in overcoming procrastination. Despite widespread recognition of the need for sustainable behavior change, individuals often struggle to break free from entrenched, unsustainable habits. LLMs, such as OpenAI’s GPT-4, represent a significant breakthrough in artificial intelligence and are increasingly used in behavior change interventions. This study introduces ProactiMate, a chatbot built using Motivational Interviewing (MI) principles and a Chain of Models approach for prompt engineering, designed to help users combat procrastination. Our research compares four LLMs (GPT-3.5 Turbo, LLaMA-3.2, Qwen-2.5, and SmolLM-1.7B) for output influence on procrastination avoidance, and assesses the impact of hyperparameters (temperature and top-p values) on procrastination avoidance. The findings reveal that GPT-3.5 outperforms other models across various evaluation metrics, and higher temperature and top-p values lead to more effective procrastination avoidance from automatic evaluation. According to expert evaluations, Qwen-2.5 and GPT-3.5 Turbo demonstrated notable effectiveness in fostering user engagement and motivation for addressing procrastination, with GPT-3.5 Turbo particularly distinguished by its capacity to provide strategies that help maintain long-term motivation. And GPT’s output aligns well with both automatic evaluation metrics and human evaluation. The results provide insights into the most effective ways to use LLMs in chatbot design, offering solutions for future usability testing.

Authors

Institutions

Publication Details

Journal
Scientific Reports
Published
2026-11-19
DOI
https://doi.org/10.1038/s41598-026-79399-w
Citations
3
Primary Topic
Digital Mental Health Interventions
Type
article
Field-Weighted Citation Impact
5.27
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
article

ProactiMate: evaluating LLM-based chatbots for behavior change interventions

Nina Dethlefs, Ben Chen
3 citations
Scientific Reports
Digital Mental Health Interventions
5.27
article

ProactiMate: evaluating LLM-based chatbots for behavior change interventions

Nina Dethlefs, Ben Chen
article en
3 citations

Abstract

This paper explores the role of Large Language Models (LLMs) in promoting sustainable behavior, specifically in overcoming procrastination. Despite widespread recognition of the need for sustainable behavior change, individuals often struggle to break free from entrenched, unsustainable habits. LLMs, such as OpenAI’s GPT-4, represent a significant breakthrough in artificial intelligence and are increasingly used in behavior change interventions. This study introduces ProactiMate, a chatbot built using Motivational Interviewing (MI) principles and a Chain of Models approach for prompt engineering, designed to help users combat procrastination. Our research compares four LLMs (GPT-3.5 Turbo, LLaMA-3.2, Qwen-2.5, and SmolLM-1.7B) for output influence on procrastination avoidance, and assesses the impact of hyperparameters (temperature and top-p values) on procrastination avoidance. The findings reveal that GPT-3.5 outperforms other models across various evaluation metrics, and higher temperature and top-p values lead to more effective procrastination avoidance from automatic evaluation. According to expert evaluations, Qwen-2.5 and GPT-3.5 Turbo demonstrated notable effectiveness in fostering user engagement and motivation for addressing procrastination, with GPT-3.5 Turbo particularly distinguished by its capacity to provide strategies that help maintain long-term motivation. And GPT’s output aligns well with both automatic evaluation metrics and human evaluation. The results provide insights into the most effective ways to use LLMs in chatbot design, offering solutions for future usability testing.

Scientific Reports
Loughborough University (GB)
Openalex Percentile: Top 6%
Digital Mental Health Interventions
5.27
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon