Objective Assessment of Medication Adherence Through Automated Pill Detection Using a Computer Vision Framework: Development and Evaluation Study

Abstract Background Medication adherence remains a significant concern in both clinical practice and public health. Nonadherence to prescribed medication regimens is associated with poorer health outcomes, higher rates of hospitalization, and increased financial burdens on health care systems. Despite its critical role in ensuring treatment efficacy, adherence assessment is still largely dependent on patient self-reports, pharmacy refill records, or caregiver observations—methods that are often subjective, inconsistent, and unreliable. Therefore, there is an urgent need for objective, automated solutions to accurately monitor medication, particularly the information on the pills taken and intake behavior. AI techniques, especially computer vision, are promising solutions; however, their application in assessing medication adherence has several key pitfalls in maintaining robustness under environmental variation that limit their reliability and real-world applicability. Objective This study aimed to develop a reliable and accurate computer vision framework for pill detection and counting using mobile phone images captured under diverse real-world conditions to support medication adherence assessment. Methods This study is carried out using a public dataset on pill detection (N=152) and a curated dataset (N=60) on medication adherence assessment. Inspired by mixture of experts (MoE), we propose and implement a robust pill detection framework that integrates two expert-level object detection models, You Only Look Once (YOLOv12) and Faster Region-based Convolutional Neural Network (Faster R-CNN), with a Segment Anything Model (SAM). The pill counting performance was evaluated using mean absolute error (MAE) and accuracy. We further conduct a pilot study on a curated dataset that mimics a subject taking medicine for a period of 20 days. We evaluate sensitivity and specificity in medication adherence assessment. Results The proposed multimodel framework substantially reduces both error types, achieving a low MAE of 0.167 on the Kaggle test set in pill counting and an accuracy of 0.933 in pill detection. In medication adherence simulation, the proposed framework achieves high accuracy (0.917), sensitivity (0.933), and specificity (0.900), significantly outperforming the evaluated YOLOv12 and Faster R-CNN baselines ( P <.05). Performance remained robust on the self-collected dataset in medication-taking scenarios in the settings of household, restaurant, office, and outdoor environments. Conclusions The proposed framework achieved high performance in pill counting, detection, and simulated medication-adherence classification. This proof-of-concept study has demonstrated the potential to enhance medication adherence among patients and support more reliable treatment monitoring.