AI Tutor, Online Course, or Both? A Randomized Experimental Pilot Study for Evaluation Capacity Building

Extension lacks literature comparing standalone generative artificial intelligence (AI) with structured professional development for capacity building. Guided by Social Cognitive Theory and Cognitive Load Theory, this study evaluated the effects of an asynchronous online course and a custom AI tutor on evaluation knowledge and self-efficacy among Cooperative Extension professionals. We conducted a randomized 2 × 2 factorial pilot study crossing access to an evaluation certificate course with access to a custom Generative Pre-trained Transformer (GPT). Of 144 registrants, 78 provided matched pre- and post-test data, with attrition differing across conditions. Data were analyzed using a three-way mixed factorial ANOVA. Course access and GPT access each increased objective knowledge, with post-test means around 91% across treatment groups. No statistically significant additive effect was detected for knowledge gain. However, given a possible ceiling effect and limited statistical power, this result is inconclusive. Course access and GPT access each increased self-efficacy, but the combined condition produced the smallest self-efficacy gain among treatment groups. Because these estimates are based on participants who completed the study, findings are preliminary. If replicated, results suggest that Extension should combine AI tutors and coursework only when each modality provides known complementary value.