Prompt Choice Is an Unreported Variable in Randomized Trials of Large Language Model Study Material

Large language models now produce summaries, flashcards, and practice questions for medical students, and randomized trials have begun to test this material. The prompt, the written instruction given to the model, shapes that material as much as the choice of model does. In a year-long randomized study in a neurology clerkship, summary quality on one model ranged from 50.9% to 76.9% on the study’s scoring rubric depending on the prompt alone. Yet the preprint does not include the prompt text. A trial that names the model but withholds the prompt does not describe a reproducible intervention. Seven reporting items would identify the intervention tested and document how its study material was generated. The items cover the prompt, the model version, how the prompt was selected, human editing, quality checks, source material, and how learners were trained to write prompts.