The analytic multiverse of AI-assisted directed content analysis: a framework, open tools, and a screening study

Background. Large language models (LLMs) are increasingly used to apply predefined codebooks to qualitative data, and studies of this practice usually report agreement with human coding for a single configuration. However, each configuration rests on many defensible choices (e.g., how the task is worded, how the codebook is presented, and how agreement is scored), and the extent to which results depend on these choices remains relatively unknown.Methods. I describe the analytic multiverse of AI-assisted directed content analysis (i.e., the set of defensible configurations for producing and scoring codes), define what a multiverse study should report, propose a staged design for exploring it, and release two open-source tools, QualFrame and QualBench, that record each choice as a setting and score every run. In a screening study, Claude Haiku 5.5 and GPT-6 Luna coded 40 open-access smoking-cessation trial reports for 94 behaviour change techniques under a half-fraction factorial design that varied task instructions, codebook division, reasoning effort, instruction wording, and code order, with repeat runs of one configuration. Agreement with historical Human Behaviour-Change Project annotations was measured with the Matthews correlation coefficient (MCC). Exploratory analyses examined majority votes and earlier runs of four models on 50 reports excluded from screening.Results. Across 32 configurations (38 runs; recorded API spend US$13.66), MCC ranged from 0.24 to 0.57. Four runs of one configuration varied by at most 0.02 in MCC; however, of the codes applied by at least one of these runs, 43% to 64% were not applied by all four. Within this design, the contrasts for task instructions, reasoning effort, and codebook division each exceeded the average difference between the two models, and their effects depended on one another and on the model; for example, asking the model to decide on every code raised MCC by 0.10 with the whole codebook but by 0.02 with the codebook divided. The apparent cost of dividing the codebook fell from 0.045 to 0.011 when scoring was restricted to the 12 techniques the annotators prioritized, and balanced accuracy ranked a different configuration best. Earlier runs of four models with similar agreement on the screening reports diverged on the 50 excluded reports, where their majority vote had agreement similar to the best model.Conclusions. Agreement with a reference depended on configuration choices, their interactions, the scoring scope, and the reports scored. Studies of AI-assisted coding should report the configurations they explored and test whether their conclusions survive these choices. To build a cumulative evidence base, I propose a research agenda that others can take up independently with the open tools and a common reporting format.