ChatGPT Access and Writing Skill: A Bayesian Reanalysis of an Online Randomized Experiment

Generative writing tools are entering professional work, and their effects differ across people. I reanalyze a published online randomized experiment in which college-educated professionals in six writing-intensive occupations completed two incentivized writing tasks, and a randomly assigned half were given access to ChatGPT before the second task (Noy & Zhang, 2023a). I ask whether access weakens the link between first-task and second-task writing grades, whether that weakening is stronger for writing than for content, whether access narrows differences between people more for originality than for writing quality, and how access relates to felt effectiveness and enjoyment on the task. Grades and ratings are standardized; the analysis plan, fixed beforehand, retests hypotheses already tested on these data. Bayesian regression models with weakly informative priors are fit to the linear and convex incentive conditions. Each estimate is a posterior median with a 95% equal-tailed credible interval and the posterior probability of the predicted direction, P(direction), with no decision cutoffs. Under access, second-task writing grades depended less on first-task grades, −0.159 [−0.317, −0.002], with P(direction) = .976; the data did not distinguish content from writing interactions, or originality from writing spread ratios. Access was associated with higher enjoyment (0.44 [0.29, 0.60]); felt effectiveness was 0.12 [−0.06, 0.29], P(direction) = .905. After a disclosed code correction, two-month workplace use of ChatGPT, the secondary outcome, was 13.4 [3.0, 23.5] percentage points higher under access. For organizations, results suggest one structured encounter was associated with higher enjoyment and followed by more use at work.