Large Language Models and Creative Content Design: a case study of email marketing at Wine Access

A sequence of three randomized controlled trials (RCTs) is conducted to support a small online business’ decision of whether and how to implement AI in the creation of email marketing content. Recent developments in frequentist statistical decision theory are used to accommodate small samples available for testing in the small-business setting. The RCTs comprise three test policy cells with email content created by (i) salaried writers (“human”), (ii) a large-language model (“LLM”), and (iii) a “hybrid” combination of a human editing the content created by the LLM, respectively. When a “no email” control policy is included, all three test cells approximately double the gross profits from orders relative to the control cell. The RCTs vary whether the hybrid cell is edited by a salaried writer or the marketing team. The LLM cells vary whether the AI is pre-trained using historic emails or uses a prompt-based generative pre-trained transformer (“GPT”). Decision theory always selects one of the AI cells over the standard human policy on the basis of total annual profit net of related labor and software overhead.