Revisiting the Federalist Papers in the Era of Large Language Models

For a long time, the authorship of the Federalist Papers had been a subject of inquiry and debate, not only by linguists and historians but also by statisticians. In what was arguably the first Bayesian case study, Mosteller and Wallace (1963) provided the first statistical evidence for attributing all disputed papers to Madison. Our paper revisits this historical dataset through the lens of modern language models alongside classical statistical language models. We review some of the more widely used neural embedding-based models, which are often trained on massive, general-purpose corpora, and examine them from a statistical point of view in the context of text classification. We investigate whether, without any task-specific fine-tuning, these high-dimensional contextual embeddings can be useful for authorship attribution and to identify writing-style features. We contrast these with classical, likelihood-based models that produce low-dimensional, interpretable representations grounded in explicit probabilistic structure. Contrary to our expectations, we find that extremely high-dimensional contextual embeddings do not necessarily outperform interpretable, lower-dimensional topic-based embeddings for attribution. Our experiments show that default neural embeddings (even after manual fine-tuning) may not consistently improve authorship attribution accuracy. Instead, Bayesian analysis using latent topic embeddings derived from “function words” yields superior out-of-sample classification performance, underscoring that interpretable statistical language models, with their theoretical grounding, can offer significant advantages in authorship attribution. The code used in this analysis is available at github.com/sowonjeong/slm-2-llm