Can We Measure Legislative Complexity with LLMs?

The complexity of legislative language is of theoretical importance to many substantive questions about legislative politics. However, most existing measures of bill complexity are either generated at the broad issue level and applied to individual bills, or they are reliant on a simple metric like length. In this paper, we apply a pairwise comparison framework to the measurement of complexity in legislative texts. We compare the results of a Bradley-Terry model (Bradley and Terry, 1952) fit on pairwise comparisons made by human coders with the results of the same model fit on comparisons made by a large language models (LLMs). There is a moderately high level of agreement between human coders and the LLMs, and the relationships between observable text features and the underlying trait of complexity are similar in comparisons made by human coders and by the LLMs. Our work demonstrates that, with researcher-selected bridging texts and carefully designed prompts, LLMs can be used to measure complexity in legislative texts.