Reading Royalty Risk at Scale: LLM-Assisted Construction and Reliability Analysis of an Offset- and-Deduction Clause Corpus from SEC-Filed Technology Licences

Abstract We introduce RODEO , a corpus of 5,111 royalty offset and deduction clause spans from material-contract exhibits filed with the U.S. Securities and Exchange Commission (709 exhibits, 417 filers), labelled for their economic function under a seven-function taxonomy. Because exhaustive human annotation is costly, we build the resource with a protocol assisted by a large language model (LLM): independent double classification, adjudication of substantive disagreements, and a stratified human spot-check. We release the corpus with source identifiers and privacy-screened excerpts, together with the per-pass and adjudicated labels, prompts and code. As a quality-control analysis we examine how far the model can be trusted. It is highly self-consistent on the surface (82.4% exact-set agreement across passes) yet poorly calibrated: 82% of predictions carry the maximal confidence score, even that top bin flips label across passes 13.8% of the time, and the model’s own uncertainty flag recovers only 1% of genuine disagreements. On a deliberately hard subset, a model from an independent family agrees with the adjudicated label only 36.6% of the time, whereas on a broad stratified sample the two LLM annotators agree substantially (macro Cohen’s κ = 0.78). Inter-pass disagreement, not self-reported confidence, is therefore the more useful signal for triaging hard cases when full human labelling is infeasible. Supervised baselines show that the labelling is learnable (a bag-of-words logistic-regression classifier reaches macro-F1 0.80; the notice-and-documentation condition is hardest). The released labels are LLM-generated with a human spot-check (“silver”), and we set out the path to a fully human-adjudicated version.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-16
DOI
https://doi.org/10.21203/rs.3.rs-10473033/v1
Primary Topic
Computational and Text Analysis Methods
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Reading Royalty Risk at Scale: LLM-Assisted Construction and Reliability Analysis of an Offset- and-Deduction Clause Corpus from SEC-Filed Technology Licences

Y Kim
4 links
Research Square
Computational and Text Analysis Methods
preprint

Reading Royalty Risk at Scale: LLM-Assisted Construction and Reliability Analysis of an Offset- and-Deduction Clause Corpus from SEC-Filed Technology Licences

Y Kim
preprint en

Abstract

Abstract We introduce RODEO , a corpus of 5,111 royalty offset and deduction clause spans from material-contract exhibits filed with the U.S. Securities and Exchange Commission (709 exhibits, 417 filers), labelled for their economic function under a seven-function taxonomy. Because exhaustive human annotation is costly, we build the resource with a protocol assisted by a large language model (LLM): independent double classification, adjudication of substantive disagreements, and a stratified human spot-check. We release the corpus with source identifiers and privacy-screened excerpts, together with the per-pass and adjudicated labels, prompts and code. As a quality-control analysis we examine how far the model can be trusted. It is highly self-consistent on the surface (82.4% exact-set agreement across passes) yet poorly calibrated: 82% of predictions carry the maximal confidence score, even that top bin flips label across passes 13.8% of the time, and the model’s own uncertainty flag recovers only 1% of genuine disagreements. On a deliberately hard subset, a model from an independent family agrees with the adjudicated label only 36.6% of the time, whereas on a broad stratified sample the two LLM annotators agree substantially (macro Cohen’s κ = 0.78). Inter-pass disagreement, not self-reported confidence, is therefore the more useful signal for triaging hard cases when full human labelling is infeasible. Supervised baselines show that the labelling is learnable (a bag-of-words logistic-regression classifier reaches macro-F1 0.80; the notice-and-documentation condition is hardest). The released labels are LLM-generated with a human spot-check (“silver”), and we set out the path to a fully human-adjudicated version.

Research Square
University of Wisconsin–Madison (US)
Computational and Text Analysis Methods
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon