Offline Preference Learning with Unknown Behavior Policies: Sharp Finite Class Rates and Constructive Robust Extensions

Version 2.0 extends this theoretical study of offline Bradley–Terry preference learning with unknown sampling distributions and a KL-regularized population objective. The new main result permits an arbitrary unknown ordered-pair law, including correlated endpoints and different endpoint marginals. A range-aware comparison-graph coefficient Γ measures the displacement of the optimal policy from the mean endpoint marginal. Finite Γ requires matching mass on every graph component, and Γ = (C* − 1)/2 for iid endpoints. For a realizable finite policy class of size M and every β > 0, a proper likelihood-constrained pessimistic estimator achieves, up to universal constants, min{aβ, sqrt(ΓΛ/N) + bβΛ/N}, where aβ = min{1, 1/β}, bβ = max{1, β}, and Λ = log M + log(1/δ). A matching worst-case lower bound holds at every exact finite Γ and allows arbitrary randomized policy outputs. The same estimator automatically recovers the sharper iid rate aβ min{1, sqrt(ΓΛ/N) + Λ/N}, without testing independence. The literal deterministic upper cap is min{2, 1/(2β)}. A bounded box-linear model with explicitly enumerable actions admits a proper algorithm polynomial in the full rational input length. Its dimension lower bound matches the upper rate up to the displayed entropy logarithm. Full proofs include the normalized moment inequality, graph transfer and nullspace conditions, exact-Γ lower constructions, automatic iid recovery, uniform linear-family bounds, and finite-precision feasibility and optimization details. The iid finite-class results, existing affine reduction, unknown reward-misspecification guarantees, and generic dimension characterization from version 1 are retained with their proofs. The robust and generic dimension results keep their iid sampling assumptions. The manuscript distinguishes its specific new guarantees from established baseline-centered pessimism, transfer coefficients, graph resistance, and convex-optimization tools through primary-source comparisons with FREEHAND, RPO, PEPO, and Han–Goyal–Ma. This work assumes iid complete tuples, a correctly specified conditional Bradley–Terry label model, bounded rewards, matched context distributions, and the stated realizability and coverage conditions. It does not establish empirical LLM improvements, globally efficient neural-policy optimization, or efficient normalization over exponential sequence spaces. The record includes the 39-page manuscript and self-contained LaTeX sources with a changelog and version comparison. Author: Zhijie Zheng, Peking University; ORCID 0009-0006-7610-0691. License: CC BY 4.0. The title and concept DOI are unchanged from version 1.