Semantic Loops in LLM Agent Trajectories: A Construct-Validity Audit of Three Public Resources

Long-horizon LLM agents can enter semantic loops: sequences in which action parameters vary at every step but the agent makes no semantic progress. We audit whether three public agent-trajectory resources provide valid ground truth for this construct: a cleaned SWE-agent corpus (3,909 windows, 582 iids), OpenClawBench (300 human-audited gold trajectories drawn from a 31,264-trajectory benchmark, with a five-class process-anomaly taxonomy), and CUAErrorBench (204 human-annotated OSWorld failure trajectories). Three findings emerge. First, the SWE-agent binary label future_has_new_res—widely used as a loop proxy—disagrees with a repetition/failure-recurrence heuristic (RFH) on 44.2% of 3,909 windows. A four-cell audit (48 windows; author plus a blind LLM second annotator) finds near-zero false positives under a two-class collapse but high false negatives on non-overlapping subsets. Inter-annotator agreement is fair for three classes (Cohen’s κ = 0.214) and moderate when YES and MAYBE are collapsed (κ = 0.500); disagreement concentrates on the YES/MAYBE boundary. In the most diagnostic cell, where both automatic labels say “not a loop,” the blind LLM confirms 3 of 7 author-YES cases. Second, OpenClawBench’s taxonomy contains no loop category, and loop-morphology indicators show no stable separation in the examined gold sample; its median normal trajectory length (3 steps) cannot support process-level stalling. Third, CUAErrorBench’s R4 (“Inefficient/Redundant Strategy”) is the only public annotation class containing parameter-varying semantic loops: 5 of 21 cases (Wilson 95% CI [10.6%, 45.1%]), buried inside a broader efficiency label. Under RFH, the strongest baseline’s TPR at 5% FPR falls from 0.536 to 0.393, but this characterizes the RFH-strict subset and cannot be extrapolated to the full construct. Among the three resources examined, parameter-varying semantic loops lack both an independent annotation slot and a precise operational boundary against exact repetition; this ambiguity is itself a barrier to detection research.