Retrospective Evaluation of Livestock Monitoring Replay Controllers Under Limited Evidence and Distribution Shift

Livestock monitoring systems must determine which records to inspect, assess whether the evidence supports an answer, and decide when to abstain. We constructed a retrospective replay benchmark using public MmCows records, with separate animals for development and evaluation. The benchmark included 120 current-state episodes and 62 scripted temporal and history checks from four evaluation cows. Across all 182 episodes, conditional routing used 2.51 stored-record calls per episode, compared with 2.82 for a fixed sequence. Task success was lower (90.11% versus 93.96%), while errors among accepted answers were higher (14/85 versus 2/68, or 16.47% versus 2.94%). Conditional routing did not save calls within the current-state episodes. Requiring two supporting cameras reduced errors and coverage, while the learned policy did not outperform a revisable fixed order. Raw confidence, temperature scaling, and isotonic calibration each retained 24 errors among 940 cases. Exploratory image analyses showed poor transfer to the Cow Behavior Video Dataset 5 (CBVD-5), and personalized forecasts showed no clear advantage over seven-day means. Verification accepted all 31 synthetically constructed correlated errors with valid record structure. The framework identifies limitations in evidence acquisition and verification, while the small number of animals and constructed task mixture limit conclusions about farm-level performance.