From detection to identity preservation: A pipeline–scenario–protocol–readiness review of datasets for visual multi-person tracking and re-identification

Abstract Multi-person tracking (MPT) requires visual systems to localize people in video frames and preserve their identities over time. Although public datasets have advanced person detection, person re-identification (Re-ID), person search, and tracking, the benchmark ecosystem remains fragmented across task definitions, annotation formats, visual scenarios, and evaluation protocols. Existing reviews mainly summarize algorithms or benchmark performance, but provide limited guidance on how different dataset families support the complete tracking-by-detection pipeline. This review analyzes 37 public datasets introduced or used between January 2005 and April 2026, with a final verification update conducted on 9 June 2026. The datasets include 8 detection-support datasets, 9 person Re-ID/person-search datasets, and 20 core tracking benchmarks spanning single-camera, multi-camera, crowded-scene, driving, sports, long-term, and synthetic tracking settings. To move beyond catalogue-style listing, we introduce a Pipeline–Scenario–Protocol–Readiness (PSPR) framework that characterizes datasets according to pipeline role, scenario complexity, evaluation protocol, deployment readiness, and benchmark limitations. The review shows that dataset choice strongly shapes what a tracking system can and cannot evaluate: detection datasets mainly test localization, Re-ID datasets support appearance-based identity modelling, and tracking benchmarks differ in whether they stress association ambiguity, crowd density, camera motion, cross-camera re-entry, or long-term identity preservation. By linking dataset families to tracking failure modes, evaluation protocols, and deployment scenarios, PSPR provides a practical guide for benchmark selection and interpretation. The curated dataset metadata, search strings, screening table, PSPR annotations, and supplementary benchmark resources are publicly available at https://github.com/ssbilakeri/pspr-mpt-dataset-review and archived on Zenodo at https://doi.org/10.5281/zenodo.20823270 .