Optimizing PTZNet for Real-World Deployment - Batch-Level Mixing to Compensate for Data Scarcity in Pose and Overlap Regression

In urban planning and traffic management, Pan, Tilt and Zoom (PTZ) cameras monitor streets, intersections, and public areas. Precisely estimating how a single PTZ camera moves between images, and how much their Fields of View (FoVs) overlap, is challenging due to scene variability, lens distortion, dynamic motion, and limited or unavailable camera parameters. Collecting large, diverse real-world datasets with precise metadata is costly and logistically complex. This thesis investigates whether our Siamese deep learning model, PTZNet, can infer relative PTZ rotations and FoV overlap with competitive performance caused by camera movement between image pairs, while relying primarily on virtually created synthetic images and mitigating the Synthetic-To-Real (S2R) domain gap, which is the discrepancy in feature distribution between real-world and synthetic data. PTZNet jointly regresses camera rotation and FoV overlap. Performance is measured with a 3D rotation error, the spherical Intersection over Union (IoU) for overlap, and a combined relative error capturing joint accuracy. A key element is the Batch-Level Mixing (BLM) method, which strategically integrates synthetic and limited real-world images within training batches to improve generalization to real-world scenes. Our experiments show that mixing synthetic images with a small amount of real-world data yields substantially lower errors in predicting rotation and overlap than training on either domain alone. This improvement persists across varying real-data amounts: When real-world image collection is costly, training primarily on synthetic images and tuning with a small targeted set of real-world samples effectively compensates for limited real-world data and improves inference. The man contributions of this work include (1) an analysis of the S2R domain gap for virtually created PTZ data and its effect on inference, (2) the development of the Batch-Level Mixing (BLM) data training strategy that outperforms Dataset-Level Mixing (DLM) in robustness to domain shift, (3) quantitative evidence that synthetic images improve relative PTZ rotation and overlap prediction accuracy when real-world images are scarce, and (4) the introduction of PTZNet, a Siamese architecture for joint PTZ rotation and FoV-overlap regression trained on mix of synthetic and real-world data.