Hierarchical Multi-Prototype Appearance Memory: A Plug-and-Play Module for Identity-Stable Online Multi-Object Tracking

Online multi-object tracking (MOT) aims to maintain consistent target identities across video frames, yet it remains vulnerable to identity switches under occlusion and appearance variation. Many existing trackers rely on single-prototype exponential moving average (EMA) memory, which is efficient but prone to contamination, over-smoothing, and staleness. To address this issue, we propose Hierarchical Multi-Prototype Appearance Memory (HMP), a plug-and-play module for online MOT. HMP separates stable long-term identity anchors from short-term transitional evidence through a multi-prototype long-term memory and a short first-in-first-out (FIFO) queue. A unified joint reliability score governs memory writing and maintenance, and a frozen two-stage association strategy first performs stable primary matching and then allows conservative short-term recovery only on residual cases. Experiments on MOT17 and MOT20 show that HMP improves identity continuity while preserving competitive overall tracking quality. Controlled ablations further support the effectiveness of the proposed memory representation, reliability control, and staged evidence usage under fixed upstream modules.