PocketMap: An Interaction-centric 3D Map Serving System for Spatial Understanding on Mobile Devices

Recently, the rapid advancements of spatially intelligent applications have greatly heightened the demand for context-rich 3D maps on mobile devices. However, existing 3D maps are typically static and interaction-agnostic, making them ill-suited for dynamic user needs, leading to excessive transmission overhead. To address this, we present PocketMap, an interaction-centric 3D map serving system that delivers only the necessary content tailored to each request, considering user intent, environmental context, and available on-device resources. PocketMap adopts a multi-layered, incrementally encoded 3D map representation with varying fidelity levels, enabling adaptive map selection. Leveraging this layered map, we propose a prompt- and context-aware task performance estimator to evaluate the contribution of each map layer. A packet planner then formulates a constrained optimization problem to jointly select appropriate map layers and compression strategies. Together, these components enable on-demand, interaction-aware 3D map serving. We evaluate PocketMap across four environments and four downstream applications: object localization, 3D shape analysis, surface analysis, and image retrieval. User studies show that PocketMap improves user satisfaction by 35.5% compared to the baseline. Extensive evaluations further demonstrate a 45.1% reduction in transmission volume and a 13.7% improvement in task performance compared to the baseline.