Beyond the looking glass: multimodal LLM-based depth-sensing for spatial behavior modeling in media architecture

Large media façades are reshaping interactions in buildings and public spaces into immersive environments, yet empirical knowledge of how pedestrians behave inside these media spcaes is still limited. This study introduces a fully automated pipeline for in-the-wild behavior analysis that integrates a system which consists of a stereo-depth camera, an object detection model with multi-target tracking algorithm, and GPT-4o with visual reasoning. Deployed at London's immersive media building Now Arcade, the system captured 2 h of depth-enhanced video and produced more than six hundred anonymised visitor trajectories without any manual annotation. It reliably identified three recurrent behaviors: passing-by, lingering, and shooting (photographing or filming). To reveal where these actions occur, we propose Behavior Instance Density (BiD) heat-maps that project frame-level behavior instances onto a floor-plan grid of 0.5m × 0.5m squares. A comparative BiD study of 2 h-long content loops with static high-contrast imagery and dynamic low-contrast animation, shows clear content-driven behavior differences. Static saturated graphics encourage longer stays and more filming at both buildings entrance and exit thresholds, while dynamic darker visuals maintain a predominantly transit-oriented flow through the corridor.The proposed pipeline uses a compact, cost-effective sensing setup, safeguards privacy by discarding raw images after processing, and can be scaled for long-term or multi-site deployments. The resulting behavioral insights offer concrete guidance for media-architecture design and lay the groundwork for responsive façades that can update their digital content in real time according to observed human engagement.