Aspen: A 630 FPS Real-Time Posit-Based Unified Accelerator for Extended Reality Perception Workloads

Extended reality (XR) applications provide immersive user experiences but are latency-sensitive and require real-time processing. These applications typically have three main pipelines (Fig. 1). The perception pipeline processes sensor data, such as from cameras and inertial measurement units (IMUs), to understand the user and surrounding environment and must run at least as fast as the sampling rate of sensors, such as 30 frames per second (FPS) for cameras. The visual and audio pipelines use information from the perception pipeline to generate content for the user to see and hear, respectively. Of the three pipelines, the perception pipeline is the most computationally dominant, taking over 60% of the total application runtime [1]. The perception pipeline contains many different tasks, some which involve computer vision and some which use deep neural networks (DNNs). Because of this task diversity, previous works have focused on accelerating just one task, such as computer visionbased visual inertial odometry (VIO) to calculate the user's pose [2], DNN-based eye gaze extraction to determine where the user is looking [3], or DNN-based object classification to understand the scene [4]. A complete XR SoC for perception would require the integration of many dedicated hardware accelerators, with VIO adding 72% in area to a DNN accelerator. To address this issue, this work presents Aspen, an SoC for the complete perception pipeline that exclusively leverages DNNs for VIO, gaze estimation and scene understanding.