A Point Transformer Accelerator With Distribution-Aware Heuristic Distance Calculation

Point clouds are an important form of 3-D data used in applications, such as computer vision and autonomous driving, but the irregular and disordered nature of point clouds makes processing them severely challenging. Recently, point-based neural networks for point clouds have been widely used in various 3-D applications. Notably, transformer-based models have demonstrated state-of-the-art accuracy. However, three significant challenges exist: 1) data interdependence hinders parallel execution in networks like Point Transformer; 2) the farthest point sampling (FPS) involves redundant memory access and computational overhead; and 3) intermediate results require repetitive memory access and calculations between FPS and K-nearest neighbor (kNN) operators. This limits Point Transformer’s processing speed to 17.80 frames/s on NVIDIA Jetson Orin, below the real-time requirement of around 30 frames/s. In this article, we introduce PTrAcc++, an innovative point transformer accelerator to address the aforementioned three challenges from the following three levels. On the computation graph level, our investigation reveals that the Point Transformer’s performance suffers minimal degradation when operating within a constrained receptive field. Leveraging this insight, PTrAcc++ strategically frees the MaxPool and attention-kNN layers, along with their associated data dependencies, achieving an inconsequential loss in accuracy. On the operator level, we identify that the variability for distance computation among accessed points during FPS iterations contributes to redundant memory accesses and computational overhead. PTrAcc++ proposes a distribution-aware heuristic for distance calculation to minimize unnecessary memory accesses and computational redundancies within the FPS operator. On the architecture level, we recognize that the transition down process (encompassing FPS and kNN operations) constitutes 71.77% of the total inference time, PTrAcc++ proposes an integrated FPS-kNN architecture to select error-driven k neighbors, reducing repeated memory accesses and distance recalculations of intermediate results. Through extensive experimentation, PTrAcc++ demonstrates remarkable performance improvements, achieving end-to-end speedups of up to <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$2.96\times $ </tex-math></inline-formula>, <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1.70\times $ </tex-math></inline-formula>, and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$1.19\times $ </tex-math></inline-formula> when compared to the state-of-the-art acceleratorsPointAcc (Lin et al., 2021), MARS (Yang et al., 2023), and PTrAcc (Lian et al., 2023), respectively, across a variety of point cloud neural networks.