Pseudo-Cache: Extending the Access Scope of Requests with Global Perspective in GPUs

GPUs are comprised of numerous streaming multiprocessors (SMs) tailored for high performance computing. SMs incorporate private L1 caches to facilitate swift data access for thousands of threads running concurrently inside SMs. Requests that miss in L1 caches are directed to L2 caches that are shared by all SMs to retrieve the desired data, which takes much longer time compared with the delay of accessing L1 cache. Due to the locality among tasks running on SMs, the same data block might be accessed by requests from various SMs, leading to data replication across multiple L1 caches. Notably, data replication is prevalent in applications exhibiting high data locality. In order to capitalize on the data replications among L1 caches, we propose routing requests destined for L2 cache to other L1 caches that are predicted to contain the desired data, extending the access scope of requests. Consequently, we introduce a Pseudo-Cache positioned adjacent to the network on chip on L2 cache side, offering a global perspective of all SMs to manage these predictions effectively. Moreover, we implement the data path for request forwarding in a cost-effective manner, leveraging the existing network structure. Experimental results underscore the efficacy of our approach, showcasing an average 16.3% performance enhancement for applications characterized by substantial data replication on GPUs.