Second-level Caches: Not for Instructions

Growing instruction footprints are straining processor front-ends, increasing fetch latency, and causing pipeline stalls. The universal approach to addressing this has been keeping instructions in each level of the cache hierarchy, but a plethora of more advanced techniques, ranging from instruction prefetching to prioritizing instruction cache-lines, have also been proposed to reduce instruction fetch latency. In this work we identify a significant subset of benchmarks that exhibit an insensitivity to instruction fetch latency to the extent that we can take the opposite approach and explicitly bypass the L2 for instructions. While conventional wisdom suggests that this will hurt performance, our detailed analysis of 2,120 traces finds that two factors mitigate this effect. First, the presence of ubiquitous effective instruction prefetching in the form of fetch-directed instruction prefetching (FDIP) means that as long as instructions are present in the LLC, the additional latency of missing in the L2 is often insignificant. And, second, by keeping instructions out of the L2, we find that the increase in data locality in the L2 leads to significant performance gains. Based on our analysis, we propose bypassing the L2 for instructions for a subset of applications and demonstrate how a simple decision tree can be trained offline and used online to cheaply and effectively choose when to bypass. Our technique increases performance by 3.4% geomean for sensitive benchmarks (those that see an improvement of at least 1% with an infinite I-cache) compared to an aggressive FDIP baseline, reaching up to 55% of the perfect I-cache performance with minimal to no overhead.