Virtual Memory Revisited for Tiered Memory

The landscape of memory technologies has grown substantially more diverse and heterogeneous, with a number of special-purpose memory technologies reaching market maturity in recent years, such as NVRAM (Non-Volatile Memory), HBM (High-Bandwidth Memory) or the upcoming CXL.mem. Future computing systems will have to contend with byte-addressable and cache-coherent main memory that is highly diverse, either in terms of the fundamental performance characteristics of its underlying memory technology, or due to the fact that CXL.mem will allow accessing homogenous but remote memory. The VM (Virtual Memory) subsystems of today have been empirically optimized over decades, but were designed for a single tier of main memory only. While recent approaches that integrated tiered memory have focused on better memory access metrics to improve page placement strategies, we seek to leverage the proven heuristics that are already in place. In times of high load we assign pages from slower memory, demote inactive pages according to standard page replacement strategies and use lazy page promotions when sufficient resources are available again to retain the inherently high reactivity of demand paging with regard to working set changes. We integrate these mechanisms into the VM subsystem of a full-featured mainstream operating system (FreeBSD) with minimal adaptations. The resulting multi-tier VM system runs in a stable and efficient manner under high stress even with highly unfavorable work loads.