Using GPU-only buffers in OpenMP Target Offload

GPUs provide significantly more compute performance than CPUs, both in terms of raw compute power and memory bandwidth, prompting scientific software developers to write codes that can execute on those compute resources. Note that this is a somewhat similar transition as the scientific compute community encountered when CPUs moved from being purely sequential resources to being massively multi-threaded. It is thus perhaps not a surprise that OpenMP, originally developed for CPU-based parallelization, can now address the parallelization needs of GPUs as well. From a purely computational point of view, many OpenMP-based codes can be ported to GPU compute with a simple tweaking of the OpenMP pragmas, i.e. by telling the OpenMP runtime to execute the parallel code portion on the GPU instead of doing it on the CPU. Unfortunately, most GPUs come with their own memory subsystems, requiring separate management of memory buffers on CPU and GPU compute cores. OpenMP Target offload provides the means for automatically copying the buffers between the two memory subsystems as needed, but at a significant cost, which can negate the GPU compute performance advantage, if used naively. The recommended alternative is to use OpenMP to create a persistent copy, and then manually synchronize the two only when needed, which is indeed performant, but can be error prone. OpenMP Target offload abstracts away the distinction between the CPU and GPU memory copies by translating at compile time all CPU pointers to their GPU equivalents, when running on GPU cores. This requires the compiler to know the conversion logic of every pointer that is accessing a memory location. It works remarkably well for contiguous buffers, but often fails for tree-like structures, especially when accessed from opaque functions, like it is common in Object Oriented programming, e.g. in C++. While there are ways to work around this limitation, they tend to be cumbersome and error prone. Memory translation is, however, not a hard requirement in OpenMP Target offload. If one stores a GPU-only pointer in a variable, it can be used as-is to access the relevant memory buffer from the GPU code, although the user does have to notify the compiler to disable the automatic conversion on it, which can be error prone in its own right. This makes accessing both flat and tree-like data structures trivial, as long as all the pointers inside the tree already point to the GPU memory subsystem. Finally, OpenMP also provides an API to create GPU-resident buffers, further simplifying the process, although it is definitely more cumbersome than using language-native memory allocation instructions. The major downside is the fact that such a buffer cannot be used directly from the CPU code, but that is often acceptable, given the performance difference between CPU and GPU compute. One final benefit of GPU-only buffers is the reduced memory usage of the CPU memory. The default memory management requires at least as much CPU memory as there is GPU memory in use. While this was not a problem when GPUs had tiny memory amounts, modern GPU systems can have more GPU than CPU memory. With GPU-only pointers, this issue is avoided.