Optimizing the DFCPP Dataflow Runtime Library for Resource Utilization in NUMA Systems

In parallel computing, Data Flow Graphs play a critical role by explicitly representing the dependencies between tasks, which is essential for task scheduling and resource utilization. In the process of scheduling optimization, it is crucial to address the dual challenges of balancing computational load and distributing data access pressure. The Dataflow for C++ (DFCPP) possesses the advantage of accurately sensing task data sizes, and based on this capability, this paper proposes a Primary-Secondary Core Selecting (PSCS) strategy specifically designed for Hyper-Threading-enabled Non-Uniform Memory Access systems (HT-NUMA). This strategy aims to mitigate interference between threads on the same physical core in a hyper-threading environment, thereby enhancing task execution stability and overall performance. Additionally, DFCPP integrates a task-stealing mechanism with a proactive allocation strategy for large-cache tasks, enabling the distribution of large tasks to low-load NUMA nodes, which reduces cache contention and improves cache hit rates. Experimental results demonstrate that, compared to common dataflow programming libraries such as Taskflow and TBB, DFCPP achieves over a 20% performance improvement when the system core count meets computational demands, and nearly a 25% improvement when handling large-dataset tasks. Furthermore, DFCPP exhibits high efficiency and adaptability in processing tasks with various directed acyclic graph topologies, fully leveraging the parallel computing advantages of NUMA systems, and demonstrates exceptional performance and broad applicability in dataflow task scheduling, resource optimization, and complex dependency management.