Liberating the Data Aware Scheduler to Achieve Locality in Layered Scientific Workflow Systems

Large scale scientific workflows are typically run using a task-based distributed workflow system. Many contemporary workflow systems are organized in a layered or modular fashion, with one component managing the DAG or workflow graph structure, and another component acting as the scheduler or executor. These components communicate and regulate the flow of tasks from the application to the remote execution site. Scientific workflows perform significant amounts of I/O, typically using an HPC parallel filesystem. Node-Local storage technologies are often used as an effective supplement to parallel filesystems, relieving them of the large amount of bandwidth consumed by intermediate data reads and writes which would otherwise cause I/O bottlenecks. Node-Local storage techniques depend upon effective scheduling to place tasks close to their necessary data, thus benefiting from data locality. Effective locality based scheduling is a challenge however. The conventional layered architecture results in the scheduler considering tasks on an individual basis with a narrow view of the greater DAG. We present a modified architecture of a workflow system which allows the efficient construction of dependency-based task groups which are passed through the DAG manager, scheduler, and finally to the remote worker node where improved use of data locality can be achieved. These modifications were implemented using the Parsl parallel library and TaskVine execution engine. We evaluate this implementation with a benchmark application and a Montage workflow. We compare the results between a conventional data-aware scheduler, our task grouping implementation, and a workflow system using only shared storage. We find that task grouping achieves a significantly greater degree of data locality, thereby improving performance and reducing total data movement in the cluster when compared to the other two methods.