TaskFlare: A Heterogeneity-aware Unified Scheduling Framework for Diverse Domain-Specific Applications in Supercomputing Environments

Efficiently scheduling heterogeneous diverse domain-specific applications in supercomputing environments remains a critical challenge. Existing frameworks (e.g., Parsl and Legion) offer partial solutions, but some are not user-friendly and others have limited scheduling performance. This paper proposes TaskFlare, a heterogeneous-aware unified scheduling framework that integrates usability with fine-grained resource control. TaskFlare enhances diverse domain-specific application support and extends a virtual allocation mechanism and two optimization modules for supercomputing and domain-specific applications. Evaluation results show that TaskFlare achieves up to a 5.8 × performance improvement in AI optimization over unoptimized implementations, with linear scalability and faster execution in classic benchmarks.