Predicting computationally critical islands for survey-scale radio source finding with PyBDSF
Next-generation radio interferometers require increasingly efficient processing of large image datasets. In highly parallel source-finding workflows, a few computationally expensive PyBDSF islands can dominate the fitting critical path and limit end-to-end throughput. We investigate whether such islands can be identified before fitting and whether this information can improve survey-scale throughput without changing the scientific workflow. We predicted a 90th-percentile (Q90) fitting-time priority from pre-fitting island properties using a histogram-based gradient-boosting quantile-regression model (HGBR-Q90) and supplied the learned priorities to a longest-processing-time (LPT)-based dispatch mechanism. We refer to this approach as computational long-tail prioritisation (CLP). We evaluated CLP with 32 workers on 1580 LoTSS DR3 images, compared it with controlled PyBDSF v1.13.0 baselines, integrated it into PyBDSF v1.14.1 as v14-CLP, and tested generalisation on five external datasets without retraining. The slowest 1% of islands contribute a median of 72.2% of the island-fitting runtime. In the controlled v1.13.0 benchmark, CLP reduces accumulated wall time from 265.85 to 169.23 h and saves 22.99 h relative to deterministic Weight-LPT. In PyBDSF v1.14.1, v14-CLP reduces accumulated wall time from 190.84 to 117.97 h, or 38.2%, while the median and 95th-percentile image runtimes decrease from 314.98 to 202.67 s and from 1015.22 to 635.85 s. Without retraining, B4 is closer to the measured-runtime reference than B3 on all five external datasets and is fastest within the controlled B1--B4 framework on four; v14-CLP further reduces complete-workflow wall time relative to B4 on all five. The computational long tail of PyBDSF island fitting is sufficiently predictable from pre-fitting properties to improve task prioritisation at survey scale. The largest gains occur for task-rich, long-tailed workloads in which island fitting contributes materially to total wall time. Catalogue outputs remain identical because CLP changes only task prioritisation and worker assignment.
Authors
- Oleg Mikhailovich Smirnov (ORCID: https://orcid.org/0000-0003-1680-7936)
- Yangfan Xie (ORCID: https://orcid.org/0000-0003-0962-2861)
- Jie Cao
- Jingying Wang
- Ying Mei
- Feng Wang
- Hui Deng
Publication Details
- Journal
- Astronomy and Astrophysics
- Published
- 2026-10-28
- DOI
- https://doi.org/10.1051/0004-6361/202662246
- Primary Topic
- Radio Astronomy Observations and Technology
- Type
- article
- Field-Weighted Citation Impact
- 0.00
Funders
- National Natural Science Foundation of China