Predicting computationally critical islands for survey-scale radio source finding with PyBDSF

Next-generation radio interferometers require increasingly efficient processing of large image datasets. In highly parallel source-finding workflows, a few computationally expensive PyBDSF islands can dominate the fitting critical path and limit end-to-end throughput. We investigate whether such islands can be identified before fitting and whether this information can improve survey-scale throughput without changing the scientific workflow. We predicted a 90th-percentile (Q90) fitting-time priority from pre-fitting island properties using a histogram-based gradient-boosting quantile-regression model (HGBR-Q90) and supplied the learned priorities to a longest-processing-time (LPT)-based dispatch mechanism. We refer to this approach as computational long-tail prioritisation (CLP). We evaluated CLP with 32 workers on 1580 LoTSS DR3 images, compared it with controlled PyBDSF v1.13.0 baselines, integrated it into PyBDSF v1.14.1 as v14-CLP, and tested generalisation on five external datasets without retraining. The slowest 1% of islands contribute a median of 72.2% of the island-fitting runtime. In the controlled v1.13.0 benchmark, CLP reduces accumulated wall time from 265.85 to 169.23 h and saves 22.99 h relative to deterministic Weight-LPT. In PyBDSF v1.14.1, v14-CLP reduces accumulated wall time from 190.84 to 117.97 h, or 38.2%, while the median and 95th-percentile image runtimes decrease from 314.98 to 202.67 s and from 1015.22 to 635.85 s. Without retraining, B4 is closer to the measured-runtime reference than B3 on all five external datasets and is fastest within the controlled B1--B4 framework on four; v14-CLP further reduces complete-workflow wall time relative to B4 on all five. The computational long tail of PyBDSF island fitting is sufficiently predictable from pre-fitting properties to improve task prioritisation at survey scale. The largest gains occur for task-rich, long-tailed workloads in which island fitting contributes materially to total wall time. Catalogue outputs remain identical because CLP changes only task prioritisation and worker assignment.

Authors

Publication Details

Journal
Astronomy and Astrophysics
Published
2026-10-28
DOI
https://doi.org/10.1051/0004-6361/202662246
Primary Topic
Radio Astronomy Observations and Technology
Type
article
Field-Weighted Citation Impact
0.00

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
article

Predicting computationally critical islands for survey-scale radio source finding with PyBDSF

Oleg Mikhailovich Smirnov, Yangfan Xie, Jie Cao, Jingying Wang et al.
Astronomy and Astrophysics
Radio Astronomy Observations and Technology
article

Predicting computationally critical islands for survey-scale radio source finding with PyBDSF

Oleg Mikhailovich Smirnov, Yangfan Xie, Jie Cao, Jingying Wang, Ying Mei, Feng Wang, Hui Deng
article en

Abstract

Next-generation radio interferometers require increasingly efficient processing of large image datasets. In highly parallel source-finding workflows, a few computationally expensive PyBDSF islands can dominate the fitting critical path and limit end-to-end throughput. We investigate whether such islands can be identified before fitting and whether this information can improve survey-scale throughput without changing the scientific workflow. We predicted a 90th-percentile (Q90) fitting-time priority from pre-fitting island properties using a histogram-based gradient-boosting quantile-regression model (HGBR-Q90) and supplied the learned priorities to a longest-processing-time (LPT)-based dispatch mechanism. We refer to this approach as computational long-tail prioritisation (CLP). We evaluated CLP with 32 workers on 1580 LoTSS DR3 images, compared it with controlled PyBDSF v1.13.0 baselines, integrated it into PyBDSF v1.14.1 as v14-CLP, and tested generalisation on five external datasets without retraining. The slowest 1% of islands contribute a median of 72.2% of the island-fitting runtime. In the controlled v1.13.0 benchmark, CLP reduces accumulated wall time from 265.85 to 169.23 h and saves 22.99 h relative to deterministic Weight-LPT. In PyBDSF v1.14.1, v14-CLP reduces accumulated wall time from 190.84 to 117.97 h, or 38.2%, while the median and 95th-percentile image runtimes decrease from 314.98 to 202.67 s and from 1015.22 to 635.85 s. Without retraining, B4 is closer to the measured-runtime reference than B3 on all five external datasets and is fastest within the controlled B1--B4 framework on four; v14-CLP further reduces complete-workflow wall time relative to B4 on all five. The computational long tail of PyBDSF island fitting is sufficiently predictable from pre-fitting properties to improve task prioritisation at survey scale. The largest gains occur for task-rich, long-tailed workloads in which island fitting contributes materially to total wall time. Catalogue outputs remain identical because CLP changes only task prioritisation and worker assignment.

Astronomy and Astrophysics
National Natural Science Foundation of China
Openalex Percentile: Top 15%
Radio Astronomy Observations and Technology
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon