Reliability of atomic-column localization in scanning transmission electron microscopy under noise, annotation variation, and domain shift

Abstract Reliable atomic-column coordinates are essential for quantitative structural analysis in scanning transmission electron microscopy (STEM), yet strong performance on clean images does not ensure reliable localization under noise or across imaging domains. Localization errors can also propagate into coordinate-based structural measurements. Here, we evaluate automated atomic-site localization using Hybrid-STEMSeg, a compact convolutional neural network (CNN)--Transformer segmentation model, alongside established segmentation and microscopy-specific baselines. We test parent-held-out Sm-doped BiFeO$_3$ images under controlled corruption, examine how localization errors affect nearest-neighbor spacing, compare annotation targets, and assess transfer to external STEM datasets with and without target-domain adaptation. Model rankings depend on corruption, annotation, and imaging domain, showing that clean-image scores alone do not identify the most reliable model. Zero-shot transfer is limited, whereas target-domain adaptation improves held-out localization. This evaluation framework helps determine whether a localization model is suited to the imaging conditions and structural analyses for which it is intended.