A heterogeneous dual-branch cross-attention transformer for engineering cost modeling

Heterogeneous engineering data contain categorical and numerical variables with distinct statistical properties and complex interactions, making direct concatenation suboptimal for regression. This study proposes a Cross-Attention Feature-aware Transformer (CAFT), a dual-branch architecture in which categorical features are embedded and encoded by a Pre-LayerNorm Transformer, while numerical features are converted into tokens and modeled by a residual multilayer perceptron. A cross-attention module allows numerical tokens to query categorical representations, followed by self-attention for interaction refinement. An improved particle swarm optimization algorithm tunes key hyperparameters independently within each cross-validation fold to prevent information leakage. For engineering cost prediction on 163 residential projects, leakage-free nested cross-validation (5 outer folds, 3 inner folds) yields an RMSE of 93.4 CNY/m2, MAE of 65.5 CNY/m2, R2of 0.927, and MAPE of 3.90%. CAFT achieves lower or comparable mean error than six tuned tree-based/kernel baselines and an FT-Transformer, although Holm-corrected paired tests do not show significance for every comparison. Ablation, sensitivity, and out-of-fold SHAP analyses further validate component contributions, robustness, and interpretability.