Cross-Attention or Concatenation? A Multi-Seed Empirical Study of Code-Comment Fusion for Pull Request Outcome Prediction

Predicting whether a pull request (PR) will be merged, rejected, or left unre solved is a recurring problem in empirical software engineering, yet it remains unclear how much explicit fusion of code changes and reviewer comments adds over simpler designs, and conclusions drawn from a single training run can be misleading. We report a multi-seed comparison of encoder-based fusion strategies on a retrospective dataset of 33,955 PRs from 155 projects, three platforms, and 12 programming languages. A CodeBERT–RoBERTa hybrid with bidirectional cross-modal attention is compared with concatenation and late-fusion variants, four alternative code encoders (GraphCodeBERT, UniXcoder, CodeT5, CodeReviewer), unimodal baselines, and two re-implemented prior systems, using five random seeds per model. The cross-attention hybrid reaches 88.11% mean accuracy (sample SD 0.39 percentage points, pp) and outperforms its code-only counterpart in every seed (+1.85 pp; Holm-adjusted paired p = 0.008). How-ever, the seven best models lie within a 0.42 pp band, and cross-attention exceeds concatenation by only 0.34 pp on average across five encoders (positive for four of five, not significant for any single encoder); at one seed (42) the ordering of the two reversed. Review comments alone are about as informative as code alone, and combining them adds roughly 1.6–1.9 pp. Project identity is strongly associated with the outcome (Cram´er’s V = 0.60), so the random split used here probably overstates generalisation. All analyses are retrospective, use features that become available only after review, and exclude LLM-based baselines.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-18
DOI
https://doi.org/10.21203/rs.3.rs-11767396/v1
Primary Topic
Software Engineering Research
Type
preprint
Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Cross-Attention or Concatenation? A Multi-Seed Empirical Study of Code-Comment Fusion for Pull Request Outcome Prediction

Ahmed Hussen
1 links
Research Square
Software Engineering Research
preprint

Cross-Attention or Concatenation? A Multi-Seed Empirical Study of Code-Comment Fusion for Pull Request Outcome Prediction

Ahmed Hussen
preprint en

Abstract

Predicting whether a pull request (PR) will be merged, rejected, or left unre solved is a recurring problem in empirical software engineering, yet it remains unclear how much explicit fusion of code changes and reviewer comments adds over simpler designs, and conclusions drawn from a single training run can be misleading. We report a multi-seed comparison of encoder-based fusion strategies on a retrospective dataset of 33,955 PRs from 155 projects, three platforms, and 12 programming languages. A CodeBERT–RoBERTa hybrid with bidirectional cross-modal attention is compared with concatenation and late-fusion variants, four alternative code encoders (GraphCodeBERT, UniXcoder, CodeT5, CodeReviewer), unimodal baselines, and two re-implemented prior systems, using five random seeds per model. The cross-attention hybrid reaches 88.11% mean accuracy (sample SD 0.39 percentage points, pp) and outperforms its code-only counterpart in every seed (+1.85 pp; Holm-adjusted paired p = 0.008). How-ever, the seven best models lie within a 0.42 pp band, and cross-attention exceeds concatenation by only 0.34 pp on average across five encoders (positive for four of five, not significant for any single encoder); at one seed (42) the ordering of the two reversed. Review comments alone are about as informative as code alone, and combining them adds roughly 1.6–1.9 pp. Project identity is strongly associated with the outcome (Cram´er’s V = 0.60), so the random split used here probably overstates generalisation. All analyses are retrospective, use features that become available only after review, and exclude LLM-based baselines.

Research Square
Addis Ababa Science and Technology University (ET)
Software Engineering Research
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon