Graph Neural Network and Fingerprint Attributions of Antibacterial Activity: Atom-Level Agreement Varies Across Random Seeds Reference distributions, topology controls and multi-instance replication

Abstract ML models with different inductive biases are routinely used to prioritise compounds for antibacterial screening, and attribution methods are used to explain them. Circular fingerprints consumed by tree ensembles and message-passing graph networks share neither featurisation nor optimisation, so agreement between their explanations would carry information that neither method alone provides. Whether two independently derived attribution schemes identify the same atoms of the same molecule, and whether such agreement survives retraining, has received little quantitative examination. We curated a published Staphylococcus aureus screen (39,312 compounds, 512 actives) to 38,748 compounds by stereochemistry-aware deduplication and partitioned it by Bemis–Murcko scaffold into training (26,345; 123 actives), validation (4651; 31), and held-out (7752; 302; 3.90%) sets. Fingerprint Random Forest reached a held-out AUPRC of 0.1233 fitted on the training portion alone, and 0.1518 when refitted on training plus validation, with a 9.75-fold enrichment in its top 50; eight AttentiveFP networks differing only in random seed reached a median of 0.0834 (range 0.0587–0.0949). TreeSHAP attributions were verified for additivity (error 3.69 × 10⁻⁷) and projected onto atoms through Morgan bit environments with a coverage normalisation that reduced their correlation with atom degree from + 0.268 to + 0.069; readout attention was obtained by a forward-pass replication verified against the model's own output (logit error 1.91 × 10⁻⁶). Across 60 molecules the median per-molecule Spearman correlation between the two maps was + 0.0890, exceeding a size-matched cross-molecule reference in 6 of 8 instances. The agreement is not topological: the attention–degree association varies in sign across instances (− 0.2562 to + 0.1497) while the |SHAP|–degree association is invariant at + 0.0439. Across instances, the agreement spans − 0.1067 to + 0.1547, a range including both signs, and no prespecified antibiotic core survived Benjamini–Hochberg correction under either scheme. Cross-model atom-level attribution agreement in this low-data regime is small, exceeds a demanding reference in most but not all training runs, and is not attributable to atom connectivity. Because its across-instance range includes both signs, a single training run can misstate both the magnitude and the direction of the agreement it reports. Attribution comparisons in comparable data regimes should be reported as distributions over independently trained instances rather than as point estimates.

Authors

Institutions

Publication Details

Journal
Research Square
Published
2026-11-13
DOI
https://doi.org/10.21203/rs.3.rs-11390965/v1
Primary Topic
Computational Drug Discovery Methods
Type
preprint

Funders

Controls
|||
ALL TIME
JAN
FEB
MAR
APR
MAY
JUN
JUL
AUG
SEP
OCT
NOV
preprint

Graph Neural Network and Fingerprint Attributions of Antibacterial Activity: Atom-Level Agreement Varies Across Random Seeds Reference distributions, topology controls and multi-instance replication

Mohamed Said Boulkrane, Mohamed Saber, Aya Darwish, Omar Elazouni et al.
2 links
Research Square
Computational Drug Discovery Methods
preprint

Graph Neural Network and Fingerprint Attributions of Antibacterial Activity: Atom-Level Agreement Varies Across Random Seeds Reference distributions, topology controls and multi-instance replication

Mohamed Said Boulkrane, Mohamed Saber, Aya Darwish, Omar Elazouni, Malikakhon Kurbonova
preprint en

Abstract

Abstract ML models with different inductive biases are routinely used to prioritise compounds for antibacterial screening, and attribution methods are used to explain them. Circular fingerprints consumed by tree ensembles and message-passing graph networks share neither featurisation nor optimisation, so agreement between their explanations would carry information that neither method alone provides. Whether two independently derived attribution schemes identify the same atoms of the same molecule, and whether such agreement survives retraining, has received little quantitative examination. We curated a published Staphylococcus aureus screen (39,312 compounds, 512 actives) to 38,748 compounds by stereochemistry-aware deduplication and partitioned it by Bemis–Murcko scaffold into training (26,345; 123 actives), validation (4651; 31), and held-out (7752; 302; 3.90%) sets. Fingerprint Random Forest reached a held-out AUPRC of 0.1233 fitted on the training portion alone, and 0.1518 when refitted on training plus validation, with a 9.75-fold enrichment in its top 50; eight AttentiveFP networks differing only in random seed reached a median of 0.0834 (range 0.0587–0.0949). TreeSHAP attributions were verified for additivity (error 3.69 × 10⁻⁷) and projected onto atoms through Morgan bit environments with a coverage normalisation that reduced their correlation with atom degree from + 0.268 to + 0.069; readout attention was obtained by a forward-pass replication verified against the model's own output (logit error 1.91 × 10⁻⁶). Across 60 molecules the median per-molecule Spearman correlation between the two maps was + 0.0890, exceeding a size-matched cross-molecule reference in 6 of 8 instances. The agreement is not topological: the attention–degree association varies in sign across instances (− 0.2562 to + 0.1497) while the |SHAP|–degree association is invariant at + 0.0439. Across instances, the agreement spans − 0.1067 to + 0.1547, a range including both signs, and no prespecified antibiotic core survived Benjamini–Hochberg correction under either scheme. Cross-model atom-level attribution agreement in this low-data regime is small, exceeds a demanding reference in most but not all training runs, and is not attributable to atom connectivity. Because its across-instance range includes both signs, a single training run can misstate both the magnitude and the direction of the agreement it reports. Attribution comparisons in comparable data regimes should be reported as distributions over independently trained instances rather than as point estimates.

Research Square
ITMO University (RU)
Russian Science Foundation, ITMO University
Computational Drug Discovery Methods
AI Navigator

Ask Laika to Summarize, Analyze, and Connect papers live on the map.

Summarize Papers & Methodologies

Extract key findings, datasets, and comparative methods across publications.

Benchmark Rankings & Visual Analytics

Rank top research institutions, authors, funders, topics, and journals by Field-Weighted Citation Impact (FWCI) and paper volume with instant charts.

Connect Distant Disciplines

Bridge topological clusters on the map to find hidden collaborative intersections.

Laika access coming soon