Geo-Hete-HyperGNN as a prior-free molecular hypergraph representation learning strategy

Abstract Molecular representation learning is a cornerstone of AI-driven chemical discovery. However, most molecular hypergraph models define hyperedges using expert-crafted rules and often overlook three-dimensional molecular geometry. Motivated by the close correspondence between molecular structure and spectroscopy, including nuclear magnetic resonance (NMR), mass spectrometry (MS), ultraviolet-visible (UV-Vis), and infrared (IR) spectroscopy, we abstract spectroscopy-related structural cues into four semantic hyperedge types: atom-type, bond-type, bond-angle-type, and conjugated-system hyperedges. We further integrate molecular geometry into hypergraph construction and propose Geo-Hete-HyperGNN, an equivariant molecular hypergraph neural network that jointly captures molecular geometry, chemical semantics, and higher-order interactions through equivariant message passing, implicit relation regularization, and a Mixture of Molecular Hypergraph Experts (MoMHE) readout. With an equivariant self-supervised pretraining strategy, Geo-Hete-HyperGNN improves transferable molecular representations. Experiments on eight MoleculeNet benchmarks demonstrate consistent gains over prior methods, while additional spectroscopy-oriented evaluation on QM9S-QM9NMR validates the proposed hyperedge semantics for UV-Vis absorption, IR frequency, and NMR chemical shift prediction. These results position Geo-Hete-HyperGNN as a strong framework for hybrid geometric-semantic-higher-order molecular modeling.