Mechanism-guided generative chemistry yields distinct agonist-like and antagonist-like ligand populations for the serotonin 5-HT2A receptor

Abstract Designing serotonin 5-HT2A receptor ligands with a specified functional direction is difficult because binding affinity, assay-derived mechanism labels, molecular properties and receptor conformation are only partially coupled. We developed a mechanism-guided generative chemistry workflow integrating curated ChEMBL data, scaffold-aware supervised learning, a recurrent SMILES prior, reinforcement learning, medicinal-chemistry prioritization, receptor-context docking, MM-GBSA rescoring and property-adjusted contact analysis. The dataset comprised 8,949 measurements and 6,252 compounds. A five-seed binding Ki ensemble achieved scaffold-test mean absolute error of 0.588 pChEMBL and R² of 0.607; non-generalizing functional EC50 models were excluded from the reward. Separate campaigns generated 500 valid unique agonist-like and 500 antagonist-like molecules. Antagonist-like candidates showed higher predicted affinity and mechanism-label confidence, whereas agonist-like candidates showed higher QED, greater internal diversity and lower training-set similarity. Matched frozen-prior sampling showed only modest policy-level shifts from the already HTR2A-focused prior, whereas the reward-retained candidate populations were strongly enriched for the campaign objectives. Native-ligand redocking reproduced four deposited poses with heavy-atom RMSD values of 0.357–1.082 Å. Under fully contact-neutral pose selection and the prespecified repeated-measures model, S242, L229 and L362 showed FDR-supported campaign-by-receptor-context interactions; S242 and L229 retained the directions observed with the original selector, whereas L362 emerged in the contact-neutral analysis. The study identifies distinct model-defined chemical regimes and links campaign-level property drift to receptor-contact topology; generated compounds remain computational hypotheses requiring experimental validation.