Cytotoxic activity of marine derived bioactive compounds from red sea sponges supported by LC-MS/MS profiling and molecular docking

Marine sponges are prolific sources of structurally diverse secondary metabolites with notable pharmacological potential. This study aimed to explore and compare the chemical composition and cytotoxic activity against liver cancer cells of three Red Sea sponge species—Stylissa carteri, Hemimycale arabica, and Negombata magnifica—collected from distinct ecological sites. Sequential solvent extraction was conducted using n-hexane, ethyl acetate, and n-butanol, followed by phytochemical screening, and LC–MS/MS-based metabolite profiling. Cytotoxicity was evaluated through MTT assay, colony formation, and wound healing (scratch) assay against HepG2 liver cancer cells. Molecular docking was carried out using the Chk2 kinase receptor to assess binding affinities and interaction modes of the identified metabolites. The total MeOH:DCM extract of S. carteri (El Gouna) showed potent cytotoxicity, with an IC50 of 37 µg/mL, 80% growth inhibition in the MTT assay, and significant suppression of colony formation (20.3 ± 2.1% survival) and cell migration (35.4 ± 2.1% wound closure). LC–MS/MS analysis tentatively identified pyrrole–imidazole alkaloids, including hymenialdisine, oroidin, and spongiacidin D. Docking revealed strong binding affinities of hymenialdisine (− 8.35 kcal/mol) and spongiacidin D (− 7.30 kcal/mol) with key Chk2 active-site residues. MD simulations indicated that the hymenialdisine–Chk2 complex maintained stability throughout 100 ns, with reduced RMSD (2.54 ± 0.48 Å), RMSF (1.46 ± 0.14 Å), Rg (20.70 ± 0.17 Å), and SASA (13,916 Å2) compared to the Apo protein, confirming a rigid and stable ligand–protein interaction. MM/GBSA calculations revealed a favorable total binding free energy (ΔG_bind = − 33.17 ± 0.17 kcal/mol), dominated by van der Waals and electrostatic contributions. ADMET predictions indicated compliance with Lipinski and Veber rules, suggesting good oral bioavailability. S. carteri from El Gouna is enriched in bioactive brominated alkaloids that exhibit potent cytotoxic activity against HepG2 cells. Integration of cytotoxic assays, LC–MS/MS profiling, docking, and MD simulations underscores the influence of species and environmental conditions on sponge-derived bioactivity. These findings provide a strong rationale for advancing S. carteri metabolites, especially hymenialdisine and spongiacidin D as potential cytotoxic agents.