Cathepsin B-Oriented Screening, Isolation, and Antitumor Validation of Bioactive Metabolites from Sargassum polycystum

Marine medicinal algae represent a valuable reservoir of bioactive metabolites for anticancer drug discovery, yet the efficient identification of target-relevant compounds from chemically complex marine matrices remains challenging. In this study, an integrated cathepsin B-oriented strategy was developed to discover, prioritize, isolate, and validate antitumor metabolites from the brown alga Sargassum polycystum. Affinity ultrafiltration LC-MS was first applied to screen CTSB-binding constituents from the crude extract, followed by molecular docking, molecular dynamics simulation, and gray relational analysis for multidimensional candidate prioritization. Seven CTSB-binding metabolites were characterized, including chlorogenic acid, caffeic acid, cynarin, loliolide, taxifolin, senkyunolide H, and dihydroactinidiolide, with binding degrees of 73.99–85.61% at 2.5 U/mL CTSB. Molecular docking showed predicted binding affinities ranging from −6.3 to −9.4 kcal/mol, compared with −10.2 kcal/mol for the positive control CA-074Me. Integrated computational and biological evaluation identified caffeic acid, cynarin, and taxifolin as the top-ranked candidates. Preparative recovery was then achieved using counter-current chromatography combined with semi-preparative HPLC, and the isolated compounds were structurally identified by LC-MS/MS and NMR. Cellular assays in NCI-H1975 cells suggested that these metabolites reduced CTSB-associated enzymatic activity and intracellular CTSB-related fluorescence signals to different extents, with phenolic acid-type compounds exhibiting comparatively stronger effects. At the extract level, S. polycystum dose-dependently suppressed NCI-H1975 xenograft tumor growth, with inhibition rates of 48.78%, 36.58%, and 22.86% in the high-, middle-, and low-dose groups, respectively, without evident hepatorenal histopathological toxicity. This effect was associated with reduced CTSB, Ki-67, and Bcl-2 staining, increased Bax staining, enhanced apoptosis, and ultrastructural alterations in tumor tissues. Overall, this study provides a practical CTSB-oriented workflow for discovering antitumor metabolites from marine medicinal algae and supports further investigation of S. polycystum as a potential source of anti-NSCLC candidates.