Predicting the anticancer activity of indole derivatives: A novel GP-tree-based QSAR model optimized by ALO with insights from molecular docking and decision-making methods

Indole derivatives have demonstrated significant potential as anticancer agents; however, the complexity of their structure-activity relationships and the high dimensionality of molecular descriptors present challenges in the drug discovery process. This study addresses these challenges by introducing a modified GP-Tree feature selection algorithm specifically designed for regression tasks and high-dimensional feature spaces. The algorithm effectively identifies relevant descriptors for predicting LogIC₅₀ values, the target variable. Furthermore, the GP-Tree method adeptly balances the selection of both positively and negatively contributing descriptors, enhancing the performance of DT, k-NN, and RF models. Additionally, the SMOGN technique was employed to address class imbalances, expanding the dataset to 1381 instances and enhancing the accuracy of IC₅₀ predictions. Various machine learning models were optimized using probabilistic and nature-inspired algorithms, with the Ant Lion Optimizer (ALO) demonstrating the highest efficacy. The AdaBoost-ALO (ADB-ALO) model outperformed all other models, such as MLR, SVR, ANN, k-NN, DT, XGBoost, and RF, achieving an R² of 0.9852 across the entire dataset, an RMSE of 0.1470, and a high CCC of 0.9925. SHAP analysis revealed critical descriptors, such as TopoPSA and electronic properties, which are essential for potent anticancer activity. Furthermore, molecular docking, in conjunction with the Weighted Sum Method (WSM), identified promising candidates, particularly N-amide derivatives of indole-benzimidazole-isoxazoles, which exhibit dual inhibition against topoisomerase I and topoisomerase II enzymes. Consequently, this research integrates computational predictions with experimental insights to accelerate the discovery of novel anticancer therapies through the accurate prediction and interpretation of the anti-prostate cancer activity of indole derivatives.