Nitrogen-containing heterocyclic structure-toxicity reliant mechanism based EC50 toxicity prediction models for carbamazepine’s transformation products

• Novel ML-driven regression and classification models for first predicting EC 50 toxicity of CBZ’s TPs using V. fischeri bioassay data. • The toxicity of CBZ’s TPs was determined by their heterocyclic structure characteristics, topological structure characteristics, and nitrogen atom properties. • Prediction model for NHCs with similar nitrogen heterocyclic structure present robust. • Developed VAEs based data augmentation to overcome limitations of small datasets model overfitting. • Established high-quality toxicity dataset with experimental value. Carbamazepine (CBZ) and its transformation products (TPs) were frequently detected in aquatic environments, and their long-term presence was linked to microbial antibiotic resistance and global health risks. In this study, we developed high-precision regression and classification models to predict the toxicity of CBZ's TPs, using bioassay data from Vibrio fischeri EC 50 values. The models were trained on experimentally determined toxicity data of 38 nitrogen-containing heterocyclic compounds (NHCs) and validated using 11 CBZ's TPs as an external validation set. Eight machine learning models were used to train regression models, and six were used to train classification models. To address overfitting due to the limited dataset size, variational autoencoder (VAE) based data augmentation expanded the training dataset from 38 to 200 samples. Among the machine learning models trained, Support vector regression (SVR) and Gradient boosting machines (GBM) were identified as the optimal regression and classification models, respectively. Additionally, Shapley additive explanations (SHAP) analysis was employed to identify the key molecular features contributing to toxicity, highlighting the critical roles of heterocyclic structures, topological properties, and nitrogen atom characteristics of TPs in determining their toxicity. It proved NHCs similar structure-based nitrogen heterocyclic structure training model present robust. This work provided a reliable framework for assessing the toxicity of CBZ's TPs in environmental monitoring and ecotoxicity risk assessment.