Multimodal Transformer to Improve In Vitro Fertilization (IVF) Success Rate Using External Factors : Enhancing Embryo Selection with Deep Learning and Environmental Data Analysis

This study investigates how artificial intelligence can assist in embryo selection during in-vitro fertilization (IVF) by combining embryo images with laboratory environmental data. A deep learning model was developed to predict embryo quality using the Gardner scoring system, which includes expansion (EXP), inner cell mass (ICM), and trophectoderm (TE). One of the main challenges in this study involved incomplete access to lab-specific environmental data. To address this, a Random Forest model was used to estimate laboratory conditions based on external weather data, allowing for the creation of a synthetic but biologically relevant dataset. Two neural network architectures were evaluated: A Vision Transformer (ViT) model and a CNN model based on Inception V3. Both models were trained using a multitask approach to predict all three Gardner criteria. The ViT-based multimodal model achieved the highest overall performance, with an accuracy of 70% and weighted F1-score of 83% for EXP, 63% accuracy and 70% weighted F1 for ICM, and 60% accuracy with 71% weighted F1 for TE. These results suggest that the model can reliably classify mid-range embryo grades, although performance decreases for classes with low representation or ambiguous morphology. The Inception V3 model also performed best on EXP prediction with an accuracy of 77% and F1-Score of 58%, but struggled with recognising all classes for ICM and TE, leading to a poorer performance of 39% accuracy and 32% F1-Score for ICM and for TE 26% accuracy and F1-Score. To better understand the role of environmental influences, a SHAP analysis was performed on the ViT results. The results indicate that temperature, air pressure, and humidity contribute meaningfully to embryo quality, especially during specific developmental windows such as Day 0, Day 2, and Day 5. This work shows that combining image analysis with contextual environmental data can improve predictive performance in embryo quality assessment. Even with limited or incomplete datasets, strategies such as data synthesis and transfer learning can help build reliable models. The findings suggest a promising direction for applying artificial intelligence in reproductive medicine and support more objective decision-making in clinical embryology.