How Advancements in AI Can Help Improve Neuro-Ophthalmologic Diagnostic Clarity

Artificial intelligence (AI) and deep learning are increasingly being used in the field of neuro-ophthalmology to aid in the diagnosis and management of complex ocular conditions. These advanced technologies leverage large data sets and sophisticated algorithms to identify patterns and incongruities that might be challenging for the human eye to recognize.1 In particular, deep learning models, a type of machine learning that use multilayer networks to build models that can then make predictions, have shown promise in automating the analysis of fundus photographs, a critical diagnostic tool in neuro-ophthalmology.2 These models can potentially improve diagnostic accuracy, reduce variability between clinicians, and expedite decision-making processes. The importance of deep learning in the current context is underscored by the complexity and heterogeneity of neuro-ophthalmologic disease presentation. An initial clinical presentation may suggest a number of diagnoses, although each diagnosis requires specific treatment and portends its own prognosis. Often, treatment decisions need to be made quickly, and the wrong decision could lead to a delay in recovery or even permanent vision loss. There is a clear need for tools that help clinicians make early and correct diagnoses. As the population ages and the prevalence of neuro-ophthalmic disease increases, so do the needs for accurate diagnosis.3 Accessible solutions should also support clinicians in regions with limited access to specialized care.4 By enhancing the ability to detect subtle features in fundus photographs, deep learning models can aid in addressing these challenges. Two recent studies in this issue of the JNO, 1 by Bénard-Séguin et al5 and the other by Sathianvichitr et al6 have made significant strides in this area, demonstrating the potential of AI and deep learning in neuro-ophthalmology. Both studies have developed and tested deep learning systems (DLS) for differentiating between specific ocular conditions based on fundus photographs. In the article by Bénard-Séguin et al, a model was tested to differentiate between eyes with optic neuritis (ON) caused by multiple sclerosis (MS) vs non-MS causes. In the article by Sathianvichitr et al, deep learning models were used to predict the presence of optic disc drusen (ODD) vs papilledema. This editorial is a companion of these 2 articles and will discuss the methodology used in deep learning models. STUDY DESIGN AND DATA COLLECTION Data collection, study design, and the establishment of inclusion and exclusion criteria are critical steps in the development of deep learning models. The quality, diversity, and size of the data set directly influence the performance and generalizability of the model.7 For instance, a data set that includes a wide range of patient demographics, disease severities, and imaging modalities can enhance the model's performance and applicability across different clinical scenarios. The criteria for including and excluding data can impact the model's ability to accurately represent the condition of interest and avoid biases. For example, including only high-quality images might improve model performance but at the cost of limiting generalizability to real-world settings where image quality varies. Similarly, excluding patients with comorbid conditions could lead to a model that performs well under ideal conditions but fails in more complex, real-world scenarios. Careful consideration of these factors is necessary to develop deep learning models that are both accurate and broadly applicable in clinical practice. In the studies by Bénard-Séguin et al and Sathianvichitr et al, both used a retrospective design, drawing on existing data sets for their analyses, which allows for the efficient use of large amounts of data. The inclusion and exclusion criteria in both studies were clearly defined. Bénard-Séguin et al included fundus photographs from patients with ON, categorized into MS ON and non-MS ON groups, and excluded photographs with inadequate resolution or unclear optic disc visibility. Sathianvichitr et al included patients with confirmed ODD or papilledema and excluded patients with concurrent ophthalmic pathologies. These criteria ensure that the data used in the models are relevant and specific to the conditions being studied, although each selected data set had its own limitations. Image Acquisition and Preprocessing In studies using deep learning methodology, it is important to clearly define the image acquisition protocols. In neuro-ophthalmology, fundus photographs are a common data source for deep learning models.8–10 Preprocessing of neuro-ophthalmic images for deep learning models involves several critical steps to prepare the data for analysis and to enhance the model's performance. One common technique is image resizing, which standardizes the size of the images across the data set, ensuring uniformity for efficient processing by the deep learning model. Another important method is data augmentation, which includes applying random rotations, cropping, and varying scaling and zoom levels. These augmentation techniques artificially expand the training data set, simulating real-world acquisition conditions and reducing overfitting, thus making the model more robust to variations in image capture.11 Many deep learning models incorporate a segmentation and feature extraction step to isolate the region of interest—such as the optic disc and peripapillary area—to allow the model to focus on the most pertinent areas of the image, minimizing background noise and enhancing the detection of relevant features, which ultimately improves diagnostic accuracy. Lastly, normalization of pixel values may be performed to standardize brightness and contrast across the data set, which reduces variability and improves consistency in model predictions. Together, these preprocessing techniques aid in optimizing the performance of deep learning models using imaging data. Deep Learning Architectures Deep learning models are often based on data architectures such as neural networks, which are AI models inspired by the human brain's method of processing information (Fig. 1). These networks are composed of layers of interconnected nodes, or “neurons,” which work together to recognize patterns in the data.12 Each node receives input, applies a mathematical function to process it, and then passes the output to the next layer. This setup allows the network to learn from large data sets by adjusting the connections between nodes based on how accurate its predictions are, similar to how the brain strengthens synapses through learning. In medical imaging, neural networks, such as convolutional neural networks (CNNs), are particularly useful for identifying specific patterns, such as subtle changes in the optic nerve or retina, which might be overlooked by the human eye. The encoder in a model architecture processes raw data into a condensed form that captures important features, while the classifier uses this encoded data to categorize inputs into predefined classes. Maintaining class balance, where each class is adequately represented, prevents the model from being biased toward more frequent classes.FIG. 1.: Comparison of the human brain network and artificial neural network. A. Schematic representation of a biologic network within the human brain. The simplified model shows 2 neurons connected in series. However, neural networks may be far more complex. B. Illustration of an artificial network with input nodes, multiple hidden layers, and output nodes (Figure created by BioRender).Commonly used architectures include U-Net, effective for image segmentation tasks, and ResNet, which improves image classification by using a series of layers that learn progressively complex features. Often, these models are pretrained on large data sets such as ImageNet, which provides a foundation of generalized knowledge that helps them quickly adapt to specific tasks with smaller data sets, such as neuro-ophthalmic imaging disease classification. In the studies by Bénard-Séguin et al and Sathianvichitr et al, both research teams clearly defined their imaging modality and methods, and effectively used preprocessing techniques. Bénard-Séguin et al used a U-Net deep learning model to segment the optic disc images and identify the region of interest. The images were then resized, and a weighted loss function was used to manage imbalance between the MS ON and non-MS ON classes. After image segmentation, they used a pretrained encoder (RETFound) to extract features (a total of 1,024). Next, they trained a classification model (Nu-Support Vector Classification) using the extracted features to classify fundus photographs for MS ON diagnosis vs non-MS ON. Sathianvichitr et al also used a U-Net architecture for segmentation with an encoder (ResNet-34) to identify the region of interest. After image segmentation, they applied various data augmentation techniques such as random rotations, horizontal flipping, warping, zooming, translation shifts, and random dropout of specific input areas. For classification, they used an EfficientNet-B5 network, which was initialized with weights that had been pretrained on the ImageNet data set and fine-tuned for their specific classification task. The classification network was trained on a large number of fundus images (3,357) to automatically classify discs with ODD or papilledema. The detailed image acquisition, preprocessing, and deep learning architecture methodologies described in these 2 studies highlight the meticulous and rigorous approach necessary for developing effective deep learning models. The careful selection and application of preprocessing techniques, the thoughtful design of the neural network architectures, and the strategic use of pretraining and fine-tuning all contribute to the robust performance of their models. These studies exemplify the potential of AI in neuro-ophthalmology and demonstrate the importance of methodological rigor and precision in the development and application of AI-based diagnostic tools. Deep Learning Model Development and Validation Deep learning model development involves splitting the data into training, validation, and testing data sets. For smaller data sets, such as those in neuro-ophthalmic studies, data can be split into training and testing data sets if cross-validation is used at the training level to reduce the risk of overfitting by validating the model across multiple subsets of the training data. Stratified sampling ensures that each subset of data is representative of the overall data set, maintaining a balance of different classes, such as disease presence, across training, validation, and test sets to prevent bias. In neuro-ophthalmic studies using 2 eyes for each patient, it is important to keep eyes from the same patients in the same data split to avoid overestimating the model's performance. Parameters are the adjustable settings within a model, such as weights in a neural network, that the model learns from the data during training, and tuning these parameters is crucial for optimal performance. Bénard-Séguin et al split their data set into an 80%–20% training-test ratio, using stratified sampling to maintain consistent patient proportions in both the training and test data sets. They used 5-fold cross-validation to identify the optimal model parameters. Sathianvichitr et al divided their primary data set into an 80%–20% training/validation-internal testing ratio. Both eyes of each patient were placed in the same data split to avoid overfitting. And they used a center-based stratified split to reduce bias. They also then tested their data on an external test data set, previously unseen by the model, which allowed them to evaluate the model's performance and generalizability beyond the training and validation data sets. Accurate model training also depends on gold standard or ground truth13 definitions that involve using the best available criteria to label data accurately and is crucial in developing a diagnostic model as it provides a reference point against which the performance of the model can be measured. Bénard-Séguin et al used medical records, imaging results, and ancillary testing as their gold standard for diagnosing ON subtypes. Sathianvichitr et al, meanwhile, used clinically confirmed intracranial hypertension for diagnosing papilledema and a combination of imaging modalities for diagnosing optic disc drusen. Data curation is the process of resolving discrepancies between raters evaluating the outcome, such as disease diagnosis, for individual cases. Sathianvichitr et al resolved discordant severity ratings through review by 2 neuro-ophthalmologists after their initial review by 2 expert neuro-ophthalmologists. Models might focus on classification of disease or disease severity, distinguishing between whether a disease is present or categorizes its severity. This relates to the type of classification the model performs—binary (predicting 1 of 2 possible outcomes, such as “disease” or “no disease”) or multiclass (predicting 1 outcome from multiple categories, such as different stages of a disease). Deep learning models can also use regression to predict continuous outcomes, however, both articles used classification models. Both Bénard-Séguin et al and Sathianvichitr et al focused on binary classification in their studies, distinguishing between the presence of different neuro-ophthalmic conditions. In addition, Sathianvichitr et al made multiple comparisons based on ODD appearance and grading of papilledema. Model Performance Evaluation Metrics commonly used for evaluating performance of deep learning classification models include the area under the receiver operating characteristic curve (AUC), sensitivity, specificity, accuracy, positive predictive value (PPV), and negative predictive value (NPV). AUC is a comprehensive metric used to evaluate model performance. It considers sensitivity, which is the model's ability to correctly identify positive cases. It also considers specificity, which measures the model's ability to correctly identify negative cases.13 Accuracy measures the proportion of total predictions that are correct. The overall classification performance results from Sathianvichitr et al demonstrated a high AUC of 0.97 (95% confidence interval [CI], 0.96–0.98), indicating that the model has a high measure of separability and is capable of distinguishing between optic discs with ODD and optic discs with papilledema. The sensitivity of the model was 86.0% (95% CI, 82.1%–90.1%), indicating that the model was able to correctly identify 86.0% of the positive cases. The specificity was 94.9% (95% CI, 92.3%–97.6%), suggesting that the model correctly identified 94.9% of the negative cases. These high values suggest that the model has a strong performance in both identifying true positive cases and avoiding false positive cases, making it a reliable tool for disease prediction. In the study by Bénard-Séguin et al, the model's performance was evaluated using a test data set that was held out from the training process and previously unseen by the model, and resulted in an AUC of 0.83 (95% CI, 0.72–0.92). This high AUC suggests that the model is effective in differentiating between the groups (MS ON and non-MS ON) using fundus photographs. The model's accuracy was reported to be 76.2% (95% CI, 68.4–83.1), indicating that it correctly predicted the disease status in about three-quarters of the cases. The model's sensitivity and specificity were 74.2% (95% CI, 55.9–87.4) and 76.9% (95% CI, 67.6–85.0), respectively. These values indicate that the model was able to correctly identify approximately 74.2% of the true positive cases and correctly exclude approximately 76.9% of the true negative cases, demonstrating its robust performance in both detecting true cases and avoiding false alarms. In this study, additional metrics were incorporated: PPV and NPV. PPV represents the likelihood that individuals with a positive screening test actually have the disease, whereas NPV indicates the likelihood that individuals with a negative screening test truly do not have the disease. These metrics provide a more comprehensive understanding of the model's performance, especially in a clinical setting where false positives and false negatives have significant implications. The model's PPV and NPV were 59.6% (95% CI, 33.0–79.2) and 86.1% (95% CI, 74.6–95.5), respectively. These metrics are particularly important in a clinical setting as they provide insight into the model's reliability in predicting true positive and true negative cases, which is crucial in avoiding misdiagnosis and unnecessary treatments. The relatively high NPV suggests that the model is particularly reliable in ruling out non-MS ON when the test result is negative. Bootstrapping for estimating confidence intervals is a resampling method that involves repeatedly sampling a data set to estimate population statistics. In both the studies by Sathianvichitr et al and Bénard-Séguin et al, bootstrapping was used to calculate 95% confident intervals. Heatmaps are a powerful tool for providing transparency into AI models. They visually represent the areas in the image that the model considers important for making its prediction. This is particularly useful in neuro-ophthalmology where the model can highlight areas of the eye that are indicative of a disease. Sathianvichitr et al used heatmaps to visualize the areas of the eye that the model focused on, providing a level of interpretability to the model's predictions. This not only enhances the transparency of the model but also provides clinicians with valuable insights into the disease's manifestation in the eye. Strengths and Pitfalls of Deep Learning Models Deep learning models have emerged as powerful tools in neuro-ophthalmology because of their capacity to handle complex tasks and large data sets such as imaging data.14 A key advantage of deep learning models is their capacity to handle and learn from large volumes of data, allowing them to detect complex patterns and relationships that may not be accessible to traditional statistical models or even viewable by the human eye. This capability is particularly beneficial in fields such as neuro-ophthalmology, where the data are often high-dimensional and intricate. Recent advancements have demonstrated the significant potential of deep learning models in neuro-ophthalmology. For example, a deep learning system has been created to accurately detect pediatric papilledema using standard ocular fundus photographs11 providing a much-needed diagnostic tool for pediatric neuro-ophthalmology where detecting such conditions can be particularly challenging. Another AI application has proven effective in the emergency room setting, where nonmydriatic fundus photographs are used to detect optic nerve abnormalities,15 thereby enhancing early diagnostic capabilities and facilitating quicker patient management in acute scenarios. Furthermore, deep learning models have shown their utility in identifying subtle retinal changes associated with neurodegenerative diseases such as Alzheimer disease.16 These studies underscore the transformative potential of AI in both expanding the diagnostic capabilities in neuro-ophthalmology and bridging gaps in care. However, these models also have limitations.17 They require substantial amounts of high-quality data for training, which can be challenging to obtain in many real-world settings. In addition, these models are often perceived as “black boxes,” as it can be difficult to understand or interpret the model processes. The lack of transparency in these models can pose a significant issue in health care, where understanding the rationale behind a diagnosis or is Furthermore, and are associated with the use of The lack of in training data can in and the of models is a as they need to be and as data In the studies by Sathianvichitr et al and Bénard-Séguin et al, these and are The used in the study by Sathianvichitr et al enhances the generalizability of the model, and the inclusion criteria ensure the quality of the data. However, both studies used retrospective data and which may and the generalizability of the The high-quality data used in the studies may not real-world particularly disease The study by Bénard-Séguin et al had a relatively size and an imbalance in the number of cases in each which can the and performance of the model. However, techniques such as cross-validation and stratified sampling were used to these It is crucial to understand that deep learning models are to not the of clinical They are capable of and accurately processing large volumes of data, making them valuable tools for tasks such as screening large for disease, in diagnosis, or predicting disease They can provide a valuable potential for or help cases based on predicted However, these models do not the clinical and that human clinicians to patient care. They the complexity of a medical or in the a human They also do not have the ability to with provide or make complex In addition, the of deep learning models depends on the quality of the data they are trained data used to the model are biased or the model can these biases. is necessary to ensure the quality of the data, interpret the model's predictions in the context of each individual patient, and make the clinical AND In the studies by Bénard-Séguin et al and Sathianvichitr et al provide for the potential of AI in neuro-ophthalmology. research is to and their models and is to understand and avoid potential their work for the use of AI in diagnosing and diseases that in the eye. research should focus on addressing the in these studies and the potential of AI in differentiating other optic disc and using other imaging tools.