The Path to Clinical AI: Evaluating the Generalizability of Tibial Plateau Fracture Detection

The key test for transitioning artificial intelligence (AI) to clinical practice remains ensuring generalizability. Specifically, to be considered trustworthy, models must demonstrate strong performance that is robust to variations in patient, institution, and imaging-specific factors, among other variables. Chen et al. offer one such candidate model for the detection of tibial plateau fractures. Trained on both anteroposterior and lateral radiographs, their model showed noteworthy validation performance, achieving an area under the receiver operating characteristic curve (AUC) of 0.97 and an overall accuracy of 88.2%. Moreover, the model was shown to perform well despite deviations in Schatzker type or fracture location. Collectively, Chen et al. make the case for setting a new bar in the detection of tibial plateau fractures on radiographic images, appearing to outshine other recent works1. This claim, however, is dampened by the lack of a standard benchmark. Specifically, this work and other works employed different datasets, which can vary in their composition of easy and hard cases to diagnose. While not unique to tibial plateau fractures, the lack of open-access datasets precludes a head-to-head comparison of model performance. Nevertheless, Chen et al. were able to demonstrate human-level capabilities, therein providing reassurance of their model’s clinical utility. Indeed, from a clinical perspective, tibial plateau fractures are particularly susceptible to misdiagnosis, in part, due to human error2. In this respect, an automated system promises to aid clinicians as an initial screen, second opinion, or even definitive diagnosis. It is particularly valuable that such performance can be achieved with the use of radiographs, which are fast and readily accessible. In this way, this work has the potential to add substantially to health care. From a machine learning perspective, the major contribution of this work pertains to its dataset, which represents perhaps the largest composition of tibial plateau fracture radiographs to date. It is notable that the data labels (i.e., the diagnoses) were confirmed with computed tomography (CT), which adds confidence to the data quality. However, on a technical level, Chen et al. employed an established off-the-shelf, pre-trained machine vision model that was fine-tuned using standard methodology. This fact, in turn, reinforces an important insight: current AI models, if sufficiently powered, are sophisticated enough today to achieve clinically relevant performance. Indeed, while a pragmatic and performant model, EfficientNet B3 was developed in 20193 and has been supplanted in terms of pure power by larger, more modern architectures. The ultimate question of whether the presented model is ready for clinical application will require additional testing on independent data. There are many reasons why models do not generalize well to new images. Neural networks can suffer from stochastic, non-continuous decision functions4. One consequence of this is that small, imperceptible changes to input data can lead to drastically different, and ultimately incorrect, classifications4. In this respect, radiographs, like other imaging modalities, suffer from measurable inter- and intra-hospital variability5, which must be accounted for when training models. A second common pitfall is that of shortcut learning, wherein a model will rely on unintentional cues present within the input data that highly correlate with its true diagnosis (e.g., the presence of a ruler indicating large melanomas). This “shortcut” invariably precludes the model from needing to learn deterministic features of pathology. Recent work has suggested that shortcut learning emerges as an inherent pitfall of passive data collection and can overestimate generalizability by up to 20%6. Chen et al. attempted to mitigate some of these risks by incorporating 12 years of images from multiple hospitals. However, beyond developing robust datasets, future work could consider a variety of regularization techniques, which collectively promote generalizability7. Moreover, approaches to incorporate explainability (e.g., saliency maps, attention mechanisms) can help to provide insight into why a model is making a particular decision8. While not directly improving robustness, this would facilitate a qualitative guard against spurious classification decisions. Ultimately, the work by Chen et al. has joined the growing chorus of works that have demonstrated that AI can meet the challenges of diagnosis. They further present perhaps the best candidate to date for a clinically ready machine vision model of tibial plateau fractures. However, proving clinical readiness will require large, diverse datasets and reproducible, independent demonstrations of model performance. In this respect, model sharing plus overcoming data silos will be the key innovation toward bringing AI into the clinic.