A Graph Convolutional Network with Localized Convolution and Readout Operations for Diagnosing Chest X-Rays Using Radiologist Gaze Data

In this work-in-progress report we propose a Graph Convolutional Network (GCN) capable of diagnosing chest x-rays using radiologist captured data for training. While other neural networks are capable of making inference on medical image exams with gaze data, the examples found in literature use architectures that combine this data with traditional CNNs that learn from the whole image. Our model, on the other hand, learns from a graph of gaze fixations as nodes, each accompanied by a feature vector describing only their region of observation. Such graph is, naturally, euclidean. Traditional convolution and readout operations in GCNs are not conceived to leverage local features and attributes of euclidean graphs, usually aggregating nodes and edges into a whole-graph representation. Our approach divides the graph in a grid, performing such operations in small regions as to preserve local features. With this we aim to prove two hypotheses: 1) a model can learn from specialist gaze data over an image without being paired with the image in its original structure and 2) it is possible to take advantage of euclidean graphs by not aggregating local features in graph convolution and readout layers.