Human-aided Explainable AI Classification of Forward-Looking Sonar Imagery

We develop a human-machine teaming approach to increase the accuracy of a convolutional neural network (CNN) used to classify forward-looking sonar images of objects. The developed approach is low complexity, targeting mid-mission collaboration between an autonomous underwater vehicle and human diver. Images automatically classified by the CNN with low certainty are passed to the human, who identifies the most relevant parts of the images. Human feedback is incorporated into the CNN via transfer learning. Fine-tuning the CNN with human feedback increases the overall accuracy by 7 percentage points, and the class-wise accuracy by as much as 75 percentage points. Feature attribution methods confirm that improvements in the CNN classification after human feedback is due to refocusing on the areas with information about the objects.