SCHOLIA- An XAI Framework for APR

Automated Program Repair (APR) can assist developers by automatically generating patches for buggy code. However, as recent techniques leverage deep learning models, developers do not know why the model generated a particular patch. Existing Explainable AI (XAI) techniques, such as SHAP, can be applied to APR, however, their complexity raises questions about whether developers find such explanations understandable. In this work, we develop a novel framework Scholia, with two extensions to feature attribution methods to make them more understandable to the developers. First generates a text explanation based on attribution scores. Second creates a visualization capturing the transformation of the patch based on the impact of code tokens, named patch transformation. We evaluated the proposed new two explanations types compared to SHAP, using a user survey. The survey received responses from 106 participants. Accordingly, 68.9% <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$(\mathrm{P} &lt;. 05)$</tex> and 64.2% <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$(\mathrm{P} &lt;. 05)$</tex> of participants agreed that text explanation and patch transformation methods are easy to understand, while only 17.9% <tex xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">$(\mathrm{P} &lt;. 05)$</tex> agreed with the original SHAP explanation. The survey responses indicate, with statistical significance, that our extensions to SHAP are easier to understand than the original SHAP explanations.