Extension, critique, and formalization of explanation in machine learning

Oftentimes, it is not sufficient for a machine learning model to achieve good predictive performance; in cases where a model is used in high-stakes decision-making, it is preferable to use an interpretable model. Examples of such applications include medical diagnosis, automated allocation of healthcare benefits, and legal sentencing. In these contexts, the study of explainable machine learning (XML) models is important to our society. This thesis contributes to the study of XML in two main ways. The first part of this thesis focuses on increasing the problem-solving capacity of XML. The problems of constructing intrinsically interpretable deep neural network (DNN) architectures for reinforcement learning (RL) agents solving MDPs with visual or spatio-temporal state-spaces are addressed. Both solutions extend the idea of prototype-based deep neural networks (DNNs), which use case-based reasoning for interpretability.We show that by using CNN-encoded MDPstates as prototypes in a prototype-based model, we can construct RL agents that play Atari games well and use their learned prototypical states to explain their policies. I also show that by using sets of concepts about spatial statistics in a study area as prototypes in a prototype-based model, we can construct interpretable clones of human behavior policies.