![](https://csdnimg.cn/release/download_crawler_static/88789678/bg1.jpg)
Faithful and Customizable Explanations of Black Box Models
Himabindu Lakkaraju
Harvard University
hlakkaraju@hbs.edu
Ece Kamar
Microsoft Research
eckamar@microsoft.com
Rich Caruana
Microsoft Research
rcaruana@microsoft.com
Jure Leskovec
Stanford University
jure@cs.stanford.edu
ABSTRACT
As predictive models increasingly assist human experts (e.g., doc-
tors) in day-to-day decision making, it is crucial for experts to be
able to explore and understand how such models behave in dier-
ent feature subspaces in order to know if and when to trust them.
To this end, we propose Model Understanding through Subspace
Explanations (MUSE), a novel model agnostic framework which
facilitates understanding of a given black box model by explaining
how it behaves in subspaces characterized by certain features of
interest. Our framework provides end users (e.g., doctors) with the
exibility of customizing the model explanations by allowing them
to input the features of interest. The construction of explanations is
guided by a novel objective function that we propose to simultane-
ously optimize for delity to the original model, unambiguity and
interpretability of the explanation. More specically, our objective
allows us to learn, with optimality guarantees, a small number of
compact decision sets each of which captures the behavior of a given
black box model in unambiguous, well-dened regions of the fea-
ture space. Experimental evaluation with real-world datasets and
user studies demonstrate that our approach can generate customiz-
able, highly compact, easy-to-understand, yet accurate explanations
of various kinds of predictive models compared to state-of-the-art
baselines.
CCS CONCEPTS
• Computing methodologies →
Supervised learning; Cost-sensitive
learning.
KEYWORDS
Interpretable machine learning, Decision making, Black box models
ACM Reference Format:
Himabindu Lakkaraju, Ece Kamar, Rich Caruana, and Jure Leskovec. 2019.
Faithful and Customizable Explanations of Black Box Models. In AAAI/ACM
Conference on AI, Ethics, and Society (AIES ’19), January 27–28, 2019, Honolulu,
HI, USA. ACM, New York, NY, USA, 8 pages. https://doi.org/10.1145/3306618.
3314229
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for prot or commercial advantage and that copies bear this notice and the full citation
on the rst page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish,
to post on servers or to redistribute to lists, requires prior specic permission and/or a
fee. Request permissions from permissions@acm.org.
AIES ’19, January 27–28, 2019, Honolulu, HI, USA
© 2019 Association for Computing Machinery.
ACM ISBN 978-1-4503-6324-2/19/01.. . $15.00
https://doi.org/10.1145/3306618.3314229
1 INTRODUCTION
The successful adoption of predictive models for real world decision
making hinges on how much decision makers (e.g., doctors, judges)
can understand and trust their functionality. Only if decision makers
have a clear understanding of the behavior of predictive models,
they can evaluate when and how much to depend on these models,
detect potential biases in them, and develop strategies for further
model renement. However, the increasing complexity and the
proprietary nature of predictive models employed today is making
this problem harder [
9
], thus, emphasizing the need for tools which
can explain these complex black boxes in a faithful and interpretable
manner.
Prior research on explaining black box models can be catego-
rized as: 1) Local explanations, which focus on explaining individual
predictions of a given black box classier [
4
,
9
,
10
] and 2) Global
explanations, which focus on explaining model behavior as a whole,
often by summarizing complex models using simpler, more inter-
pretable approximations such as decision sets or lists [
5
,
7
]. In this
paper, we focus on a new form of explanation that is designed to
help end users (e.g., decision makers such as judges, doctors) gain
deeper understanding of model behavior: a dierential explanation
that describes how the model logic varies across dierent subspaces
of interest in a faithful and interpretable fashion. To illustrate, let us
consider a scenario where a doctor is trying to understand a model
which predicts if a given patient has depression or not. The doctor
might be keen on understanding how the model makes predictions
for dierent patient subgroups (See Figure 1 left). Furthermore,
she might be interested in asking questions such as "how does
the model make predictions on patient subgroups associated with
dierent values of exercise and smoking?" and might like to see
explanations customized to her interest (See Figure 1 right). The
problem of constructing such explanations has not been studied by
previous research aimed at understanding black box models.
Here, we propose a novel framework, Model Understanding
through Subspace Explanations (MUSE), which constructs global
explanations of black box classiers which highlight their behav-
ior in subspaces characterized by features of user interest. To the
best of our knowledge, this is the rst work to study the notion of
incorporating user input when generating explanations of black
box classiers while successfully trading o notions of delity, un-
ambiguity and interpretability. Our framework takes as input a
dataset of instances with semantically meaningful or interpretable
features (e.g. age, gender), and the corresponding class labels as-
signed by the black box model. It also accepts as an optional input
a set of features that are of interest to the end user in order to
Session 2: Norms and Explanations
AIES’19, January 27–28, 2019, Honolulu, HI, USA