Early AI systems were predominantly logical and
symbolic; they performed some form of logical in-
ference and could provide a trace of their inference
steps, which became the basis for explanation. There
was substantial work on making these systems more
explainable, but they fell short of user needs for
comprehension (for example, simply summarizing
the inner workings of a system does not yield a suf-
ficient explanation) and proved too brittle against
real-world complexities.
Recent AI success is due largely to new ML tech-
niques that construct models in their internal repre-
sentations. These include support vector machines,
random forests, probabilistic graphical models, re-
inforcement learning (RL), and deep learning (DL)
neural networks. Although these models exhibit high
performance, they are opaque. As their use has in-
creased, so has research on explainability from the
perspectives of ML (Chakraborty et al. 2017; Ras et al.
2018) and cognitive psychology (Miller 2017). Simi-
larly, many XAI-related workshops have been held re-
cently on ML (for example, the International Conference
on Machine Learning, the Conference on Neural In-
formation Processing Systems), AI (for example, the In-
ternational Joint Conference on Artificial Intelligence),
and HCI (for example, the Conference on Human-
Computer Interaction, Intelligent User Interfaces) con-
ferences, as have special topic meetings related to XAI.
There seems to be an inherent tension between ML
performance (for example, predictive accuracy) and
explainability; often the highest-performing methods
(for example, DL) are the least explainable, and the
most explainable (for example, decision trees) are the
least accurate. Figure 1 illustrates this with a notional
graph of the performance-explainability trade-off for
various ML techniques.
When DARPA formulated the XAI program, it envi-
sioned three broad strategies to improve explainability,
while maintaining a high level of learning performance,
based on promising research at the time (figure 2): deep
explanation, interpretable models, and model induction.
Deep explanation refers to modified or hybrid DL
techniques that learn more explainable features or
representations or that include explanation genera-
tion facilities. Several design choices might produce
more explainable representations (for example, training
data selection, architectural l ayers, loss functions,
regularization, optimization techniques, training
sequences). Researchers have used deconvolutional
networks to visualize convolutional network layers, and
techniques existed for associating semantic concepts
with deep network nodes. Approaches for generating
image captions could be extended to train a second deep
network that generates explanations without explicitly
identifying the original network’s semantic features.
Interpretable models are ML techniques that learn
more structured, interpretable, or causal models. Early
examples included Bayesian rule lists (Letham et al.
2015), Bayesian program learning, learning models of
causal relationships, and use of stochastic grammars
to learn more interpretable structure.
Model induction refers to techniques that experi-
ment with any given ML model— such as a black
box— to infer an approximate explainable model. For
example, the model-agnostic explanation system of
Ribeiro et al. (2016) inferred explanations by ob-
serving and analyzing the input-output behavior of a
black box model.
DARPA used these strategies to categorize a portfolio
of new ML techniques and provide future practitioners
with a wider range of design options covering the
performance-explainability trade space.
XAI Concept and Approach
The XAI program’s goal is to create a suite of new or
modified ML techniques that produce explainable
models that, when combined with effective explanation
techniques, enable end users to understand, appropri-
ately trust, and effectively manage the emerging gen-
eration of AI systems. The target of XAI is an end user
who depends on decisions or recommendations pro-
duced by an AI system, or actions taken by it, and
therefore needs to understand the system’s rationale.
For example, an intelligence analyst who receives rec-
ommendations from a big data analytics system needs
to understand why it recommended certain activity for
further investigation. Similarly, an operator who tasks
an autonomous vehicle to drive a route needs to un-
derstand the system’s decision-making model to ap-
propriately use it in future missions. Figure 3 illustrates
the XAI concept: provide users with explanations that
enable them to understand the system’s overall
strengths and weaknesses, convey an understanding of
how it will behave in future or different situations, and
perhaps permit users to correct the system’smistakes.
This user-centered concept poses interrelated re-
search challenges: (1) how to produce more explain-
able models, (2) how to design explanation interfaces,
and (3) how to understand the psychologic require-
ments for effective explanations. The first two chal-
lenges are being addressed by the 11 XAI research
teams, which are developing new ML techniques to
produce explainable models, and new principles,
strategies, and HCI techniques (for example, visuali-
zation, language understanding, language generation)
to generate effective explanations. The third challenge
is the focus of another XAI research team that is
summarizing, extending, and applying psychologic
theories of explanation.
The XAI program addresses two operationally rel-
evant challenge problem areas (figure 4): data analytics
(classification of events of interest in heteroge neous
multimedia data) and autonomy (decision policies
for autonomous systems). These areas represent
two important ML problem categories (supervised
learning and RL) and Department of Defense in-
terests (intelligence analysis and autonomous systems).
The data analytics challenge was motivated by a
common problem: intelligence analysts are presented
with decisions and recommendations from big data
SUMMER 2019 45
Deep Learning and Security
评论0