![](https://csdnimg.cn/release/download_crawler_static/88779758/bg1.jpg)
Findings of the Association for Computational Linguistics: EMNLP 2020, pages 489–498
November 16 - 20, 2020.
c
2020 Association for Computational Linguistics
489
ConceptBert: Concept-Aware Representation for Visual Question
Answering
Franc¸ois Gard
`
eres
∗
Ecole Poyltechnique
Paris, France
Maryam Ziaeefard
†
McGill University
Montreal, Canada
Baptiste Abeloos
Thales
Montreal, Canada
Freddy Lecue
Inria, France
Thales, Canada
Abstract
Visual Question Answering (VQA) is a chal-
lenging task that has received increasing at-
tention from both the computer vision and
the natural language processing communities.
Current works in VQA focus on questions
which are answerable by direct analysis of
the question and image alone. We present
a concept-aware algorithm, ConceptBert, for
questions which require common sense, or
basic factual knowledge from external struc-
tured content. Given an image and a ques-
tion in natural language, ConceptBert requires
visual elements of the image and a Knowl-
edge Graph (KG) to infer the correct an-
swer. We introduce a multi-modal represen-
tation which learns a joint Concept-Vision-
Language embedding. We exploit ConceptNet
KG for encoding the common sense knowl-
edge and evaluate our methodology on the Out-
side Knowledge-VQA (OK-VQA) and VQA
datasets. Our code is available at https://
github.com/ZiaMaryam/ConceptBERT
1 Introduction
Visual Question Answering (VQA) was firstly in-
troduced to bridge the gap between natural lan-
guage processing and image understanding appli-
cations in the joint space of vision and language
(Malinowski and Fritz, 2014).
Most VQA benchmarks compute a question rep-
resentation using word embedding techniques and
Recurrent Neural Networks (RNNs), and a set of
object descriptors comprising bounding box coordi-
nates and image features vectors. Word and image
representations are then fused and fed to a network
to train a VQA model. However, these approaches
are practical when no knowledge beyond the visual
content is required.
∗
This work was done when the author was an intern at
Thales. Contact email: francois.garderes@student.ecp.fr
†
This work was done when the author worked at Thales.
Incorporating the external knowledge introduces
several advantages. External knowledge and sup-
porting facts can improve the relational representa-
tion between the objects detected in the image, or
between entities in the question and objects in the
image. It also provides information on how the an-
swer can be derived from the question. Therefore,
the complexity of the questions can be increased
based on the supporting knowledge base.
Organizing the world’s facts and storing them
in a structured database, large scale Knowledge
Bases (KB), have become important resources for
representing the external knowledge. A typical
KB consists of a collection of subject-predicate-
object triplets also known as a fact. A KB in
this form is often called a Knowledge Graph
(KG) (Bollacker et al.) due to its graphical rep-
resentation. The entities are nodes and the rela-
tions are the directed edges that link the nodes.
The triples specify that two entities are connected
by a particular relation, e.g.,
(Shakespeare,
writerOf, Hamlet).
A VQA system that exploits KGs is an emerg-
ing research topic, and is not well-studied. Recent
research has started integrating knowledge-based
methods into VQA models (Wang et al., 2017,
2016; Narasimhan et al., 2018; Narasimhan and
Schwing, 2018; Zhu et al., 2015; Marino et al.,
2019). These methods incorporate the external
knowledge through two approaches: i) they ex-
ploit a set of associated facts for each question pro-
vided in VQA datasets (Narasimhan et al., 2018;
Narasimhan and Schwing, 2018), or ii) they collect
possible search queries for each question-image
pair and use a search API to retrieve the answers
(Wang et al., 2017, 2016; Zhu et al., 2015; Marino
et al., 2019). However, we go one step further and
implement an end-to-end VQA model that is fully
trainable. Our model does not require knowledge
annotations in VQA datasets or search queries.