Beihang-MSRA at SemEval-2017 Task 3: A Ranking System with Neural
Matching Features for Community Question Answering
Wenzheng Feng
†
, Yu Wu
†
, Wei Wu
‡
, Zhoujun Li
†∗
, Ming Zhou
‡
†
State Key Lab of Software Development Environment, Beihang University, Beijing, China
‡
Microsoft Research, Beijing, China
{wuyu,lizj,wenzhengfeng}@buaa.edu.cn {wuwei,mingzhou}@microsoft.com
Abstract
This paper presents the system in
SemEval-2017 Task 3, Community Ques-
tion Answering (CQA). We develop a
ranking system that is capable of captur-
ing semantic relations between text pairs
with little word overlap. In addition to
traditional NLP features, we introduce
several neural network based matching
features which enable our system to mea-
sure text similarity beyond lexicons. Our
system significantly outperforms baseline
methods and holds the second place in
Subtask A and the fifth place in Subtask
B, which demonstrates its efficacy on
answer selection and question retrieval.
1 Introduction
In task 3 of SemEval 2017, participants are
required to address typical problems in mod-
ern CQA forums. We participate two sub-
tasks: question-comment similarity (Subtask A)
and question-question similarity (Subtask B). In
Subtask A, given a question and 10 comments in
its comment thread, one is required to re-rank the
10 comments according to their relevance with the
question. Subtask B gives a question and asks par-
ticipants to re-rank 10 related questions according
to their similarity to the input question.
The challenge of both subtasks is that two natu-
ral language sentences often express similar mean-
ings with different but semantically related words,
which results in semantic gaps between them. To
bridge the semantic gaps, we build a ranking sys-
tem with a variety of features. In addition to tra-
ditional NLP features such as tf-idf (Salton and
Buckley, 1988), the longest common subsequence
(Allison and Dix, 1986), translation models (Jeon
∗
Corresponding Author
et al., 2005), and tree kernels (Schlkopf et al.,
2003; Collins and Duffy, 2002; Moschitti, 2006),
which match sentences based on word overlap,
syntax (tree kenerls), and word-word translations
(translation models), we also introduce neural net-
work based matching models into the system as
features. The neural matching features, includ-
ing a long short term memory network (LSTM)
(Schuster and Paliwal, 1997) and a 2D matching
network which is a variant of our model in (Wu
et al., 2016), can extract high level matching sig-
nals from distributed representations of the sen-
tences and capture their similarity beyond lexi-
cons. We also design some specific features for
each subtask. All the features are combined as
a ranking model by a gradient boosted regression
tree which is implemented by Xgboost (Chen and
Guestrin, 2016). Our system significantly outper-
forms baseline methods on the two subtasks. On
Subtask A, it holds the second place and is compa-
rable with the best system. On Subtask B, it holds
the fifth place. The results demonstrate that our
system can alleviate the semantic gaps in the tasks
of CQA and effectively rank relevant comments
and similar questions to high positions.
2 System Description
Our system is built under a learning to rank frame-
work (Liu et al., 2009). It takes a question and
a group of candidates (comments or related ques-
tions) as input, and outputs a ranking list of the
candidates based on scores of question-candidate
pairs. The ranking scores are calculated in three
steps: text preprocessing, feature extraction, and
feature combination. In preprocessing, we replace
special characters and punctuations with spaces,
normalize all letters to their lowercase, remove
stop-words, and conduct stemming and syntax
analysis. Subsequently, we extract a variety of fea-