Crowdsourcing Argumentation Structures in Chinese Hotel Reviews
Mengxue Li
†
, Shiqiang Geng
‡
∗, Yang Gao
†
, Shuhua Peng
‡
, Haijing Liu
†
and Hao Wang
†
†
Institute of Software, Chinese Academy of Sciences, University of Chinese Academy of Sciences
‡
School of Automation, Beijing Information Science and Technology University, China
Email: mengxue2015@iscas.ac.cn, {gsqwxh,psh01}@163.com, {gaoyang, haijing2015, wanghao}@iscas.ac.cn
Abstract—Argumentation mining aims at automatically extract-
ing the premises-claim discourse structures in natural language
texts. There is a great demand for argumentation corpora for
customer reviews. However, due to the controversial nature
of the argumentation annotation task, there exist very few
large-scale argumentation corpora for customer reviews. In this
work, we novelly use the crowdsourcing technique to collect
argumentation annotations in Chinese hotel reviews. As the
first Chinese argumentation dataset, our corpus includes 4814
argument component annotations and 411 argument relation
annotations, and its annotations qualities are comparable to
some widely used argumentation corpora in other languages.
1. Introduction
In customer reviews, users usually not only give their
opinions on the products/services, but also provide reasons
supporting their opinions. For example, consider the follow-
ing review excerpt posted on Tripadvisor.com:
Example 1:
1
房间的电器设施让人很失望。
2
有一台很老 很小 的 黑 白 电 视。
3
空 调 也是
坏的。
1
Appalling in room electrical facilities.
2
There was an old, small, black TV.
3
Air
conditioner did not work.
Clause
1
gives the customer’s opinion (or claim) on
the appliances in the room, and clauses
2
and
3
are
reasons/evidences (or premises) supporting the claim. Such
discourse structures are known as arguments [11], and the
techniques for automatically extracting arguments and their
relations (e.g. support/attack) from natural language texts
are known as argumentation mining [10]. Performing ar-
gumentation mining on customer reviews can reveal the
reasons behind users’ opinions, thus can greatly facilitate
the product producers and service providers to figure out
their weaknesses and hence has huge commercial potentials.
There exists a great demand for reliably annotated ar-
gumentation corpora on customer reviews, since they are
required for training supervised-learning-based argumenta-
tion mining techniques. Existing argumentation corpora are
mostly constructed from highly professional genres, e.g.
legal documents [12], persuasive essays [16], newspapers
and court cases [13]. Compared to these genres, customer
∗The first two authors contributed equally to this work and should be
considered co-first authors.
reviews are written by novice users, so their linguistic com-
plexities are usually lower and do not contain much domain
knowledge. As a result, we believe that even non-experts
in argumentation are able to identify the argumentation
structures in customer reviews. Crowdsourcing has been
widely recognised as a reliable and economic method for
some annotating tasks [14]. In this work, we investigate the
applicability of crowdsourcing for argumentation annotation
in Chinese hotel reviews. Specifically, the contributions of
this work are threefold:
• We propose a novel argumentation model
1
for hotel
reviews, which extends the classic “premise-claim”
model, and can be potentially used for defining
argumentation structures in other types of customer
reviews and in other languages.
• We novelly employ crowdsourcing to annotate ar-
gumentation structures (i.e. argument components
and their relations) in Chinese hotel reviews, design
some mechanisms so as to help the workers reduce
their chances of making mistakes, and use a cluster-
ing algorithm to aggregate collected annotations.
• The aggregated annotations are published as a pub-
licly available corpus, and the annotating quality of
the corpus is comparable to the state-of-the-art En-
glish argumentation corpora. Furthermore, because
of the controversial nature of the argumentation
annotation task, we provide a confidence score to
each label, so as to help users understand the contro-
versy degree of each annotation. To the best of our
knowledge, this is the first Chinese argumentation
corpus, and the first use of confidence score in
argumentation corpora.
2. Related Work
We first review existing argumentation corpora for cus-
tomer reviews; in particular, we highlight the argumentation
models they used to define arguments. After that, we review
works on crowdsourcing for argumentation annotation and
some related tasks, e.g. annotating discourse structures.
1. An argumentation model gives the definition of arguments, e.g. what
components an argument is consisting of, what kinds of relations are
allowed between different argument and argument components.
2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC)
Banff Center, Banff, Canada, October 5-8, 2017
978-1-5386-1645-1/17/$31.00 ©2017 IEEE 87