36 L. Gui et al. / Knowledge-Based Systems 124 (2017) 34–45
2.2. Sentiment classification in product reviews
If we focus on the sentiment classification in product reviews,
the methods can be categorized as text classification based meth-
ods and users/products modeling based methods.
2.2.1. Text classification based method
In principle, any text classification method can be implemented
in sentiment classification. Training a classifier (such as SVM) with
unigrams, bi-grams and trigrams as features is a strong baseline
for sentiment classification [14] . Beside the text features above, the
sentiment lexicon features or sentiment-specific word embeddings
are also important in sentiment classification [18] .
In recent years, the deep learning based method have achieved
the state-of-the-art performance on this problem. Recursive Neural
Tensor Network [8] , Convolutional Neural Network [7] and Gated
Recurrent Neural Network [9] achieved great success. The Recur-
sive Neural Tensor Network [8] composes words into sentences by
sharing parameters and syntactic structure. Convolutional Neural
Network [7] uses convolutional filter to extract phrase level fea-
tures, then implement a pooling operation to select the most rele-
vant features to model a sentences. In Gated Recurrent Neural Net-
work [9] , sentences can be regard as a sequence of words, and any
sequence can be modeled by the last word and the previous sub
sequence. These deep learning methods achieves state-of-the-art
performance in different data set of sentiment classification task
in product reviews.
However, the text classification based methods do not consider
the user or product information, which is important for sentiment
classification in product reviews.
2.2.2. User modeling based method
Apart from text, user information can also be used in sentiment
classification. Gao et al. [28] designed user specific features to cap-
ture opinion holder leniency. Dong et al. [29] incorporated textual
topics and user-word factors into supervised topic modeling. Hovy
[30] used demographic information in sentiment analysis. Tan et al.
[31] and Hu et al. [32] used user-user relationships for Twitter sen-
timent analysis.
Here, the deep learning method incorporating with user infor-
mation achieves the best performance. For example, Tang et al.
[33] incorporated the user and product information into convo-
lutional neural networks for sentiment analysis. Recently, we also
proposed a method to model users based on the concept of inter-
subjectivity [11] . The basic idea is to learn user embeddings based
on their shared words in reviews which have similar polarities.
However, what has been overlooked in these two methods is that
the meanings of words may be also impacted by users and prod-
ucts (opinion targets) and hence representations of words should
be updated together with user and product representations.
In summary, existing methods mainly considered three types of
user information, namely: (1) personal profiles such as ages, gen-
der, etc., to characterize users; (2) latent topics extracted from text
as a proxy measure of users sharing similar topical interests; (3)
rating patterns of users on product reviews. None of the above
methods considered the subtle interplays between users and the
words used by users sharing similar sentiments.
In this paper, we first propose a heterogeneous network embed-
ding method to represent words, users and products in a unified
embedding space based on the constructed heterogeneous network
built on the word level. As will be shown in our experiments, three
kinds of embeddings learned offer a better performance on the
text classification based method. The incorporating of user/product
embeddings and convolutional neural network improves the user
modeling based method and gives the state-of-the-art results on
three product review datasets in document-level sentiment classi-
fication.
3.
Our approach
Our approach is inspired by the theory in sociology. In this
section, we will first define our problem, and then discuss how
to construct a heterogeneous network from text and other infor-
mation. Next, we will present the network embedding method to
model users, products and words in the same embedding space.
Finally, we will explain how to incorporate the learned representa-
tions into a CNN for sentiment classification.
3.1. Problem setup
In the corpora used in our experiments, we assume there are
a total of D review articles written by | U | users for | P | products.
Here, U and P stand for the set of users or products, | · | means
the size of set. Each review article x
i
∈ X is represented by 3-tuple
consisting of its author (or user) u
i
, the opinion target (or product)
p
i
and the text content d
i
, i.e.,
x
i
= { u
i
, p
i
, d
i
} , i ∈ { 1 , 2 , . . . , D } (1)
Note that a user may post multiple reviews, and a product may
receive reviews from different users.
We also assume that a review document d
i
contains L
i
words,
d
i
= { w
i
1
, w
i
2
, . . . , w
i
L
i
} , where w
i
j
is the j th word in document d
i
.
Given a fixed set of sentiment classes Y = { y
1
, y
2
, . . . , y
Y
} , the goal
of sentiment classification is to training a function F to map re-
views to sentiment classes:
F : X → Y (2)
3.2. Word/user/product representation learning with network
embedding
For learning representations of words, users and products, it is
important to define context for each of them. In traditionally word
representation learning methods, such as continuous bag-of-words
(CBOW) or skip gram, the context for each word is typically de-
fined as a 5-word window (two words before and after the tar-
get word). In our previously proposed user representation learning
method [11] , users should be similar to each other if they share
similar subjective terms. Hence, the context for users is defined
as their shared subjective terms in their reviews. In our work here,
we argue that there exist subtle interplays among words, users and
products. For example, words such as ‘ freezes ’ and ‘ hangs ’ are used
often in negative reviews towards mobile phones. These two words
should carry similar semantic meanings and hence their represen-
tations should be placed in nearby locations in the embedding
space. As such, word, user and product representation learning
should be performed simultaneously to map them into a unified
embedding space.
We propose to first build a heterogeneous network, in which
words, users, products, as well as sentiment labels are vertices and
statistical relations between them are edges. Then we use a net-
work embedding method to learn the distributed representation of
each vertex including words, users and products.
3.2.1. Construction of heterogeneous network
Here, we define the network as a graph: G = { E, V } . The E is the
set of edges and V is the set of vertices, which is the union of all
words, users, and products. We need to learn the weight of each
edge, define as ω(e ) , e = { u, v } , e ∈ E and u, v ∈ V here.
In order to capture the information of users and products, we
have included an additional type of vertices, polarities. Four cate-
gories of relations should be considered in the construction of het-
erogeneous network ( Fig. 1 ):