J. Liao, S. Wang and D. Li / Knowledge-Based Systems 165 (2019) 197–207 199
2.2. Representation learning
Recently, representation learning has attracted a large amount
of attention in NLP research. Many neural-based representation
learning methods have been proposed to encode the semantics of
words, sentences, documents or the relations in low-dimensional
embeddings.
For semantic relation modeling, Bordes et al. [17] proposed
the TransE model to learn the representation of entities and re-
lations, and the authors regarded the relations between entities
as a translation embedding. This model achieved amazing perfor-
mance in knowledge graph research. Wang et al. [18] overcame the
weakness of TransE, which is only adaptive for one-to-one rela-
tions, and the authors proposed the TransH model, which added
a hyperplane vector to project the entities to different spaces and
effectively increased the accuracy. Similar to Wang’s work, Lin et al.
[19] remolded the TransE model by adding a projection matrix to
project entities from entity space to relation space, and the new
TransR model performed better than the two former models. Based
on this study, they later categorized existing knowledge graph
relations into attributes and relations, and they proposed a new
knowledge representation model that has entities, attributes and
relations [20]. However, this model aimed to learn the semantic
relations between the entities while neglecting the necessity and
effectiveness of the language expression, which was proven to be a
significant feature when addressing aspect extraction, as shown in
many previous studies. Nguyen et al. [21] combined insights from
previous link prediction models into a new embedding model,
STransE, which is a combination of structure embedding and the
TransE model. They projected each entity to a special matrix that
can be used to transfer the head entities and tail entities to different
spaces. Poria et al. [22] used a deep CNN combined with a set
of linguistic patterns to tag each word in opinion sentences as
either an aspect or non-aspect word. Trouillon et al. [23] make
use of complex valued embeddings. The composition of complex
embeddings can handle a large variety of binary relations, among
them symmetric and antisymmetric relations. Shi and Weninger
[24] present a shared variable neural network model called ProjE
that fills-in missing information in a knowledge graph by learning
joint embeddings of the knowledge graph’s entities and edges. Shi
and Weninger [25] introduce an open-world KGC model called
ConMask. This model learns embeddings of the entity’s name and
parts of its text-description to connect unseen entities to the KG.
For sentence modeling, CNNs [26,27] and recursive neural net-
works (RNNs) [28,29] are widely used methods that have achieved
amazing performance in many sentence embedding learning tasks.
In the CNN model, a fixed width window feature detector is
adopted to slide over the sentence for feature extraction. It can
effectively extract the ‘‘local’’ features that appear in the same
convolution window. RNNs are built upon a binary tree and regard
words as leaf nodes and sentence constituency relations as non-
leaf nodes. RNNs can encode structure information but may have
problems if the path of dependency is too long [30]. To combine
the advantages of the two models, Lili et al. [31,32] proposed
a tree-based CNN (TBCNN) model to embed the constituency or
dependency tree structure into a representation of sentences. The
TBCNN model can extract sentences’ structural features through
short propagation paths between the output layer and underlying
feature detectors, enabling effective structural feature learning
and extraction. However, in the TBCNN model, they totally dis-
card the word sequence information. Moreover, they initialize
the weight matrix of the constituency/dependency relation in the
convolutional layer randomly, which may fail to include some
significant related information. In this paper, we propose SDT-
CNN to overcome the weakness of TBCNN and use it to learn the
structure-embedded representation of sentences.
3. Characteristic analysis and formal definition of fact-implied
implicit sentiment
Because there is no explicit sentiment word as a sentiment
clue for identification, traditional dictionary-based methods are
no longer effective. We need to find other features for implicit
sentiment analysis.
In this section, we detailed analysis the characteristic and give
a formal definition of fact-implied implicit sentiment.
3.1. Characteristic analysis
During our implicit sentiment corpus construction, we analyzed
and summarized the characteristics of fact-implied implicit sen-
timent expressions in detail. When a sentence expresses implicit
sentiment through an implied factual statement, it usually has the
following characteristics.
(1) Sentiment polarity consistency between the context seman-
tic background and fact-implied implicit sentence.
Chen and Chen [7] found that a sentence containing implicit
sentiment usually has the same sentiment polarity as its context.
During our corpus annotation, we also found fact-implied implicit
sentiment sentences to be consistent with their context explicit
ones. Take the following weibo
1
as an example. E3-1 is a fact-
implied implicit sentiment sentence that expresses negative po-
larity, and it shares the same polarity with its explicit sentiment
context E3-2 and E3-3.
E3-1. Before the Dragon Boat Festival, there were reports that a
manufacturer sold zongzi that had expired one year ago, and tens of
thousands of zongzi were removed from the store.[implicit negative]
E3-2. A century-old shop fell down in food safety.[explicit negative]
E3-3. Human greed is everywhere.[explicit negative]
(2) Sentiment target relevance.
The fact, implied in the implicit sentiment sentence, is often
concerning people’s subjective views, appraisals, evaluations, and
feelings. We reference the selectional restrictions violation view
of Wilks [33,34] in detecting metaphors and consider the implied
fact to have relevance, especially violation or conflict with the
sentiment target. Take the following sentences as examples.
E4. The gas station is only 100 m from my home.
E5. The metro station is only 100 m from my home.
E4 and E5 are almost the same except for the sentiment target
(gas station vs. metro station). However, E4 is usually a negative
sentence, while E5 is relatively positive. Living too close to a gas
station often means potential hazards and pollution, while the
metro station can bring more convenience. Such examples demon-
strate that the sentiment target will influence the implicit senti-
ment polarity. There is another type of situation that shows how
the target influences the implicit sentiment strength. Compare the
following two examples:
E6. There are hidden dangers in the food safety of a century-old
restaurant.
E7. There are hidden dangers in the food safety of a sidewalk snack
vendor.
From the subjective cognition of most people, a century-old
restaurant certainly has high standards in food safety, while they
may have higher tolerance for a sidewalk snack vendor. For that
reason, sentence E6 expresses a stronger negative sentiment com-
pared with E7.
(3) Semantic background relevance.
Fact-implied implicit sentiment will also be influenced by its
context semantics or topic background. Take E5 as an example
again. The complete weibo context of E5 is:
E8-1. The metro station is only 100 m from my home.
1
Weibo is a famous social media platform in China, similar to Twitter.