没有合适的资源?快使用搜索试试~ 我知道了~
首页LSTM-RNN在句子嵌入中的应用与信息检索分析
本文主要探讨了"深度句子嵌入使用长短期记忆网络"(Deep Sentence Embedding Using Long Short-Term Memory Networks),这是当前自然语言处理领域的一个热门研究课题。论文提出了一种基于循环神经网络(Recurrent Neural Networks, RNN)特别是长短期记忆(Long Short-Term Memory, LSTM)单元的模型。LSTM-RNN模型的工作原理是逐个处理句子中的每个词,提取其语义信息,并将其转换成一个具有语义含义的向量。LSTM的优势在于其能够捕获长期依赖性,这意味着在处理整个句子时,模型会逐渐积累更丰富的上下文信息。 在训练过程中,该模型是弱监督学习的方式,利用商业搜索引擎记录的用户点击数据进行指导。这种学习方法使得模型能够在没有明确标注的情况下,学习到文本中的关键信息。作者通过可视化和分析来深入理解句子嵌入过程的工作机制,发现模型能自动忽略不重要的词语,并能有效地识别出句子中的关键主题或重点部分。 实验结果表明,LSTM-RNN模型在信息检索等自然语言处理任务中表现出色,其生成的语义表示不仅有助于提高文本相似度计算的准确性,还能为诸如文本分类、情感分析等下游任务提供高质量的输入。这种模型的应用潜力巨大,不仅可以提升搜索引擎的用户体验,还有望在机器翻译、对话系统等领域发挥重要作用。 这篇论文对深度学习在自然语言处理中的应用进行了深入研究,特别是在利用LSTM技术生成高质量句子嵌入方面,为我们提供了有价值的理论基础和技术手段。未来的研究可以进一步优化模型结构,提高效率,并探索更多的应用场景,推动自然语言处理技术的发展。
资源详情
资源推荐
![](https://csdnimg.cn/release/download_crawler_static/9768030/bg5.jpg)
5
y
Q
(T
Q
)
y
D
+
(T
D
+
)
y
D
1
(T
D
1
)
y
D
n
(T
D
n
)
Click:'1'
…'
0'
0'
Nega/ve'samples'
Posi/ve'sample'
Fig. 3. The click-through signal can be used as a (binary) indication
of the semantic similarity between the sentence on the query side and
the sentence on the document side. The negative samples are randomly
sampled from the training data.
tasks where two sentences belong to two different lan-
guages with similar meanings and we want to make their
semantic representation as close as possible.
We now describe how to train the model to achieve the
above objective using the click-through data logged by a
commercial search engine. For a complete description of
the click-through data please refer to section 2 in [32].
To begin with, we adopt the cosine similarity between
the semantic vectors of two sentences as a measure for
their similarity:
R(Q, D) =
y
Q
(T
Q
)
T
y
D
(T
D
)
ky
Q
(T
Q
)k · ky
D
(T
D
)k
(3)
where T
Q
and T
D
are the lengths of the sentence
Q and sentence D, respectively. In the context of
training over click-through data, we will use Q and
D to denote “query” and “document”, respectively.
In Figure 3, we show the sentence embedding vec-
tors corresponding to the query, y
Q
(T
Q
), and all the
documents, {y
D
+
(T
D
+
), y
D
−
1
(T
D
−
1
), . . . , y
D
−
n
(T
D
−
n
)},
where the subscript D
+
denotes the (clicked) positive
sample among the documents, and the subscript D
−
j
denotes the j-th (un-clicked) negative sample. All these
embedding vectors are generated by feeding the sen-
tences into the RNN or LSTM-RNN model described
in Sec. III and take the y corresponding to the last word
— see the blue box in Figure 1.
We want to maximize the likelihood of the clicked
document given query, which can be formulated as the
following optimization problem:
L(Λ) = min
Λ
(
− log
N
Y
r=1
P (D
+
r
|Q
r
)
)
= min
Λ
N
X
r=1
l
r
(Λ)
(4)
where Λ denotes the collection of the model parameters;
in regular RNN case, it includes W
rec
and W in Figure
1, and in LSTM-RNN case, it includes W
1
, W
2
, W
3
,
W
4
, W
rec1
, W
rec2
, W
rec3
, W
rec4
, W
p1
, W
p2
, W
p3
,
b
1
, b
2
, b
3
and b
4
in Figure 2. D
+
r
is the clicked
document for r-th query, P (D
+
r
|Q
r
) is the probability
of clicked document given the r-th query, N is number
of query / clicked-document pairs in the corpus and
l
r
(Λ) = − log
e
γR(Q
r
,D
+
r
)
e
γR(Q
r
,D
+
r
)
+
P
n
i=j
e
γR(Q
r
,D
−
r,j
)
= log
1 +
n
X
j=1
e
−γ·∆
r,j
(5)
where ∆
r,j
= R(Q
r
, D
+
r
) − R(Q
r
, D
−
r,j
), R(·, ·) was
defined earlier in (3), D
−
r,j
is the j-th negative candidate
document for r-th query and n denotes the number of
negative samples used during training.
The expression in (5) is a logistic loss over ∆
r,j
.
It upper-bounds the pairwise accuracy, i.e., the 0 - 1
loss. Since the similarity measure is the cosine function,
∆
r,j
∈ [−2, 2]. To have a larger range for ∆
r,j
, we use
γ for scaling. It helps to penalize the prediction error
more. Its value is set empirically by experiments on a
held out dataset.
To train the RNN and LSTM-RNN, we use Back Prop-
agation Through Time (BPTT). The update equations for
parameter Λ at epoch k are as follows:
4Λ
k
= Λ
k
− Λ
k−1
4Λ
k
= µ
k−1
4Λ
k−1
−
k−1
∇L(Λ
k−1
+ µ
k−1
4Λ
k−1
)
(6)
where ∇L(·) is the gradient of the cost function in (4),
is the learning rate and µ
k
is a momentum parameter
determined by the scheduling scheme used for training.
Above equations are equivalent to Nesterov method
in [33]. To see why, please refer to appendix A.1 of
[34] where Nesterov method is derived as a momentum
method. The gradient of the cost function, ∇L(Λ), is:
∇L(Λ) = −
N
X
r=1
n
X
j=1
T
X
τ=0
α
r,j
∂∆
r,j,τ
∂Λ
| {z }
one large update
(7)
where T is the number of time steps that we unfold the
network over time and
α
r,j
=
−γe
−γ∆
r,j
1 +
P
n
j=1
e
−γ∆
r,j
. (8)
∂∆
r,j,τ
∂Λ
in (7) and error signals for different param-
eters of RNN and LSTM-RNN that are necessary for
training are presented in Appendix A. Full derivation of
gradients in both models is presented in section III of
supplementary materials.
To accelerate training by parallelization, we use mini-
batch training and one large update instead of incremen-
tal updates during back propagation through time. To
resolve the gradient explosion problem we use gradient
剩余24页未读,继续阅读
![pdf](https://img-home.csdnimg.cn/images/20210720083512.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://profile-avatar.csdnimg.cn/f1cd3282997940929250f83e9749dffb_ken_henderson.jpg!1)
ken_henderson
- 粉丝: 2
- 资源: 11
上传资源 快速赚钱
我的内容管理 展开
我的资源 快来上传第一个资源
我的收益
登录查看自己的收益我的积分 登录查看自己的积分
我的C币 登录后查看C币余额
我的收藏
我的下载
下载帮助
![](https://csdnimg.cn/release/wenkucmsfe/public/img/voice.245cc511.png)
最新资源
- 多模态联合稀疏表示在视频目标跟踪中的应用
- Kubernetes资源管控与Gardener开源软件实践解析
- MPI集群监控与负载平衡策略
- 自动化PHP安全漏洞检测:静态代码分析与数据流方法
- 青苔数据CEO程永:技术生态与阿里云开放创新
- 制造业转型: HyperX引领企业上云策略
- 赵维五分享:航空工业电子采购上云实战与运维策略
- 单片机控制的LED点阵显示屏设计及其实现
- 驻云科技李俊涛:AI驱动的云上服务新趋势与挑战
- 6LoWPAN物联网边界路由器:设计与实现
- 猩便利工程师仲小玉:Terraform云资源管理最佳实践与团队协作
- 类差分度改进的互信息特征选择提升文本分类性能
- VERITAS与阿里云合作的混合云转型与数据保护方案
- 云制造中的生产线仿真模型设计与虚拟化研究
- 汪洋在PostgresChina2018分享:高可用 PostgreSQL 工具与架构设计
- 2018 PostgresChina大会:阿里云时空引擎Ganos在PostgreSQL中的创新应用与多模型存储
资源上传下载、课程学习等过程中有任何疑问或建议,欢迎提出宝贵意见哦~我们会及时处理!
点击此处反馈
![](https://img-home.csdnimg.cn/images/20220527035711.png)
![](https://img-home.csdnimg.cn/images/20220527035711.png)
![](https://img-home.csdnimg.cn/images/20220527035111.png)
安全验证
文档复制为VIP权益,开通VIP直接复制
![](https://csdnimg.cn/release/wenkucmsfe/public/img/green-success.6a4acb44.png)