Sentence A Sentence B
BERT BERT
u v
pooling pooling
(u, v, |u-v|)
Softmax classifier
Figure 1: SBERT architecture with classification ob-
jective function, e.g., for fine-tuning on SNLI dataset.
The two BERT networks have tied weights (siamese
network structure).
computed candidate embeddings using attention.
This idea works for finding the highest scoring
sentence in a larger collection. However, poly-
encoders have the drawback that the score function
is not symmetric and the computational overhead
is too large for use-cases like clustering, which
would require O(n
2
) score computations.
Previous neural sentence embedding methods
started the training from a random initialization.
In this publication, we use the pre-trained BERT
and RoBERTa network and only fine-tune it to
yield useful sentence embeddings. This reduces
significantly the needed training time: SBERT can
be tuned in less than 20 minutes, while yielding
better results than comparable sentence embed-
ding methods.
3 Model
SBERT adds a pooling operation to the output
of BERT / RoBERTa to derive a fixed sized sen-
tence embedding. We experiment with three pool-
ing strategies: Using the output of the CLS-token,
computing the mean of all output vectors (MEAN-
strategy), and computing a max-over-time of the
output vectors (MAX-strategy). The default config-
uration is MEAN.
In order to fine-tune BERT / RoBERTa, we cre-
ate siamese and triplet networks (Schroff et al.,
2015) to update the weights such that the produced
sentence embeddings are semantically meaningful
and can be compared with cosine-similarity.
The network structure depends on the available
Sentence A Sentence B
BERT BERT
u v
pooling pooling
cosine-sim(u, v)
-1 … 1
Figure 2: SBERT architecture at inference, for exam-
ple, to compute similarity scores. This architecture is
also used with the regression objective function.
training data. We experiment with the following
structures and objective functions.
Classification Objective Function. We con-
catenate the sentence embeddings u and v with
the element-wise difference |u− v| and multiply it
with the trainable weight W
t
∈ R
3n×k
:
o = softmax(W
t
(u, v, |u − v|))
where n is the dimension of the sentence em-
beddings and k the number of labels. We optimize
cross-entropy loss. This structure is depicted in
Figure 1.
Regression Objective Function. The cosine-
similarity between the two sentence embeddings
u and v is computed (Figure 2). We use mean-
squared-error loss as the objective function.
Triplet Objective Function. Given an anchor
sentence a, a positive sentence p, and a negative
sentence n, triplet loss tunes the network such that
the distance between a and p is smaller than the
distance between a and n. Mathematically, we
minimize the following loss function:
max(||s
a
− s
p
|| − ||s
a
− s
n
|| + , 0)
with s
x
the sentence embedding for a/n/p, || · ||
a distance metric and margin . Margin ensures
that s
p
is at least closer to s
a
than s
n
. As metric
we use Euclidean distance and we set = 1 in our
experiments.
3.1 Training Details
We train SBERT on the combination of the SNLI
(Bowman et al., 2015) and the Multi-Genre NLI
剩余10页未读，继续阅读
林戈的IT生涯
- 粉丝: 8779
- 资源: 110
我的内容管理
展开
最新资源
- Hadoop生态系统与MapReduce详解
- MDS系列三相整流桥模块技术规格与特性
- MFC编程：指针与句柄获取全面解析
- LM06：多模4G高速数据模块，支持GSM至TD-LTE
- 使用Gradle与Nexus构建私有仓库
- JAVA编程规范指南：命名规则与文件样式
- EMC VNX5500 存储系统日常维护指南
- 大数据驱动的互联网用户体验深度管理策略
- 改进型Booth算法：32位浮点阵列乘法器的高速设计与算法比较
- H3CNE网络认证重点知识整理
- Linux环境下MongoDB的详细安装教程
- 压缩文法的等价变换与多余规则删除
- BRMS入门指南：JBOSS安装与基础操作详解
- Win7环境下Android开发环境配置全攻略
- SHT10 C语言程序与LCD1602显示实例及精度校准
- 反垃圾邮件技术：现状与前景
资源上传下载、课程学习等过程中有任何疑问或建议，欢迎提出宝贵意见哦~我们会及时处理！
点击此处反馈
