![](https://csdnimg.cn/release/download_crawler_static/89005910/bg3.jpg)
Sentence A Sentence B
BERT BERT
u v
pooling pooling
(u, v, |u-v|)
Softmax classifier
Figure 1: SBERT architecture with classification ob-
jective function, e.g., for fine-tuning on SNLI dataset.
The two BERT networks have tied weights (siamese
network structure).
computed candidate embeddings using attention.
This idea works for finding the highest scoring
sentence in a larger collection. However, poly-
encoders have the drawback that the score function
is not symmetric and the computational overhead
is too large for use-cases like clustering, which
would require O(n
2
) score computations.
Previous neural sentence embedding methods
started the training from a random initialization.
In this publication, we use the pre-trained BERT
and RoBERTa network and only fine-tune it to
yield useful sentence embeddings. This reduces
significantly the needed training time: SBERT can
be tuned in less than 20 minutes, while yielding
better results than comparable sentence embed-
ding methods.
3 Model
SBERT adds a pooling operation to the output
of BERT / RoBERTa to derive a fixed sized sen-
tence embedding. We experiment with three pool-
ing strategies: Using the output of the CLS-token,
computing the mean of all output vectors (MEAN-
strategy), and computing a max-over-time of the
output vectors (MAX-strategy). The default config-
uration is MEAN.
In order to fine-tune BERT / RoBERTa, we cre-
ate siamese and triplet networks (Schroff et al.,
2015) to update the weights such that the produced
sentence embeddings are semantically meaningful
and can be compared with cosine-similarity.
The network structure depends on the available
Sentence A Sentence B
BERT BERT
u v
pooling pooling
cosine-sim(u, v)
-1 … 1
Figure 2: SBERT architecture at inference, for exam-
ple, to compute similarity scores. This architecture is
also used with the regression objective function.
training data. We experiment with the following
structures and objective functions.
Classification Objective Function. We con-
catenate the sentence embeddings u and v with
the element-wise difference |u− v| and multiply it
with the trainable weight W
t
∈ R
3n×k
:
o = softmax(W
t
(u, v, |u − v|))
where n is the dimension of the sentence em-
beddings and k the number of labels. We optimize
cross-entropy loss. This structure is depicted in
Figure 1.
Regression Objective Function. The cosine-
similarity between the two sentence embeddings
u and v is computed (Figure 2). We use mean-
squared-error loss as the objective function.
Triplet Objective Function. Given an anchor
sentence a, a positive sentence p, and a negative
sentence n, triplet loss tunes the network such that
the distance between a and p is smaller than the
distance between a and n. Mathematically, we
minimize the following loss function:
max(||s
a
− s
p
|| − ||s
a
− s
n
|| + , 0)
with s
x
the sentence embedding for a/n/p, || · ||
a distance metric and margin . Margin ensures
that s
p
is at least closer to s
a
than s
n
. As metric
we use Euclidean distance and we set = 1 in our
experiments.
3.1 Training Details
We train SBERT on the combination of the SNLI
(Bowman et al., 2015) and the Multi-Genre NLI