![](https://csdnimg.cn/release/download_crawler_static/18879733/bg3.jpg)
0100101
1010110
1010111
1001010
S
0101000
0110101
1011010
0101001
1-S
-
1
1
+
-
11
+
-
11
+
-
1
Intra-Modal
Similarity
𝐻
-
11
+
-
1
1
+
-
1
-
11
+
-
1
1
+
-
1
𝐻
-
11
+
-
1
1
+
-
1
-
11
+
-
11
+
-
1
-
11
+
-
11
+
-
1
1
+
-
11
+
-
1
CNN-Based Network
Multilayer Perceptron
Inter-Modal
Similarity
1
+
1
+
1
+
1
+
Hash
Layer
𝐻
𝐻
Hash
Layer
Figure 2: The pipeline of our proposed CMLA for cross-modal hash learning.
data-dependent methods, data-independent methods always require long bits to encode multi-modal
data. Thus, most currently proposed methods are data-dependent ones, which can learn compact
hash codes using partial data as a training set. Collective Matrix Factorization Hashing (CMFH) [
14
]
learns hash codes for different modalities by executing collective matrix factorization of different
views. Brostein et al. [
3
] presented a cross-modal hashing by preserving intra-class similarity via
eigen-decomposition and boosting. Semantics-preserving hashing (SePH) [
28
] produces unified
binary codes by modeling an affinity matrix in a probability distribution while minimizing the
Kullback-Leibler divergence. Most of these methods, which depend on hand-crafted features, lack
the capacity of fully exploiting heterogeneous relations across modalities. Recently, models based
on deep neural networks can easily access more discriminative features, leading to a boost in the
development of deep cross-modal hashing techniques [
6
,
15
,
23
,
43
,
26
,
11
]. Deep cross-modal
hashing (DCMH) [
23
] utilizes a deep neural network to perform feature extraction and hash code
learning from scratch. Pairwise relationship guided deep hashing (PRDH) [
48
] integrates different
pairwise constraints to purify the similarities of hash codes from inter-modality and intra-modality.
Self-supervised adversarial hashing (SSAH) [
25
] introduces a label network into the hash learning
process, which facilitates hash code generation.
Despite the outstanding performance and remarkable achievements, the DNN systems have recently
been shown to be vulnerable to adversarial attacks. Szegedy et al. [
42
] first showed that a well-
designed small perturbation on images can fool state-of-the-art deep neural networks with high
probability. In the following, a series of research efforts on attack [
41
,
46
,
34
] and defense [
7
,
35
,
38
]
have been presented. Fast gradient sign method (FGSM) [
17
], which uses a sign function on gradients
for the inputs to learn adversarial examples, is one of the representative efficient attack algorithms.
For the defense methods, adversarial training [
42
,
17
] is adopted to augment their training data of the
classifier by learning adversarial samples. Distillation technique [
38
] was presented to reduce the
magnitude of the gradients used for adversarial sample creation and also to increase the minimum
feature numbers to be modified. However, concerning large-scale cross-modal retrieval, although
plenty of deep cross-modal hashing networks have been constructed, there is no attempt to focus on
the security of DNNs in deep cross-modal hashing models.
3 Cross-Modal Learning with Adversarial Samples
3.1 Problem Definition
Given a cross-modal benchmark
O = {o
i
}
N
i=1
with
N
data points,
o
i
= (o
v
i
, o
t
i
, o
l
i
)
, where
o
v
i
and
o
t
i
are collected in pair, respectively, denoting image and textual description for the
i
th data
point,
o
l
i
is a multi-label annotation assigned to
o
i
. Let
S
denote a pairwise similarity matrix which
describes semantic similarity between each pair of data points, where
S
ij
= 1
means that
o
i
and
o
j
are semantically similar, otherwise
S
ij
= 0
. In a multi-label setting, when
o
i
and
o
j
share at
3