最新的对比自监督学习（ContrastiveSelf-supervisedLearning）综述论文_对比学习综述

对比学习

需积分: 0 1.1k 浏览量更新于2023-05-27 评论 1 收藏 5.72MB PDF 举报

身份认证购VIP最低享 7 折!

领优惠券(最高得80元）

资源详情

资源评论

资源推荐

A SURVEY ON CONTRASTIVE SELF-SUPERVISED LEARNING

Ashish Jaiswal

The University of Texas at Arlington

Arlington, TX 76019

ashish.jaiswal@mavs.uta.edu

Ashwin Ramesh Babu

The University of Texas at Arlington

Arlington, TX 76019

ashwin.rameshbabu@mavs.uta.edu

Mohammad Zaki Zadeh

The University of Texas at Arlington

Arlington, TX 76019

mohammad.zakizadehgharie@mavs.uta.edu

Debapriya Banerjee

The University of Texas at Arlington

Arlington, TX 76019

debapriya.banerjee2@mavs.uta.edu

Fillia Makedon

The University of Texas at Arlington

Arlington, TX 76019

makedon@uta.edu

ABSTRACT

Self-supervised learning has gained popularity because of its ability to avoid the cost of annotating

large-scale datasets. It is capable of adopting self-deﬁned pseudo labels as supervision and use the

learned representations for several downstream tasks. Speciﬁcally, contrastive learning has recently

become a dominant component in self-supervised learning methods for computer vision, natural

language processing (NLP), and other domains. It aims at embedding augmented versions of the

same sample close to each other while trying to push away embeddings from different samples. This

paper provides an extensive review of self-supervised methods that follow the contrastive approach.

The work explains commonly used pretext tasks in a contrastive learning setup, followed by different

architectures that have been proposed so far. Next, we have a performance comparison of different

methods for multiple downstream tasks such as image classiﬁcation, object detection, and action

recognition. Finally, we conclude with the limitations of the current methods and the need for further

techniques and future directions to make substantial progress.

Keywords

contrastive learning

self-supervised learning

discriminative learning

image/video classiﬁcation

object

detection · unsupervised learning · transfer learning

1 Introduction

The advancements in deep learning have elevated it to become one of the core components in most intelligent systems in

existence. The ability to learn rich patterns from the abundance of data available today has made deep neural networks

(DNNs) a compelling approach in the majority of computer vision (CV) tasks such as image classiﬁcation, object

detection, image segmentation, activity recognition as well as natural language processing (NLP) tasks such as sentence

classiﬁcation, language models, machine translation, etc. However, the supervised approach to learning features from

labeled data has almost reached its saturation due to intense labor required in manually annotating millions of data

samples. This is because most of the modern computer vision systems (that are supervised) try to learn some form of

image representations by ﬁnding a pattern between the data points and their respective annotations in large datasets.

Works such as GRAD-CAM [

] have proposed techniques that provide visual explanations for decisions made by a

model to make them more transparent and explainable.

Traditional supervised learning approaches heavily rely on the amount of annotated training data available. Even

though there’s a plethora of data available out there, the lack of annotations has pushed researchers to ﬁnd alternative

arXiv:2011.00362v3 [cs.CV] 7 Feb 2021

A PREPRINT - FEBRUARY 9, 2021

PixelRNN [

], Text2Image [

], DiscoGAN [

], etc. These methods inspired more researchers to switch to training deep

learning models with unlabeled data in an self-supervised setup. Despite their success, researchers started realizing

some of the complications in GAN-based approaches. They are harder to train because of two main reasons: (a)

non-convergence–the model parameters oscillate a lot and rarely converge, and (b) the discriminator gets too successful

that the generator network fails to create real-like fakes due to which the learning cannot be continued. Also, proper

synchronization is required between the generator and the discriminator that prevents the discriminator to converge and

the generator to diverge.

Figure 3: Top-1 classiﬁcation accuracy of different contrastive learning methods against baseline supervised method on

ImageNet

Unlike generative models, contrastive learning (CL) is a discriminative approach that aims at grouping similar samples

closer and diverse samples far from each other as shown in ﬁgure 1. To achieve this, a similarity metric is used to

measure how close two embeddings are. Especially, for computer vision tasks, a contrastive loss is evaluated based

on the feature representations of the images extracted from an encoder network. For instance, one sample from the

training dataset is taken and a transformed version of the sample is retrieved by applying appropriate data augmentation

techniques. During training referring to ﬁgure 2, the augmented version of the original sample is considered as a

positive sample, and the rest of the samples in the batch/dataset (depends on the method being used) are considered

negative samples. Next, the model is trained in a way that it learns to differentiate positive samples from the negative

ones. The differentiation is achieved with the help of some pretext task (explained in section 2). In doing so, the model

learns quality representations of the samples and is used later for transferring knowledge to downstream tasks. This

idea is advocated by an interesting experiment conducted by Epstein [

] in 2016, where he asked his students to draw a

dollar bill with and without looking at the bill. The results from the experiment show that the brain does not require

complete information of a visual piece to differentiate one object from the other. Instead, only a rough representation of

an image is enough to do so.

Most of the earlier works in this area combined some form of instance-level classiﬁcation approach[

][

] with

contrastive learning and were successful to some extent. However, recent methods such as SwAV [

], MoCo [

], and

SimCLR [

] with modiﬁed approaches have produced results comparable to the state-of-the-art supervised method on

A PREPRINT - FEBRUARY 9, 2021

ImageNet [

] dataset as shown in ﬁgure 3. Similarly, PIRL [

], Selﬁe [

], and [

] are some papers that reﬂect the

effectiveness of the pretext tasks being used and how they boost the performance of their models.

2 Pretext Tasks

Pretext tasks are self-supervised tasks that act as an important strategy to learn representations of the data using pseudo

labels. These pseudo labels are generated automatically based on the attributes found in the data. The learned model

from the pretext task can be used for any downstream tasks such as classiﬁcation, segmentation, detection, etc. in

computer vision. Furthermore, these tasks can be applied to any kind of data such as image, video, speech, signals,

and so on. For a pretext task in contrastive learning, the original image acts as an anchor, its augmented(transformed)

version acts as a positive sample, and the rest of the images in the batch or in the training data act as negative samples.

Most of the commonly used pretext tasks are divided into four main categories: color transformation, geometric

transformation, context-based tasks, and cross-modal based tasks. These pretext tasks have been used in various

scenarios based on the problem intended to be solved.

2.1 Color Transformation

Figure 4: Color Transformation as pretext task [

]. (a) Original (b) Gaussian noise (c) Gaussian blur (d) Color

distortion (Jitter)

Color transformation involves basic adjustments of color levels in an image such as blurring, color distortions, converting

to grayscale, etc. Figure 4 represents an example of color transformation applied on a sample image from the ImageNet

dataset [15]. During this pretext task, the network learns to recognize similar images invariant to their colors.

2.2 Geometric Transformation

A geometric transformation is a spatial transformation where the geometry of the image is modiﬁed without altering

its actual pixel information. The transformations include scaling, random cropping, ﬂipping (horizontally, vertically),

etc. as represented in ﬁgure 5 through which global-to-local view prediction is achieved. Here the original image is

considered as the global view and the transformed version is considered as the local view. Chen et. al. [

] performed

such transformations to learn features during pretext task.

2.3 Context-Based

2.3.1 Jigsaw puzzle

Traditionally, solving jigsaw puzzles has been a prominent task in learning features from an image in an unsupervised

way. It involves identifying the correct position of the scrambled patches in an image by training an encoder (ﬁgure 6).

A PREPRINT - FEBRUARY 9, 2021

Figure 5: Geometric Transformation as pretext task [

]. (a) Original (b) Crop and Resize (c) Rotate(90

◦

, 180

◦

, 270

◦

)

(d) crop, resize, ﬂip

In terms of contrastive learning, the original image is the anchor, and an augmented image formed by scrambling the

patches in the original image acts as a positive sample. The rest of the images in the dataset/batch are considered to be

negative samples [17].

Figure 6: Solving jigsaw puzzle being used as a pretext task to learn representation. (a) Original Image (b) reshufﬂed

image. The original image is the anchor and the reshufﬂed image is the positive sample.

2.3.2 Frame order based

This approach applies to data that extends through time. An ideal application would be in the case of sensor data or

a sequence of image frames (video). A video contains a sequence of semantically related frames. This implies that

frames that are nearby with respect to time are closely related and the ones that are far away are less likely to be related.

Intuitively, the motive for using such an approach is, solving a pretext task that allows the model to learn useful visual

representations while trying to recover the temporal coherence of a video. Here, a video with shufﬂed order in the

sequence of its image frames acts as a positive sample while all other videos in the batch/dataset would be negative

samples.

Similarly, other possible approaches include randomly sampling two clips of the same length from a longer video or

applying spatial augmentation for each video clip. The goal is to use a contrastive loss to train the model such that clips

taken from the same video are arranged closer whereas clips from different videos are pushed away in the embedding

space. In the work proposed by Qian et. al. [

], the framework contrasts the similarity between two positive samples

to those of negative samples. The positive pairs are two augmented clips from the same video. As a result, it separates

all encoded videos into non-overlapping regions such that an augmentation used in the training perturbs an encoded

video only within a small region in the representation space.

剩余20页未读，继续阅读

syp_net

粉丝: 147
资源: 1197

会员权益专享

最新的对比自监督学习（Contrastive Self-supervised Learning）综述论文

评论0

会员权益专享

最新资源

最新的对比自监督学习（Contrastive Self-supervised Learning）综述论文

评论0

dense contrastive learning for self-supervised visual pre-training

Contrastive Trajectory Similarity Learning with Dual-Feature Attention

用R语言 写一个基于自监督学习的对比学习网络

deep graph contrastive representation learning

graph contrastive learning with augmentations

contrastive predictive coding

contrastive multiview coding

Contrastive Representation Distillation该文章的创新点

contrastive clustering

contrastive representation distillation

CLIP可以详细说说嘛

simcse transform

contrastive loss代码实现

https://arxiv.org/pdf/2106.06561.pdf

contrastive loss pytorch

试比较 CE，CV，QCV 理论的优点及其局限性

蓝色简洁风工程师成套Word简历模板.docx

基于ssm+mysql的编程类在线答题系统源码数据库.zip

【老生谈算法】matlab实现基于动态粒子群算法的动态环境寻优算法.doc

会员权益专享

最新资源

用R语言写一个基于自监督学习的对比学习网络