最新《小样本学习(Few-shotlearning)》2020综述论文【香港科技大学】.pdf_适合小样本的机器学习-机器学习文档类

Few-shot

learnin

4星 · 超过85%的资源需积分: 0 4.7k 浏览量更新于2023-05-28 评论 11 收藏 6.57MB PDF 举报

身份认证购VIP最低享 7 折!

领优惠券(最高得80元）

资源详情

资源评论

资源推荐

Generalizing from a Few Examples: A Survey on Few-Shot

Learning

YAQING WANG, Hong Kong University of Science and Technology and Baidu Research

QUANMING YAO

∗

, 4Paradigm Inc.

JAMES T. KWOK, Hong Kong University of Science and Technology

LIONEL M. NI, Hong Kong University of Science and Technology

Machine learning has been highly successful in data-intensive applications, but is often hampered when the

data set is small. Recently, Few-Shot Learning (FSL) is proposed to tackle this problem. Using prior knowledge,

FSL can rapidly generalize to new tasks containing only a few samples with supervised information. In this

paper, we conduct a thorough survey to fully understand FSL. Starting from a formal denition of FSL, we

distinguish FSL from several relevant machine learning problems. We then point out that the core issue in

FSL is that the empirical risk minimizer is unreliable. Based on how prior knowledge can be used to handle

this core issue, we categorize FSL methods from three perspectives: (i) data, which uses prior knowledge to

augment the supervised experience; (ii) model, which uses prior knowledge to reduce the size of the hypothesis

space; and (iii) algorithm, which uses prior knowledge to alter the search for the best hypothesis in the given

hypothesis space. With this taxonomy, we review and discuss the pros and cons of each category. Promising

directions, in the aspects of the FSL problem setups, techniques, applications and theories, are also proposed

to provide insights for future research.

CCS Concepts:

• Computing methodologies → Articial intelligence

;

Machine learning

;

Learning

paradigms.

Additional Key Words and Phrases: Few-Shot Learning, One-Shot Learning, Low-Shot Learning, Small Sample

Learning, Meta-Learning, Prior Knowledge

ACM Reference Format:

Yaqing Wang, Quanming Yao, James T. Kwok, and Lionel M. Ni. 2020. Generalizing from a Few Examples: A

Survey on Few-Shot Learning. ACM Comput. Surv. 1, 1, Article 1 (March 2020), 34 pages. https://doi.org/10.

1145/3386252

1 INTRODUCTION

“Can machines think?” This is the question raised in Alan Turing’s seminal paper entitled “Comput-

ing Machinery and Intelligence” [

134

] in 1950. He made the statement that “The idea behind digital

∗

Corresponding Author

A list of references, which will be updated periodically, can be found at https:// github.com/tata1661/FewShotPapers.git.

Authors’ addresses: Yaqing Wang, ywangcy@connect.ust.hk, Department of Computer Science and Engineering, Hong

Kong University of Science and Technology, Business Intelligence Lab and National Engineering Laboratory of Deep

Learning Technology and Application, Baidu Research; Quanming Yao, yaoquanming@4paradigm.com, 4Paradigm Inc.;

James T. Kwok, jamesk@cse.ust.hk, Department of Computer Science and Engineering, Hong Kong University of Science

and Technology; Lionel M. Ni, ni@ust.hk, Department of Computer Science and Engineering, Hong Kong University of

Science and Technology.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee

provided that copies are not made or distributed for prot or commercial advantage and that copies bear this notice and

the full citation on the rst page. Copyrights for components of this work owned by others than ACM must be honored.

Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires

prior specic permission and/or a fee. Request permissions from permissions@acm.org.

0360-0300/2020/3-ART1 $15.00

https://doi.org/10.1145/3386252

ACM Comput. Surv., Vol. 1, No. 1, Article 1. Publication date: March 2020.

arXiv:1904.05046v3 [cs.LG] 29 Mar 2020

1:2 Yaqing Wang, anming Yao, James T. Kwok, and Lionel M. Ni

computers may be explained by saying that these machines are intended to carry out any opera-

tions which could be done by a human computer”. In other words, the ultimate goal of machines

is to be as intelligent as humans. In recent years, due to the emergence of powerful computing

devices (e.g., GPU and distributed platforms), large data sets (e.g., ImageNet data with 1000 classes

[

]), advanced models and algorithms (e.g., convolutional neural networks (CNN) [

] and long

short-term memory (LSTM) [

]), AI speeds up its pace to be like humans and defeats humans

in many elds. To name a few, AlphaGo [

120

] defeats human champions in the ancient game of

Go; and residual network (ResNet) [

] obtains better classication performance than humans on

ImageNet. AI also supports the development of intelligent tools in many aspects of daily life, such

as voice assistants, search engines, autonomous driving cars, and industrial robots.

Albeit its prosperity, current AI techniques cannot rapidly generalize from a few examples.

The aforementioned successful AI applications rely on learning from large-scale data. In contrast,

humans are capable of learning new tasks rapidly by utilizing what they learned in the past. For

example, a child who learned how to add can rapidly transfer his knowledge to learn multiplication

given a few examples (e.g., 2

2 and 1

1). Another example is that given a

few photos of a stranger, a child can easily identify the same person from a large number of photos.

Bridging this gap between AI and humans is an important direction. It can be tackled by machine

learning, which is concerned with the question of how to construct computer programs that

automatically improve with experience [

]. In order to learn from a limited number of examples

with supervised information, a new machine learning paradigm called Few-Shot Learning (FSL)

[

] is proposed. A typical example is character generation [

], in which computer programs

are asked to parse and generate new handwritten characters given a few examples. To handle this

task, one can decompose the characters into smaller parts transferable across characters, and then

aggregate these smaller components into new characters. This is a way of learning like human [

Naturally, FSL can also advance robotics [

], which develops machines that can replicate human

actions. Examples include one-shot imitation [

147

], multi-armed bandits [

], visual navigation

[37], and continuous control [156].

Another classic FSL scenario is where examples with supervised information are hard or im-

possible to acquire due to privacy, safety or ethic issues. A typical example is drug discovery,

which tries to discover properties of new molecules so as to identify useful ones as new drugs [

Due to possible toxicity, low activity, and low solubility, new molecules do not have many real

biological records on clinical candidates. Hence, it is important to learn eectively from a small

number of samples. Similar examples where the target tasks do not have many examples include

FSL translation [

], and cold-start item recommendation [

137

]. Through FSL, learning suitable

models for these rare cases can become possible.

FSL can also help relieve the burden of collecting large-scale supervised data. For example,

although ResNet [

] outperforms humans on ImageNet, each class needs to have sucient labeled

images which can be laborious to collect. FSL can reduce the data gathering eort for data-intensive

applications. Examples include image classication [

138

], image retrieval [

130

], object tracking [

gesture recognition [

102

], image captioning, visual question answering [

], video event detection

[151], language modeling [138], and neural architecture search [19].

Driven by the academic goal for AI to approach humans and the industrial demand for inexpensive

learning, FSL has drawn much recent attention and is now a hot topic. Many related machine

learning approaches have been proposed, such as meta-learning [

106

114

], embedding learning

[

126

138

] and generative modeling [

113

]. However, currently, there is no work that

provides an organized taxonomy to connect these FSL methods, explains why some methods work

while others fail, nor discusses the pros and cons of dierent approaches. Therefore, in this paper,

ACM Comput. Surv., Vol. 1, No. 1, Article 1. Publication date: March 2020.

Generalizing from a Few Examples: A Survey on Few-Shot Learning 1:3

we conduct a survey on the FSL problem. In contrast, the survey in [

118

] only focuses on concept

learning and experience learning for small samples.

Contributions of this survey can be summarized as follows:

•

We give a formal denition on FSL, which naturally connects to the classic machine learning

denition in [

]. The denition is not only general enough to include existing FSL works,

but also specic enough to clarify what the goal of FSL is and how we can solve it. This

denition is helpful for setting future research targets in the FSL area.

•

We list the relevant learning problems for FSL with concrete examples, clarifying their

relatedness and dierences with respect to FSL. These discussions can help better discriminate

and position FSL among various learning problems.

•

We point out that the core issue of FSL supervised learning problem is the unreliable empirical

risk minimizer, which is analyzed based on error decomposition [

] in machine learning.

This provides insights to improve FSL methods in a more organized and systematic way.

•

We perform an extensive literature review, and organize them in an unied taxonomy

from the perspectives of data, model and algorithm. We also present a summary of insights

and a discussion on the pros and cons of each category. These can help establish a better

understanding of FSL methods.

•

We propose promising future directions for FSL in the aspects of problem setup, techniques,

applications and theories. These insights are based on the weaknesses of the current develop-

ment of FSL, with possible improvements to make in the future.

1.1 Organization of the Survey

The remainder of this survey is organized as follows. Section 2 provides an overview for FSL,

including its formal denition, relevant learning problems, core issue, and a taxonomy of existing

works in terms of data, model and algorithm. Section 3 is for methods that augment data to solve

FSL problem. Section 4 is for methods that reduce the size of hypothesis space so as to make FSL

feasible. Section 5 is for methods that alter the search strategy of algorithm to deal with the FSL

problem. In Section 6, we propose future directions for FSL in terms of problem setup, techniques,

applications and theories. Finally, the survey closes with conclusion in Section 7.

1.2 Notation and Terminology

Consider a learning task

, FSL deals with a data set

D = {D

train

, D

test

}

consisting of a training set

train

= {(x

, y

)}

i=1

where

is small, and a testing set

test

= {x

test

}

. Let

p(x, y)

be the ground-truth

joint probability distribution of input

and output

, and

be the optimal hypothesis from

. FSL learns to discover

by tting

train

and testing on

test

. To approximate

, the FSL model

determines a hypothesis space

of hypotheses

h(·

;

θ)

’s, where

denotes all the parameters used

. Here, a parametric

is used, as a nonparametric model often requires large data sets, and

thus not suitable for FSL. A FSL algorithm is an optimization strategy that searches

in order to

nd the

that parameterizes the best

∗

∈ H

. The FSL performance is measured by a loss function

ℓ(

y, y) dened over the prediction

y = h(x; θ ) and the observed output y.

2 OVERVIEW

In this section, we rst provide a formal denition of the FSL problem in Section 2.1 with concrete

examples. To dierentiate the FSL problem from relevant machine learning problems, we discuss

their relatedness and dierences in Section 2.2. In Section 2.3, we discuss the core issue that makes

FSL dicult. Section 2.4 then presents a unied taxonomy according to how existing works handle

the core issue.

ACM Comput. Surv., Vol. 1, No. 1, Article 1. Publication date: March 2020.

1:4 Yaqing Wang, anming Yao, James T. Kwok, and Lionel M. Ni

2.1 Problem Definition

As FSL is a sub-area in machine learning, before giving the denition of FSL, let us recall how

machine learning is dened in the literature.

Denition 2.1 (

Machine Learning

[

]). A computer program is said to learn from experience

with respect to some classes of task

and performance measure

if its performance can improve

with E on T measured by P.

For example, consider an image classication task (

), a machine learning program can improve

its classication accuracy (

) through

obtained by training on a large number of labeled images

(e.g., the ImageNet data set [

]). Another example is the recent computer program AlphaGo [

120

which has defeated the human champion in playing the ancient game of Go (

). It improves its

winning rate (

) against opponents by training on a database (

) of around 30 million recorded

moves of human experts as well as playing against itself repeatedly. These are summarized in

Table 1.

Table 1. Examples of machine learning problems based on Definition 2.1.

task T

experience E

performance P

image classication [73] large-scale labeled images for each class

classication

accuracy

the ancient game of Go [120]

a database containing around 30 million recorded moves

of human experts and self-play records

winning rate

Typical machine learning applications, as in the examples mentioned above, require a lot of

examples with supervised information. However, as mentioned in the introduction, this may be

dicult or even not possible. FSL is a special case of machine learning, which targets at obtaining

good learning performance given limited supervised information provided in the training set

train

which consists of examples of inputs

’s along with their corresponding output

’s [

]. Formally,

we dene FSL in Denition 2.2.

Denition 2.2.

Few-Shot Learning

(FSL) is a type of machine learning problems (specied by

and

), where

contains only a limited number of examples with supervised information for the

target T .

Existing FSL problems are mainly supervised learning problems. Concretely, few-shot classication

learns classiers given only a few labeled examples of each class. Example applications include

image classication [

138

], sentiment classication from short text [

157

] and object recognition

[

]. Formally, using notations from Section 1.2, few-shot classication learns a classier

which

predicts label

for each input

. Usually, one considers the

-way-

-shot classication [

138

in which

train

contains

I = KN

examples from

classes each with

examples. Few-shot regression

[

156

] estimates a regression function

given only a few input-output example pairs sampled

from that function, where output

is the observed value of the dependent variable

, and

the input which records the observed value of the independent variable

. Apart from few-shot

supervised learning, another instantiation of FSL is few-shot reinforcement learning [3, 33], which

targets at nding a policy given only a few trajectories consisting of state-action pairs.

We now show three typical scenarios of FSL (Table 2):

•

Acting as a test bed for learning like human. To move towards human intelligence, it is vital

that computer programs can solve the FSL problem. A popular task (

) is to generate samples

of a new character given only a few examples [

]. Inspired by how humans learn, the

ACM Comput. Surv., Vol. 1, No. 1, Article 1. Publication date: March 2020.

Generalizing from a Few Examples: A Survey on Few-Shot Learning 1:5

computer programs learn with the

consisting of both the given examples with supervised

information and pre-trained concepts such as parts and relations as prior knowledge. The

generated characters are evaluated through the pass rate of visual Turing test (

), which

discriminates whether the images are generated by humans or machines. With this prior

knowledge, computer programs can also learn to classify, parse and generate new handwritten

characters with a few examples like humans.

•

Learning for rare cases. When obtaining sucient examples with supervised information

is hard or impossible, FSL can learn models for the rare cases. For example, consider a

drug discovery task (

) which tries to predict whether a new molecule has toxic eects [

The percentage of molecules correctly assigned as toxic or non-toxic (

) improves with

obtained by both the new molecule’s limited assay, and many similar molecules’ assays as

prior knowledge.

•

Reducing data gathering eort and computational cost . FSL can help relieve the burden of

collecting large number of examples with supervised information. Consider few-shot image

classication task (

) [

]. The image classication accuracy (

) improves with the

obtained

by a few labeled images for each class of the target

, and prior knowledge extracted from

the other classes (such as raw images to co-training). Methods succeed in this task usually

have higher generality. Therefore, they can be easily applied for tasks of many samples.

Table 2. Three FSL examples based on Definition 2.2.

task T

experience E

performance P

supervised information prior knowledge

character generation [76]

a few examples of new

character

pre-learned knowledge of

parts and relations

pass rate of visual

Turing test

drug toxicity discovery [4]

new molecule’s limited

assay

similar molecules’ assays

classication

accuracy

image classication [70]

a few labeled images for

each class of the target T

raw images of other classes,

or pre-trained models

classication

accuracy

In comparison to Table 1, Table 2 has one extra column under “experience

" which is marked

as “prior knowledge". As

only contains a few examples with supervised information directly

related to

, it is natural that common supervised learning approaches often fail on FSL problems.

Therefore, FSL methods make the learning of target

feasible by combining the available supervised

information in

with some prior knowledge, which is “any information the learner has about the

unknown function before seeing the examples" [

]. One typical type of FSL methods is Bayesian

learning [

]. It combines the provided training set

train

with some prior probability distribution

which is available before D

train

is given [15].

Remark 1. When there is only one example with supervised information in

, FSL is called

one-shot

learning

[

138

]. When

does not contain any example with super vised information for the

target

, FSL becomes a

zero-shot learning

problem (ZSL) [

]. As the target class does not contain

examples with supervised information, ZSL requires

to contain information from other modalities

(such as attributes, WordNet, and word embeddings used in rare object recognition tasks), so as to

transfer some supervised information and make learning possible.

2.2 Relevant Learning Problems

In this section, we discuss some relevant machine learning problems. The relatedness and dierence

with respect to FSL are claried.

ACM Comput. Surv., Vol. 1, No. 1, Article 1. Publication date: March 2020.

剩余33页未读，继续阅读

alto1394

2021-02-19

Generalizing from a Few Examples: A Survey on Few-Shot Learning

syp_net

粉丝: 147
资源: 1197

会员权益专享

最新《小样本学习(Few-shot learning)》2020综述论文【香港科技大学】.pdf

评论1

会员权益专享

最新资源

最新《小样本学习(Few-shot learning)》2020综述论文【香港科技大学】.pdf

评论1

few-shot learning

A New Meta-Baseline for Few-Shot Learning.docx

few-shot learning代码

learning to compare: relation network for few-shot learning

meta-baseline: exploring simple meta-learning for few-shot learning

few-shot learning代码详解

edge-labeling graph neural network for few-shot learning

few-shot learning with graph neural networks

few-shot network anomaly detection via cross-network meta-learning

prototypical networks for few-shot learning

现在有小样本模型中用到mask吗

prototypical networks代码

商业银行桌面虚拟化解决方案.pdf

人工智能-智慧城市.ppt

数据库实验报告CR0106.pdf

《计算机应用基础》习题及答案(1).doc

人工智能制造-产业发展研究报告.pptx

AntDesign3.9Axure组件

SAP Integrated Business Planning.zip

centos7-mysql-5.7.40.tar

会员权益专享

最新资源