深度元学习：2021全面综述

深度学习

需积分: 9 158 浏览量更新于2024-07-13 2 收藏 3.1MB PDF 举报

身份认证购VIP最低享 7 折!

领优惠券(最高得80元）

"这篇资源是一篇2021年的深度元学习综述论文，由Mike Huisman、Jan N. van Rijn和Aske Plaat撰写。文章旨在提供深度元学习领域的全面概述，填补该领域缺乏统一深入研究的空白。作者们首先为读者构建理论基础，然后探讨并总结了元学习的主要方法，将其分为三类：i) 基于度量的，ii) 基于模型的，和iii) 基于优化的技术。此外，他们还指出了当前元学习面临的主要挑战，如异构基准上的性能评估和降低计算成本。关键词包括元学习、学习如何学习、少样本学习、迁移学习和深度学习。" 深度元学习是机器学习的一个分支，它借鉴了人类的学习能力，即从有限的经验中快速学习新任务的能力。在传统的深度学习中，模型需要大量的标注数据来训练，而元学习则尝试让模型具备从少量样本中学习新任务的能力，这被称为“少样本学习”。基于度量的元学习方法主要关注如何通过学习相似性度量来快速适应新任务。这类方法通常涉及到学习一个距离函数，使得相似的任务样本被拉近，不相似的任务样本被推远，从而在新任务上进行快速预测。基于模型的元学习则更侧重于构建一个可以泛化到新任务的模型结构。这个模型能够捕获不同任务之间的共性，并利用这些共性来适应新任务。例如，通过学习一个能够适应不同任务的参数初始化或架构，模型可以在遇到新任务时更快地收敛。基于优化的元学习方法关注于学习一个优化过程，使模型在新任务上能更快达到良好的性能。这可能涉及到学习适应性的学习率调整策略或者学习如何更新权重，以加速训练过程。尽管深度元学习在提高模型的泛化能力和适应性方面取得了进展，但仍然存在一些挑战。其中，如何在不同的任务分布（异构基准）上公正地评估元学习算法的性能是一个关键问题，因为不同的任务可能需要不同的学习策略。此外，元学习往往需要大量的计算资源，尤其是在训练过程中需要多次迭代，因此如何减少计算成本是另一个需要解决的难题。这篇综述论文对深度元学习领域的研究者和实践者来说具有很高的价值，因为它提供了对该领域的深入理解，同时指明了未来的研究方向。通过了解和应用这些元学习技术，我们可以期待在诸如机器人控制、自然语言处理和计算机视觉等领域的更快、更高效的学习算法。

资源详情

资源推荐

12 Mike Huisman et al.

Metric Model Optimization

Key idea Input similarity

Internal task

representation

Optimize for fast

adaptation

Strength Simple and eﬀective Flexible

More robust general-

izability

(Y |x, D

)

)∈D

(x, x

(x, D

) f

ϕ(θ,D

)

(x)

Table 2 High-level overview of the three Deep Meta-Learning categories, i.e., i) metric-,

ii) model-, and iii) optimization-based techniques, and their main strengths and weaknesses.

Recall that T

is a task, D

the corresponding support set, k

(x, x

) a kernel function

returning the similarity between the two inputs x and x

, y

are true labels for known

inputs x

, θ are base-learner parameters, and g

is a (learned) optimizer with parameters

ϕ.

(a set of assumptions that guide predictions). We now present a brief histor-

ical overview of developments in this ﬁeld of Deep Meta-Learning, based on

Hospedales et al. (2020).

Pioneering work was done by Schmidhuber (1987) and Hinton and Plaut

(1987). Schmidhuber developed a theory of self-referential learning, where the

weights of a neural network can serve as input to the model itself, which then

predicts updates (Schmidhuber, 1987, 1993). In that same year, Hinton and

Plaut (1987) proposed to use two weights per neural network connection, i.e.,

slow and fast weights, which serve as long- and short-term memory respec-

tively. Later came the idea of meta-learning learning rules (Bengio et al., 1991,

1997). Meta-learning techniques that use gradient-descent and backpropaga-

tion were proposed by Hochreiter et al. (2001) and Younger et al. (2001). These

two works have been pivotal to the current ﬁeld of Deep Meta-Learning, as the

majority of techniques rely on backpropagation, as we will see on our journey

of contemporary Deep Meta-Learning techniques.

2.4 Overview of the rest of this Work

In the remainder of this work, we will look in more detail at individual meta-

learning methods. As indicated before, the techniques can be grouped into

three main categories (Vinyals, 2017), namely i) metric-, ii) model-, and iii) opti-

mization-based methods. We will discuss them in that order.

To help give an overview of the methods, we draw your attention to the fol-

lowing tables. Table 2 summarizes the three categories and provides key ideas,

and strengths of the approaches. The terms and technical details are explained

more fully in the remainder of this paper. Table 3 contains an overview of all

techniques that are discussed further on.

A Survey of Deep Meta-Learning 13

Name RL Key idea Bench.

Metric-based Input similarity -

Siamese networks 7

Two-input, shared-weight, class identity

network

1, 8

Matching networks 7

Learn input embeddings for cosine-

similarity weighted predictions

1, 2

Prototypical networks 7

Input embeddings for class prototype

clustering

1, 2, 7

Relation networks 7

Learn input embeddings and similarity

metric

1, 2, 7

ARC 7

LSTM-based input fusion through inter-

leaved glimpses

1, 2

GNN 7

Propagate label information to unlabeled

inputs in a graph

1, 2

Model-based

Internal and stateful latent task

representations

Reccurrent ml. X

Deploy Recurrent networks on RL prob-

lems

MANNs 7

External short-term memory module for

fast learning

Meta networks X

Fast reparameterization of base-learner

by distinct meta-learner

1, 2

SNAIL X

Attention mechanism coupled with tem-

poral convolutions

1, 2

CNP 7

Condition predictive model on embedded

contextual task data

1, 8

Neural stat. 7

Similarity between latent task embed-

dings

1, 8

Opt.-based

Optimize for fast task-speciﬁc

adaptation

LSTM optimizer 7

RNN proposing weight updates for base-

leaner

6, 8

LSTM ml. X

Embed base-learner parameters in cell

state of LSTM

RL optimizer 7 View optimization as RL problem 4, 6

MAML X

Learn initialization weights θ for fast

adaptation

1, 2

iMAML X

Approx. higher-order gradients, indepen-

dent of optimization path

1, 2

Meta-SGD X Learn both the initialization and updates 1, 2

Reptile X

Move initialization towards task-speciﬁc

updated weights

1, 2

LEO 7

Optimize in lower-dimensional latent pa-

rameter space

2, 3

Online MAML 7

Accumulate task data for MAML-like

training

4, 8

LLAMA 7

Maintain probability distribution over

post-update parameters θ

PLATIPUS 7

Learn a probability distribution over

weight initializations θ

BMAML X

Learn multiple initializations Θ, jointly

optimized by SVGD

Diﬀ. solvers 7

Learn input embeddings for simple base-

learners

1, 2, 3, 4, 5

Table 3 Overview of the discussed Deep Meta-Learning techniques. The table is partitioned

into three sections, i.e., metric-, model-, and optimization-based techniques. All methods

in one section adhere to the key idea of its corresponding category, which is mentioned

in bold font. The columns RL and Bench show whether the techniques are applicable to

reinforcement learning settings and the used benchmarks for testing the performance of

the techniques. Note that all techniques are applicable to supervised learning, with the

exception of RMLs. The benchmark column displays which benchmarks from Section 2.2.2

were used in the paper proposing the technique. The used coding scheme for this column is

the following. 1: Omniglot, 2: miniImageNet, 3: tieredImageNet, 4: CIFAR-100, 5: CIFAR-

FS, 6: CIFAR-10, 7: CUB, 8: MNIST, “-": used other evaluation method that are non-

standard in Deep Meta-Learning and thus not covered in Section 2.2.2. Used abbreviations:

“opt.": optimization, “diﬀ.": diﬀerentiable, “bench.": benchmarks.

14 Mike Huisman et al.

3 Metric-based Meta-Learning

At a high level, the goal of metric-based techniques is to acquire—among

others—meta-knowledge ω in the form of a good feature space that can be

used for various new tasks. In the context of neural networks, this feature

space coincides with the weights θ of the networks. Then, new tasks can be

learned by comparing new inputs to example inputs (of which we know the

labels) in the meta-learned feature space. The higher the similarity between a

new input and an example, the more likely it is that the new input will have

the same label as the example input.

Metric-based techniques are a form of meta-learning as they leverage their

prior learning experience (meta-learned feature space) to ‘learn’ new tasks

more quickly. Here, ‘learn’ is used in a non-standard way since metric-based

techniques do not make any network changes when presented with new tasks,

as they rely solely on input comparisons in the already meta-learned feature

space. These input comparisons are a form of non-parametric learning, i.e.,

new task information is not absorbed into the network parameters.

More formally, metric-based learning techniques aim to learn a similarity

kernel, or equivalently, attention mechanism k

(parameterized by θ), that

takes two inputs x

and x

, and outputs their similarity score. Larger scores

indicate larger similarity. Class predictions for new inputs x can then be made

by comparing x to example inputs x

, of which we know the true labels y

The underlying idea being that the larger the similarity between x and x

, the

more likely it becomes that x also has label y

Given a task T

= (D

, D

test

) and an unseen input vector x ∈ D

test

, a

probability distribution over classes Y is computed/predicted as a weighted

combination of labels from the support set D

, using similarity kernel k

, i.e.,

(Y |x, D

) =

)∈D

(x, x

. (7)

Importantly, the labels y

are assumed to be one-hot encoded, meaning that

they are represented by zero vectors with a ‘1’ on the position of the true

class. For example, suppose there are ﬁve classes in total, and our example

has true class 4. Then, the one-hot encoded label is y

= [0, 0, 0, 1, 0].

Note that the probability distribution p

(Y |x, D

) over classes is a vec-

tor of size |Y |, in which the i-th entry corresponds to the probability that

input x has class Y

(given the support set). The predicted class is thus

ˆy = arg max

i=1,2,...,|Y |

(Y |x, S)

, where p

(Y |x, S)

is the computed proba-

bility that input x has class Y

3.1 Example

Suppose that we are given a task T

= (D

, D

test

). Furthermore, suppose that

= {([0, −4], 1), ([−2, −4], 2), ([−2, 4], 3), ([6, 0], 4)}, where a tuple denotes

剩余67页未读，继续阅读

syp_net

粉丝: 158
资源: 1187

深度元学习：2021全面综述

「元学习」最新研究综述(中文版)

元学习综述，meta learning，A Survey of Zero-Shot Learning + poster

元学习（meta learning）综述论文（2020年）

写一篇基于深度学习文字识别的综述论文框架

深度学习综述.pdf

深度学习无监督图像分割综述

面向深度学习的多模态融合技术研究综述_何俊.pdf

深度强化学习毕业要求

为一篇基于深度学习的超材料结构设计的硕士毕业论文创建一个大纲，这篇论文的创新点是通过深度学习的方法逆向生成二维超材料结构，并对生成的结构进行频域分析，时域分析和案例分析

写篇联邦学习分布式综述

写一篇综述论文关于 机器学习方法在CRISPR/Cas9的应用

深度学习sci三区期刊

基于深度学习的滞留物与失窃物检测算法综述

写一下Attention机制的综述论文

基于论文Learning to Discretely Compose Reasoning Module Networks for Video Captioning，写一篇文献综述

图像语义分割 深度学习

交通标志识别论文综述

请写一篇关于光声显微镜的综述论文

deep learning for 3d point clouds: a survey

写一篇关于滑坡遥感识别发展进程的综述论文

最新资源

写一篇综述论文关于机器学习方法在CRISPR/Cas9的应用

图像语义分割深度学习