Meta-Learning with Differentiable Convex Optimization
Kwonjoon Lee
2
Subhransu Maji
1,3
Avinash Ravichandran
1
Stefano Soatto
1,4
1
Amazon Web Services
2
UC San Diego
3
UMass Amherst
4
UCLA
kwl042@ucsd.edu {smmaji,ravinash,soattos}@amazon.com
Abstract
Many meta-learning approaches for few-shot learning
rely on simple base learners such as nearest-neighbor clas-
sifiers. However, even in the few-shot regime, discrimina-
tively trained linear predictors can offer better generaliza-
tion. We propose to use these predictors as base learners to
learn representations for few-shot learning and show they
offer better tradeoffs between feature size and performance
across a range of few-shot recognition benchmarks. Our
objective is to learn feature embeddings that generalize well
under a linear classification rule for novel categories. To
efficiently solve the objective, we exploit two properties of
linear classifiers: implicit differentiation of the optimality
conditions of the convex problem and the dual formulation
of the optimization problem. This allows us to use high-
dimensional embeddings with improved generalization at a
modest increase in computational overhead. Our approach,
named MetaOptNet, achieves state-of-the-art performance
on miniImageNet, tieredImageNet, CIFAR-FS, and FC100
few-shot learning benchmarks. Our code is available on-
line
1
.
1. Introduction
The ability to learn from a few examples is a hallmark
of human intelligence, yet it remains a challenge for mod-
ern machine learning systems. This problem has received
significant attention from the machine learning community
recently where few-shot learning is cast as a meta-learning
problem (e.g., [22, 8, 33, 28]). The goal is to minimize gen-
eralization error across a distribution of tasks with few train-
ing examples. Typically, these approaches are composed of
an embedding model that maps the input domain into a fea-
ture space and a base learner that maps the feature space
to task variables. The meta-learning objective is to learn
an embedding model such that the base learner generalizes
well across tasks.
While many choices for base learners exist, nearest-
neighbor classifiers and their variants (e.g., [28, 33]) are
1
https://github.com/kjunelee/MetaOptNet
popular as the classification rule is simple and the approach
scales well in the low-data regime. However, discrimina-
tively trained linear classifiers often outperform nearest-
neighbor classifiers (e.g., [4, 16]) in the low-data regime
as they can exploit the negative examples which are often
more abundant to learn better class boundaries. Moreover,
they can effectively use high dimensional feature embed-
dings as model capacity can be controlled by appropriate
regularization such as weight sparsity or norm.
Hence, in this paper, we investigate linear classifiers as
the base learner for a meta-learning based approach for few-
shot learning. The approach is illustrated in Figure 1 where
a linear support vector machine (SVM) is used to learn a
classifier given a set of labeled training examples and the
generalization error is computed on a novel set of examples
from the same task. The key challenge is computational
since the meta-learning objective of minimizing the gener-
alization error across tasks requires training a linear classi-
fier in the inner loop of optimization (see Section 3). How-
ever, the objective of linear models is convex and can be
solved efficiently. We observe that two additional properties
arising from the convex nature that allows efficient meta-
learning: implicit differentiation of the optimization [2, 11]
and the low-rank nature of the classifier in the few-shot set-
ting. The first property allows the use of off-the-shelf con-
vex optimizers to estimate the optima and implicitly differ-
entiate the optimality or Karush-Kuhn-Tucker (KKT) con-
ditions to train embedding model. The second property
means that the number of optimization variables in the dual
formation is far smaller than the feature dimension for few-
shot learning.
To this end, we have incorporated a differentiable
quadratic programming (QP) solver [1] which allows end-
to-end learning of the embedding model with various linear
classifiers, e.g., multiclass support vector machines (SVMs)
[5] or linear regression, for few-shot classification tasks.
Making use of these properties, we show that our method
is practical and offers substantial gains over nearest neigh-
bor classifiers at a modest increase in computational costs
(see Table 3). Our method achieves state-of-the-art perfor-
mance on 5-way 1-shot and 5-shot classification for popu-
arXiv:1904.03758v1 [cs.CV] 7 Apr 2019
下载后可阅读完整内容，剩余8页未读，立即下载
liz_lee
- 粉丝: 68
- 资源: 36
我的内容管理
展开
最新资源
- WebLogic集群配置与管理实战指南
- AIX5.3上安装Weblogic 9.2详细步骤
- 面向对象编程模拟试题详解与解析
- Flex+FMS2.0中文教程：开发流媒体应用的实践指南
- PID调节深入解析：从入门到精通
- 数字水印技术：保护版权的新防线
- 8位数码管显示24小时制数字电子钟程序设计
- Mhdd免费版详细使用教程：硬盘检测与坏道屏蔽
- 操作系统期末复习指南：进程、线程与系统调用详解
- Cognos8性能优化指南：软件参数与报表设计调优
- Cognos8开发入门：从Transformer到ReportStudio
- Cisco 6509交换机配置全面指南
- C#入门：XML基础教程与实例解析
- Matlab振动分析详解：从单自由度到6自由度模型
- Eclipse JDT中的ASTParser详解与核心类介绍
- Java程序员必备资源网站大全
资源上传下载、课程学习等过程中有任何疑问或建议，欢迎提出宝贵意见哦~我们会及时处理！
点击此处反馈
