T. Lesort, V. Lomonaco and A. Stoian et al. Information Fusion 58 (2020) 52–68
the supervision signal. Regardless of where exactly the shift happened
it has to be detected to avoid catastrophic interference with non related
skills or knowledge.
2.4. Learning paradigms orthogonal to continual learning
In this section we describe the relationship of continual learning with
respect to the main three, generally acknowledged machine learning
paradigms: supervised, unsupervised and reinforcement learning.
2.4.1. Supervised continual learning
Supervised learning is the machine learning problem of learning
from input-output example pairs [135] . For each input-output pair ( X
t
,
Y
t
), the model should learn to predict Y
t
from X
t
. X
t
is the input data,
Y
t
is the supervision signal. Supervised continual learning is a particu-
lar case where the data is not available all at once. The function should
then be learned from a sequence of data points in order to be able to
map data to labels at the end of the sequence for the whole dataset. Su-
pervised Continual Learning approaches have been mostly focused on
classication [73,92,97] .
While the study of continual learning in this context may help disen-
tangling the complexity introduced by algorithms that learn continually,
in the context of robotics, the lack of supervision does not allow, most
of the time, to apply directly supervised methods.
2.4.2. Unsupervised continual learning
Unsupervised learning refers to machine learning algorithms that do
not have labels or rewards to learn from. In the context of robotics, un-
supervised continual learning may play an important role in building in-
creasingly robust multi-modal representations over time to be later ne-
tuned with an external and very sparse feedback signal from the envi-
ronment. In order to learn robust and adaptive representations with un-
supervised learning, the main objective is to nd suitable surrogate and
meaningful learning signals, as robotics priors [64,84] , self-supervised
models or curiosity driven techniques.
A particular unsupervised task learned in a continual learning set-
ting is the generation of images. Image generation is achieved by train-
ing generative models to reproduce images from a dataset. In a CL
setting, the distribution changes over time and the generative model
should be able to produce at the end images from the whole distri-
bution. This problem has been studied for various generative models
(cf. Section 4 ) as adversarial models [81,171] , variational auto-encoders
[1,41,81,111,124] and standard auto-encoders [161,178] .
There is also a dierent relation between unsupervised learning and
CL, since unsupervised models can be used to learn representations
from vast amounts of data sources and can then generate such data (cf
Section 4.4 ). This capacity can then be used to perform CL for classi-
cation [83,150,161,172] or reinforcement learning tasks [20] . Another
use case is using data generation as a data augmentation strategy.
2.4.3. Continual reinforcement learning
Reinforcement Learning is a machine learning paradigm where the
goal is to train an agent to perform actions in a particular environment
in order to maximize the expected cumulative reward. In traditional RL,
the world is modeled as a stationary MDP: i.e., xed dynamics and states
that can recur innitely often [129]
4
. Since in general, complex RL envi-
ronments have no access to all data gathered at once, RL could often be
framed as a CL situation. Moreover, RL borrows several tools used in CL
models, such as approximating data to an i.i.d. distribution, via either i)
setting multiple agents or actors to learn in parallel [99] , or ii) using a
replay buer (or experience replay [108] ), that is equivalent to a partic-
ular category of CL (rehearsal, see Section 4.3 ). An analogy of a popular
stable method in RL is PPO algorithm [143] , which constrains learning
4
This MDP assumption was recognized and rst removed in [129] .
by using the Fisher information matrix to improve learning continually,
in the same way as some CL strategies (e.g., EWC, see Section 4.2.1 ).
Most of Continual Learning approaches in RL have been applied in sim-
ulation settings such as Atari games [73] . However, many approaches
[6,13,68,160] also tackle use cases on real robots.
3. A Framework For continual learning
Despite the rapidly growing interest in continual learning and mainly
empirical developments of the recent years [114] , very little research
and eort has been devoted to a common formalization of algorithms
that learn continually in dynamic environments. However, the availabil-
ity of a common ground for thoroughly evaluating and understanding
continual learning algorithms is essential to reduce ambiguities, enhanc-
ing fair comparisons and ultimately better advancing research in this
direction.
Being able to better compare and evaluate continual learning strate-
gies, while still being general enough to overlook implementation-
dependent details over dierent learning paradigms, becomes essential.
This is specially true when targeting deployment of CL paradigms in
real-word applications, such as robotics. Nowadays, despite the exis-
tence of a basic set of shared practices, many are the fundamental ques-
tions often overlooked in recent continual learning research. For exam-
ple, questions about the data availability during training and evaluation,
the amount of supervision with respect to the tasks separation and com-
position, as well as common but biased assumptions on the nature of
the data among others. A list of questions of interest we would like to
address and report are the following:
(a) Data Availability
•
Q
1
: Does some data need to be stored? if yes, how and what for?
(e.g. regularization, re-training, validation)?
•
Q
2
: Is the algorithm tuned based on the final performance? I.e. is it
possible to go back in time to improve performance?
•
Q
3
: Are data distributions assumed i.i.d. at any point?
•
Q
4
: Is each task assumed to be encountered only once?
(b) Prior Knowledge
•
Q
5
: Is the continual learning algorithm agnostic with respect to the
structure of the training data stream? (e.g. number of classes, num-
bers of tasks, number of learning objectives...)
•
Q
6
: Does the approach need a pretrained model for the CL setting? If
so, what is the new knowledge that needs to be acquired while learning
continually?
(c) Memory and Computational Constraints
•
Q
7
: How much available memory does the algorithm require while
learning? Does the memory capacity requirement changes as more
tasks are learned?
•
Q
8
: Is the continual learning algorithm constrained in terms of com-
putational overhead for each learning experience? Does the compu-
tational overhead increase over the task sequence?
•
Q
9
: Is the continual learning algorithm agnostic with respect to the
data type? (e.g. images, video, text,...)
•
Q
10
: Is the continual learning algorithm able to handle situations
where there is not enough time to learn?
(d) Amount/Type of Supervision
•
Q
11
: In the presence of multiple tasks, is the task label available to
the algorithm during the training phase? And during evaluation?
•
Q
12
: Are all the data labeled? or only the first training set? Can the
user provide sparse label/feedback (e.g. active learning) to correct
the system errors?
(e) Performance Expectation
•
Q
13
: What is expected from the algorithm to remember at the end of
the full stream? Is it acceptable to forget somehow, when task, context
or supervision change?
To summarize these questions, in any new CL algorithm proposition,
it is fundamental to clearly describe the data stream, its use, the algo-
55
剩余16页未读，继续阅读
syp_net
- 粉丝: 158
- 资源: 1187
我的内容管理
展开
最新资源
- 中国微型数字传声器：技术革新与市场前景
- 智能安防：基于Hi3515的嵌入式云台控制系统设计
- 手机电量低时辐射真增千倍？解析手机使用谣言
- 56F803型DSP驱动的高精度大功率超声波电源控制策略研究
- ARM与GPRS结合的远程监测系统设计
- GPS与RFID技术结合的智能巡检系统设计
- CPLD驱动的低功耗爆炸场温度测试系统设计
- 基于FPGA的智能驱动控制系统：可扩展设计与工业网络协议
- 基于ATmega128和CH374的嵌入式USB接口设计
- 基于AT89C52的温度补偿超声波测距仪：高精度设计与应用
- MSP430F448单片机在交流数字电压表中的应用
- 提升变频器应用效率的12项实用技巧
- STM32F103在数字电镀电源并联均流系统中的应用
- PSpice仿真下的升压开关电源设计：拓扑分析与CCM稳定性提升
- 轻巧高效：MSP430主导的低成本无线传感器网络节点设计
- FPGA在EDA/PLD中实现LVDS接口的应用解析
资源上传下载、课程学习等过程中有任何疑问或建议，欢迎提出宝贵意见哦~我们会及时处理！
点击此处反馈
