![](https://csdnimg.cn/release/download_crawler_static/13656679/bg4.jpg)
T. Lesort, V. Lomonaco and A. Stoian et al. Information Fusion 58 (2020) 52–68
the supervision signal. Regardless of where exactly the shift happened
it has to be detected to avoid catastrophic interference with non related
skills or knowledge.
2.4. Learning paradigms orthogonal to continual learning
In this section we describe the relationship of continual learning with
respect to the main three, generally acknowledged machine learning
paradigms: supervised, unsupervised and reinforcement learning.
2.4.1. Supervised continual learning
Supervised learning is the machine learning problem of learning
from input-output example pairs [135] . For each input-output pair ( X
t
,
Y
t
), the model should learn to predict Y
t
from X
t
. X
t
is the input data,
Y
t
is the supervision signal. Supervised continual learning is a particu-
lar case where the data is not available all at once. The function should
then be learned from a sequence of data points in order to be able to
map data to labels at the end of the sequence for the whole dataset. Su-
pervised Continual Learning approaches have been mostly focused on
classication [73,92,97] .
While the study of continual learning in this context may help disen-
tangling the complexity introduced by algorithms that learn continually,
in the context of robotics, the lack of supervision does not allow, most
of the time, to apply directly supervised methods.
2.4.2. Unsupervised continual learning
Unsupervised learning refers to machine learning algorithms that do
not have labels or rewards to learn from. In the context of robotics, un-
supervised continual learning may play an important role in building in-
creasingly robust multi-modal representations over time to be later ne-
tuned with an external and very sparse feedback signal from the envi-
ronment. In order to learn robust and adaptive representations with un-
supervised learning, the main objective is to nd suitable surrogate and
meaningful learning signals, as robotics priors [64,84] , self-supervised
models or curiosity driven techniques.
A particular unsupervised task learned in a continual learning set-
ting is the generation of images. Image generation is achieved by train-
ing generative models to reproduce images from a dataset. In a CL
setting, the distribution changes over time and the generative model
should be able to produce at the end images from the whole distri-
bution. This problem has been studied for various generative models
(cf. Section 4 ) as adversarial models [81,171] , variational auto-encoders
[1,41,81,111,124] and standard auto-encoders [161,178] .
There is also a dierent relation between unsupervised learning and
CL, since unsupervised models can be used to learn representations
from vast amounts of data sources and can then generate such data (cf
Section 4.4 ). This capacity can then be used to perform CL for classi-
cation [83,150,161,172] or reinforcement learning tasks [20] . Another
use case is using data generation as a data augmentation strategy.
2.4.3. Continual reinforcement learning
Reinforcement Learning is a machine learning paradigm where the
goal is to train an agent to perform actions in a particular environment
in order to maximize the expected cumulative reward. In traditional RL,
the world is modeled as a stationary MDP: i.e., xed dynamics and states
that can recur innitely often [129]
4
. Since in general, complex RL envi-
ronments have no access to all data gathered at once, RL could often be
framed as a CL situation. Moreover, RL borrows several tools used in CL
models, such as approximating data to an i.i.d. distribution, via either i)
setting multiple agents or actors to learn in parallel [99] , or ii) using a
replay buer (or experience replay [108] ), that is equivalent to a partic-
ular category of CL (rehearsal, see Section 4.3 ). An analogy of a popular
stable method in RL is PPO algorithm [143] , which constrains learning
4
This MDP assumption was recognized and rst removed in [129] .
by using the Fisher information matrix to improve learning continually,
in the same way as some CL strategies (e.g., EWC, see Section 4.2.1 ).
Most of Continual Learning approaches in RL have been applied in sim-
ulation settings such as Atari games [73] . However, many approaches
[6,13,68,160] also tackle use cases on real robots.
3. A Framework For continual learning
Despite the rapidly growing interest in continual learning and mainly
empirical developments of the recent years [114] , very little research
and eort has been devoted to a common formalization of algorithms
that learn continually in dynamic environments. However, the availabil-
ity of a common ground for thoroughly evaluating and understanding
continual learning algorithms is essential to reduce ambiguities, enhanc-
ing fair comparisons and ultimately better advancing research in this
direction.
Being able to better compare and evaluate continual learning strate-
gies, while still being general enough to overlook implementation-
dependent details over dierent learning paradigms, becomes essential.
This is specially true when targeting deployment of CL paradigms in
real-word applications, such as robotics. Nowadays, despite the exis-
tence of a basic set of shared practices, many are the fundamental ques-
tions often overlooked in recent continual learning research. For exam-
ple, questions about the data availability during training and evaluation,
the amount of supervision with respect to the tasks separation and com-
position, as well as common but biased assumptions on the nature of
the data among others. A list of questions of interest we would like to
address and report are the following:
(a) Data Availability
•
Q
1
: Does some data need to be stored? if yes, how and what for?
(e.g. regularization, re-training, validation)?
•
Q
2
: Is the algorithm tuned based on the final performance? I.e. is it
possible to go back in time to improve performance?
•
Q
3
: Are data distributions assumed i.i.d. at any point?
•
Q
4
: Is each task assumed to be encountered only once?
(b) Prior Knowledge
•
Q
5
: Is the continual learning algorithm agnostic with respect to the
structure of the training data stream? (e.g. number of classes, num-
bers of tasks, number of learning objectives...)
•
Q
6
: Does the approach need a pretrained model for the CL setting? If
so, what is the new knowledge that needs to be acquired while learning
continually?
(c) Memory and Computational Constraints
•
Q
7
: How much available memory does the algorithm require while
learning? Does the memory capacity requirement changes as more
tasks are learned?
•
Q
8
: Is the continual learning algorithm constrained in terms of com-
putational overhead for each learning experience? Does the compu-
tational overhead increase over the task sequence?
•
Q
9
: Is the continual learning algorithm agnostic with respect to the
data type? (e.g. images, video, text,...)
•
Q
10
: Is the continual learning algorithm able to handle situations
where there is not enough time to learn?
(d) Amount/Type of Supervision
•
Q
11
: In the presence of multiple tasks, is the task label available to
the algorithm during the training phase? And during evaluation?
•
Q
12
: Are all the data labeled? or only the first training set? Can the
user provide sparse label/feedback (e.g. active learning) to correct
the system errors?
(e) Performance Expectation
•
Q
13
: What is expected from the algorithm to remember at the end of
the full stream? Is it acceptable to forget somehow, when task, context
or supervision change?
To summarize these questions, in any new CL algorithm proposition,
it is fundamental to clearly describe the data stream, its use, the algo-
55