![](https://csdnimg.cn/release/download_crawler_static/10488303/bg4.jpg)
are addressed similarly by considering the level sets of f, defined by Ω
t
= {x : f(x) = t} if f is continuous.
For classification, each level set is a particular class. Linear separability means that one can find w such
that f(x) ≈ hΦ(x), wi. If x ∈ Ω
t
then hΦ(x), wi ≈ t, so all Ω
t
are mapped by Φ in different hyperplanes
orthogonal to some w. The change of variable linearizes the level sets of f.
Symmetries To linearize level sets, we need to find directions along which f (x) does not vary locally, and
then linearize these directions in order to map them in a linear space. It is tempting to try to do this with
some local data analysis along x. This is not possible because the training set includes few close neighbors
in high dimension. We thus consider simultaneously all points x ∈ Ω and look for common directions along
which f(x) does not vary. This is where groups of symmetries come in. Translations and diffeomorphisms
will illustrate the difficulty to linearize high dimensional symmetries, and provide a first mathematical ground
to analyze convolution networks architectures.
We look for invertible operators which preserve the value of f. The action of an operator g on x is
written g.x. A global symmetry is an invertible and often non-linear operator g from Ω to Ω, such that
f(g.x) = f(x) for all x ∈ Ω. If g
1
and g
2
are global symmetries then g
1
.g
2
is also a global symmetry, so
products define groups of symmetries. Global symmetries are usually hard to find. We shall first concentrate
on local symmetries. We suppose that there is a metric |g|
G
which measures the distance between g ∈ G
and the identity. A function f is locally invariant to the action of G if
∀x ∈ Ω , ∃C
x
> 0 , ∀g ∈ G with |g|
G
< C
x
, f(g.x) = f (x) . (5)
We then say that G is a group of local symmetries of f. The constant C
x
is the local range of symmetries
which preserve f. Since Ω is a continuous subset of R
d
, we consider groups of operators which transport
vectors in Ω with a continuous parameter. They are called Lie groups if the group has a differential structure.
Translations and diffeomorphisms Let us interpolate the d samples of x and define x(u) for all u ∈ R
n
,
with n = 1, 2, 3 respectively for time-series, images and volumetric data. The translation group G = R
n
is an
example of Lie group. The action of g ∈ G = R
n
over x ∈ Ω is g.x(u) = x(u − g). The distance |g|
G
between
g and the identity is the Euclidean norm of g ∈ R
n
. The function f is locally invariant to translations if
sufficiently small translations of x do not change f (x). Deep convolutional networks compute convolutions,
because they assume that translations are local symmetries of f. The dimension of a group G is the number
of generators which define all group elements by products. For G = R
n
it is equal to n.
Translations are not powerful symmetries because they are defined by only n variables, and n = 2 for
images. Many image classification problems are also locally invariant to small deformations, which provide
much stronger constraints. It means that f is locally invariant to diffeomorphisms G = Diff(R
n
), which
transform x(u) with a differential warping of u ∈ R
n
. We do not know in advance what is the local range of
diffeomorphism symmetries. For example, to classify images x of hand-written digits, certain deformations
of x will preserve a digit class but modify the class of another digit. We shall linearize small diffeomorphims
g. In a space where local symmetries are linearized, we can find global symmetries by optimizing linear
projectors which preserve the values of f(x), and thus reduce dimensionality.
Local symmetries are linearized by finding a change of variable Φ(x) which locally linearizes the action
of g ∈ G. We say that Φ is Lipschitz continuous if
∃C > 0 , ∀(x, g) ∈ Ω × G , kΦ(g.x) − Φ(x)k ≤ C |g|
G
kxk . (6)
The norm kxk is just a normalization factor often set to 1. The Radon-Nikodim property proves that the
map that transforms g into Φ(g.x) is almost everywhere differentiable in the sense of Gateaux. If |g|
G
is
small then Φ(x) − Φ(g.x) is closely approximated by a bounded linear operator of g, which is the Gˆateaux
derivative. Locally, it thus nearly remains in a linear space.
4