![](https://csdnimg.cn/release/download_crawler_static/19347404/bg3.jpg)
Published as a conference paper at ICLR 2020
Deformation Modeling: We refer to deformation modeling as learning geometric transformations
in 2D image space without regard to 3D. One angle to attack deformation modeling is to craft cer-
tain geometric invariances into networks. However, this usually requires designs specific to certain
kinds of deformation, such as shift, rotation, reflection and scaling (Sifre & Mallat, 2013; Bruna &
Mallat, 2013; Kanazawa et al., 2016; Cohen & Welling, 2016; Worrall et al., 2017; Esteves et al.,
2018). Another line of work on this topic learns to recompose data by either semi-parameterized or
completely free-form sampling in image space: Spatial Transformers (Jaderberg et al., 2015) learns
2D affine transformations, Deep Geometric Matchers (Rocco et al., 2017) learns thin-plate spline
transformations, Deformable Convolutions (Dai et al., 2017; Zhu et al., 2019) learns free-form trans-
formations.
We interpret sampling data space as an effective approach to adapt effective receptive fields (ERF)
by directly changing receptive field. At a high-level, our Deformable Kernels (DKs) share intu-
itions with this line of works for learning geometric transformations, yet are instantiated by learning
to sample in kernel space which directly adapt ERF while leaving theoretical receptive fields un-
touched. While kernel space sampling is also studied in Deformable Filter (Xiong et al., 2019) and
KPConv (Thomas et al., 2019), but in their contexts, sampling grids are computed from input point
clouds rather than learned from data corpora.
Dynamic Inference: Dynamic inference adapts the model or individual operators to the observed
data. The computation of our approach differs from self-attention (Vaswani et al., 2017; Wang et al.,
2018) in which linear or convolution modules are augmented with subsequent queries that extract
from the same input. We consider our closest related works in terms of implementation as those
approaches that adapt convolutional kernels at run time. It includes but is not limited to Dynamic
Filters (Jia et al., 2016), Selective Kernels (Li et al., 2019) and Conditional Convolutions (Yang
et al., 2019). All of these approaches can learn and infer customized kernel spaces with respect
to the data, but are either less inefficient or are loosely formulated. Dynamic Filters generate new
filters from scratch, while Conditional Convolutions extend this idea to linear combinations of a set
of synthesized filters. Selective Kernels are, on the other hand, comparably lightweight, but aggre-
gating activations from kernels of different size is not as compact as directly sampling the original
kernel space. Another line of works contemporary to ours (Shelhamer et al., 2019; Wang et al.,
2019) is to compose free-form filters with structured Gaussian filters, which essentially transforms
kernel spaces by data. Our DKs also differ from these works with the emphasize of direct adaptation
the ERF rather than the theoretical receptive field. As mentioned previously, the true goal should be
to adapt the ERF, and to our knowledge, our work is the first to study dynamic inference of ERFs.
3 APPROACH
We start by covering preliminaries on convolutions, including the definition of effective receptive
field (ERF). We then formulate a theoretical framework for analyzing ERFs, and thus motivate our
Deformable Kernels (DKs). We finally elaborate different DK variants within such a framework.
Our analysis suggests compatibility between DKs and the prior work.
3.1 A DIVE INTO CONVOLUTIONS
2D Convolution: Let us first consider an input image I ∈ R
D×D
. By convolving it with a ker-
nel W ∈ R
K×K
of stride 1, we have an output image O whose pixel values at each coordi-
nate j ∈ R
2
can be expressed as
O
j
=
X
k∈K
I
j+k
W
k
, (1)
by enumerating discrete kernel positions k within the support K = [−K/2, K/2]
2
∩Z. This defines
a rigid grid for sampling data and kernels.
Theoretical Receptive Field: The same kernel W can be stacked repeatedly to form a linear con-
volutional network with n layers. The theoretical receptive field can then be imagined as the “ac-
cumulative coverage” of kernels at each given output unit on the input image by deconvolving back
through the network. This property characterizes a set of input fields that could fire percepts onto
corresponding output pixels. The size of a theoretical receptive field scales linearly with respect to
the network depth n and kernel size K (He et al., 2016).
3