深度学习中的不确定性估计：理论与应用

5星 · 超过95%的资源需积分: 41 171 浏览量更新于2023-05-22 收藏 8.83MB PDF 举报

"不确定性在深度学习中的研究构成了这篇博士学位论文的核心内容。作者Yarin Gal在剑桥大学工程系进行工作，通过对深度学习模型的理论构建和发展，提出了将现代深度学习工具转化为具有实际不确定性估计的Bayesian模型的方法，而无需改变模型结构或优化过程。论文的第一部分着重于理论构建，探讨了dropout和其他随机正则化技术与近似贝叶斯模型之间的联系，并通过实证评估了这些关联。通过这些工具，论文展示了深度学习与贝叶斯建模之间的应用，例如在图像数据的主动学习和数据高效的强化学习中的实践应用。此外，论文还涵盖了语言处理、医疗诊断、生物信息学、图像处理和自动驾驶等领域中的案例研究，证明了这些技术的实用性。论文进一步探索了深度学习与贝叶斯建模之间的深刻见解，以及这些关联对模型不确定性的决定因素。作者深入分析了线性情况下的近似推理，并理论研究了诸如spike-and-slab先验等不同类型的先验。通过这些分析，读者可以理解模型的不确定性是如何被建模和控制的。值得注意的是，这项工作的原创性和独立性得到了强调，声明了除了明确引用他人的工作外，论文的内容都是作者独立完成，没有在其他学位或学术机构中提交过。此外，作者表达了对多位同行、导师如Christof Angermüller、Yoshua Bengio、Phil Blunsom等人在论文写作过程中提供评论和讨论的感谢。这篇论文为深度学习领域的不确定性估计提供了一种新的视角，不仅推动了理论的发展，也促进了深度学习在众多实际场景中的应用，尤其是在需要可靠预测和解释能力的领域，如医疗和自动驾驶等。"

2 Introduction: The Importance of Knowing What We Don’t Know

a thing. The main goal of this thesis is to develop such practical tools to reason about

uncertainty in deep learning.

1.1 Deep learning

To introduce deep learning, I shall start from the simplest of the statistical tools: linear

regression [Gauss, 1809; Legendre, 1805; Seal, 1967]. In linear regression we are given

a set of

input-output pairs

{

(

, y

)

, ...,

(

, y

)

}

, for example CO

-temperature

observations, or the average number of accidents for diﬀerent driving speeds. We assume

that there exists a linear function mapping each

∈ R

(with

potentially

corrupted with observation noise). Our model in this case is a linear transformation of the

inputs:

(

) =

, with

some

matrix over the reals and

a real vector

with

elements. Diﬀerent parameters

W, b

deﬁne diﬀerent linear transformations, and

our aim is to ﬁnd parameters that, for example, would minimise the average squared

error over our observed data:

||y

− (x

W + b)||

In more general cases where the relation between

and

need not be linear, we may

wish to deﬁne a non-linear function

(

) mapping the inputs to the outputs. For this we

can resort to linear basis function regression [Bishop, 2006; Gergonne, 1815; Smith, 1918],

where the input

is fed through

ﬁxed scalar-valued non-linear transformations

(

)

to compose a feature vector Φ(

) = [

(

)

, ..., ϕ

(

)]. We then perform linear regression

with this vector instead of

itself. The transformations

are our basis functions, and

with scalar input

, these can be wavelets parametrised by

, polynomials of diﬀerent

degrees

, or sinusoidals with various frequencies. When

(

) :=

and

, basis

function regression reduces to linear regression. The basis functions are often assumed to

be ﬁxed and orthogonal to each other, and an optimal combination of these functions is

sought.

Relaxing the constraint for the basis functions to be ﬁxed and mutually orthogonal, we

can use parametrised basis functions instead [Bishop, 2006]. For example, we may deﬁne

the basis functions to be

where the scalar-valued function

is applied to the

inner-product

⟨w

, x⟩

. In this case

are often deﬁned to be identical for all

, for

example

(

) =

sin

(

) giving

(

) =

sin

(

⟨w

, x⟩

). The feature vector composed

of the basis functions’ output is again fed as the input to a linear transformation. The

model output can then be written more compactly as

(

) = Φ

(

)

with

(

) =

(

a matrix of dimensions

a vector with

elements,

a matrix of dimensions

, and

a vector with

elements. To

1.1 Deep learning 3

perform regression now we can ﬁnd

, b

as well as

, b

that minimise the average

squared error over our observed data, ||y − f(x)||

The most basic model in deep learning can be described as a hierarchy of these

parametrised basis functions (such a hierarchy is referred to as a neural network for

historical reasons, with each feature vector in the hierarchy referred to as a layer). In the

simplest setting of regression we would simply compose multiple basis function regression

models, and for classiﬁcation we would further compose a logistic function at the end

(which “squashes” the linear model’s output to obtain a probability vector). Each layer

in the hierarchy can be seen as a “building block”, and the modularity in the composition

of such blocks embodies the versatility of deep learning models. The simplicity of each

block, together with the many possibilities of model combinations, might be what led

many engineers to work in the ﬁeld. This in turn has led to the development of tools

that work well and scale well.

We will continue with a review of simple neural network models, relating the notation

in the ﬁeld of deep learning to the mathematical formalism of the above linear basis

function regression models. We then extend these to specialised models designed to

process image data and sequence data. In the process we will introduce some of the

terminology and mathematical notation used throughout the work. We will describe

the models formally but succinctly, which will allow us to continue our discussion in the

introduction using a more precise language.

Feed-forward neural networks (NNs).

We will ﬁrst review a neural network model

[Rumelhart et al., 1985] for a single hidden layer. This is done for ease of notation, and

the generalisation to multiple layers is straightforward. We denote by

the model input

(referred to as input layer, a row vector with

elements), and transform it with an

aﬃne transformation to a row vector with

elements. We denote by

the linear

map (referred to as a weight matrix) and by

the translation (referred to as a bias)

used to transform the input

to obtain

. An element-wise non-linearity

(

)

(such as the rectiﬁed linear

(ReLU) or TanH) is then applied to the transformation

output, resulting in a hidden layer with each element referred to as a network unit. This

is followed by a second linear transformation with weight matrix

mapping the hidden

layer to the model output (referred to as output layer, a row vector with

elements).

These two layers are also referred to as inner-product layers. We thus have

is a

Q × K

matrix,

is a

K × D

matrix, and

is a

dimensional vector. A standard

relu(x) = max(x, 0).

4 Introduction: The Importance of Knowing What We Don’t Know

network would output

y = σ(xW

+ b)W

given some input

To use the network for regression we might use the Euclidean loss,

(X, Y) =

i=1

||y

−

(1.1)

where

, ..., y

}

are

observed outputs, and

{

, ...,

}

are the outputs of the model

with corresponding observed inputs

, ..., x

}

. Minimising this loss w.r.t.

, W

, b

would hopefully result in a model that can generalise well to unseen test data

test

, Y

test

To use the model for classiﬁcation, predicting the probability of

being classiﬁed with

a label in the set

{

, ..., D}

, we pass the output of the model

through an element-wise

softmax function to obtain normalised scores:

exp

(

)

/ (

′

exp(

′

))

. Taking the

log of

(with d being the observed label) results in a softmax loss,

(X, Y) = −

i=1

log(

i,d

) (1.2)

where d

∈ {1, 2, ..., D} is the observed class for input i.

A big diﬃculty with the models above is their tendency to overﬁt—decrease their

loss on the training set

X, Y

while increasing their loss on the test set

test

, Y

test

. For

this reason a regularisation term is often added during optimisation. We often use

regularisation for each parameter weighted by some weight decays

, resulting in a

minimisation objective (often referred to as a cost),

L(W

, W

, b) := E

(X, Y) + λ

||W

+ λ

||W

+ λ

||b||

. (1.3)

The above single hidden layer NN with the Euclidean loss is identical to a basis function

regression model. Extending this simple NN model to multiple layers results in a more

expressive model.

Remark

(Model expressiveness)

An intuitive deﬁnition for model expressiveness

might be the complexity of functions a model can capture (deﬁning what complex

functions are is not trivial by itself, although when dealing with polynomials one

might deﬁne polynomials of high degree to be more complex than polynomials of

Note that the output bias was omitted here; this is equivalent to centring the observed outputs.

1.1 Deep learning 5

low degree). In this sense hierarchical basis function models are more expressive

than their “ﬂat” counter-parts. It is interesting to note that even though “ﬂat”

basis function regression can model any function up to any given precision with

a large enough number of basis functions [Cybenko, 1989; Hornik, 1991], with a

hierarchy we can use much smaller models. Consider the example of basis function

regression with polynomial basis functions

∈ {

, x, x

, ..., x

K−1

}

: the set of

functions expressible with these

basis functions is {all polynomials up to degree

K −

1}. Composing the basis function regression model

times results in a

model that can capture (a subset of) functions from the set {all polynomials up to

degree (

K −

}. A “ﬂat” model capturing polynomials up to degree (

K −

would require

basis functions, whereas a hierarchical model with similar (but

not identical) expressiveness would only require

K × L

basis functions. Model

expressiveness is further discussed in [Bengio and LeCun, 2007] for example, where

binary circuits are used as an illustrative example.

The simple model structure presented above can be extended to specialised models,

aimed at treating image inputs or sequence inputs. We will next review these models

quickly.

Convolutional neural networks (CNNs).

CNNs [LeCun et al., 1989; Rumelhart

et al., 1985] are popular deep learning tools for image processing, which can solve tasks

that until recently were considered to lie beyond our reach [Krizhevsky et al., 2012;

Szegedy et al., 2014]. The model is made of a recursive application of convolution and

pooling layers, followed by inner product layers at the end of the network (simple NNs

as described above). A convolution layer is a linear transformation that preserves spatial

information in the input image (depicted in ﬁgure 1.1). Pooling layers simply take the

output of a convolution layer and reduce its dimensionality (by taking the maximum of

each (2

2) block of pixels for example). The convolution layer will be explained in more

detail in section §3.4.1.

Similarly to CNNs, recurrent neural networks are specialised models designed to

handle sequence data.

Recurrent neural networks (RNNs).

RNNs [Rumelhart et al., 1985; Werbos, 1988]

are sequence-based models of key importance for natural language understanding, lan-

guage generation, video processing, and many other tasks [Kalchbrenner and Blunsom,

2013; Mikolov et al., 2010; Sundermeyer et al., 2012; Sutskever et al., 2014]. The model’s

剩余173页未读，继续阅读

amour0425

粉丝: 0
资源: 6

深度学习中的不确定性估计：理论与应用

uncertainty_estimation_deep_learning:该存储库提供了用于实现框架的代码，该框架提供了具有总不确定性估计的深度学习模型，如“深度学习不确定性估计通用框架”（Loquercio，Segù，Scaramuzza。RA-L 2020）中所述。

Uncertainty.rar

ConvNet_Uncertainty：使用Dropout计算不确定性

高斯过程回归的文献综述

system-level-RUL uncertainty

matlab Uncertainty Toolbox下载

matlab如何下载Uncertainty Toolbox

multi-task learning using uncertainty to weigh losses for scene geometry and

mutual information (MI)

Matlab代码：将lstm网络中的权重和偏差设置为正态分布，然后用SVI在训练过程中生成参数的后验分布，并且预测变量的不确定区间

最新资源