J. Sirignano, K. Spiliopoulos / Journal of Computational Physics 375 (2018) 1339–1364 1343
3. A Monte Carlo method for fast computation of second deriva tives
This section describes a modified algorithm which may be more computationally efficient in some cases. The term
L f (t, x; θ) contains second derivatives
∂
2
f
∂x
i
x
j
(t, x; θ) which may be expensive to compute in higher dimensions. For instance,
20,000 second derivatives must be calculated in d = 200 dimensions.
The complicated architectures of neural networks can make it computationally costly to calculate the second derivatives
(for example, see the neural network architecture (4.2)). The computational cost for calculating second derivatives (in both
total arithmetic operations and memory) is O(d
2
× N) where d is the spatial dimension of x and N is the batch size.
In comparison, the computational cost for calculating first derivatives is O(d × N). The cost associated with the second
derivatives is further increased since we actually need the third-order derivatives ∇
θ
∂
2
f
∂x
2
(t, x; θ) for the stochastic gradient
descent algorithm. Instead of directly calculating these second derivatives, we approximate the second derivatives using a
Monte Carlo method.
Suppose the sum of the second derivatives in L f (t, x, ; θ) is of the form
1
2
d
i
, j=1
ρ
i, j
σ
i
(x)σ
j
(x)
∂
2
f
∂x
i
x
j
(t, x; θ), assume
[ρ
i, j
]
d
i
, j=1
is a positive definite matrix, and define σ (x) =
σ
1
(x), ..., σ
d
(x)
. For example, such PDEs arise when consid-
ering
expectations of functions of stochastic differential equations, where the σ (x) represents the diffusion coefficient. See
equation (4.1) and the corresponding discussion. A generalization of the algorithm in this section to second derivatives with
nonlinear coefficient dependence on u(t, x) is also possible. Then,
d
i, j=1
ρ
i, j
σ
i
(x)σ
j
(x)
∂
2
f
∂x
i
x
j
(t, x;θ)= lim
→0
E
d
i=1
∂ f
∂x
i
(t, x +σ (x)W
;θ) −
∂ f
∂x
i
(t, x;θ)
σ
i
(x)W
i
,
(3.1)
where W
t
∈R
d
is a Brownian motion and ∈ R
+
is the step-size. The convergence rate for (3.1)is O(
√
).
2
Define:
G
1
(θ
n
, s
n
) :=
∂
f
∂t
(t
n
, x
n
;θ
n
) +L f (t
n
, x
n
;θ
n
)
2
,
G
2
(θ
n
, s
n
) :=
f (τ
n
, z
n
;θ
n
) − g(τ
n
, z
n
)
2
,
G
3
(θ
n
, s
n
) :=
f (0, w
n
;θ
n
) −u
0
(w
n
)
2
,
G(θ
n
, s
n
) := G
1
(θ
n
, s
n
) + G
2
(θ
n
, s
n
) + G
3
(θ
n
, s
n
).
The DGM algorithm use the gradient ∇
θ
G
1
(θ
n
, s
n
), which requires the calculation of the second derivative terms in
L f (t
n
, x
n
; θ
n
). Define the first derivative operators as
L
1
f (t
n
, x
n
;θ
n
) := L f (t
n
, x
n
;θ
n
) −
1
2
d
i, j=1
ρ
i, j
σ
i
(x
n
)σ
j
(x
n
)
∂
2
f
∂x
i
x
j
(t
n
, x
n
;θ).
Using (3.1), ∇
θ
G
1
is approximated as
˜
G
1
with a fixed constant > 0:
˜
G
1
(θ
n
, s
n
) := 2
∂
f
∂t
(t
n
, x
n
;θ
n
) +L
1
f (t
n
, x
n
;θ
n
) +
1
2
d
i=1
∂ f
∂x
i
(t, x
n
+σ (x
n
)W
;θ) −
∂ f
∂x
i
(t, x
n
;θ)
σ
i
(x
n
)W
i
×∇
θ
∂
f
∂t
(t
n
, x
n
;θ
n
) +L
1
f (t
n
, x
n
;θ
n
) +
1
2
d
i=1
∂ f
∂x
i
(t, x
n
+σ (x
n
)
˜
W
;θ) −
∂ f
∂x
i
(t, x
n
;θ)
σ
i
(x
n
)
˜
W
i
,
where W
is a d-dimensional normal random variable with E[W
] = 0and Cov[(W
)
i
, (W
)
j
] = ρ
i, j
.
˜
W
has the same
distribution as W
. W
and
˜
W
are independent.
˜
G
1
(θ
n
, s
n
) is a Monte Carlo approximation of ∇
θ
G
1
(θ
n
, s
n
).
˜
G
1
(θ
n
, s
n
)
has O(
√
) bias as an approximation for ∇
θ
G
1
(θ
n
, s
n
). This approximation error can be further improved via the following
scheme using “antithetic variates”:
2
Let f be a three-times differentiable function in x with bounded third-order derivatives in x. Then, it directly follows from a Taylor expansion that
d
i
, j=1
ρ
i, j
σ
i
(x)σ
j
(x)
∂
2
f
∂x
i
x
j
(t, x; θ) − E
d
i
=1
∂ f
∂x
i
(t,x+σ (x)W
;θ)−
∂ f
∂x
i
(t,x;θ)
σ
i
(x)W
i
≤
C(x)
√
. The constant C(x) depends upon ρ, f
xxx
(t, x; θ) and σ (x).
剩余25页未读，继续阅读
cs架构娃
- 粉丝: 0
- 资源: 5
我的内容管理
展开
最新资源
- Ansys Comsol实现力磁耦合仿真及其在电磁无损检测中的应用
- 西门子数控系统调试与配置实战案例教程
- ELM多输出拟合预测模型：简易Matlab实现指南
- 一维光子晶体的Comsol能带拓扑分析研究
- Borland-5技术资料压缩包分享
- Borland 6 技术资料分享包
- UE5压缩包处理技巧与D文件介绍
- 机器学习笔记：深入探讨中心极限定理
- ProE使用技巧及文件管理方法分享
- 增量式百度图片爬虫程序修复版发布
- Emlog屏蔽用户IP黑名单插件：自定义跳转与评论限制
- 安装Prometheus 2.2.1所需镜像及配置指南
- WinRARChan主题包：个性化你的压缩软件
- Neo4j关系数据映射转换测试样例集
- 安装heapster-grafana-amd64-v5-0-4所需镜像介绍
- DVB-C语言深度解析TS流
资源上传下载、课程学习等过程中有任何疑问或建议，欢迎提出宝贵意见哦~我们会及时处理！
点击此处反馈
