FEDformer：融合傅里叶变换的长期序列预测Transformer
需积分: 0 107 浏览量
更新于2024-06-13
收藏 540KB PDF 举报
"FEDformer.pdf，一篇关于时间序列预测的学术论文，提出了一种名为FEDformer的新方法，该方法结合了Transformer模型和季节性趋势分解技术，旨在解决长期序列预测中的计算复杂度高和全局视野捕捉不足的问题。通过引入频率增强的Transformer，利用Fourier变换来提升对稀疏表示的捕获能力，从而提高长期预测的性能。"
在时间序列分析和预测领域，Transformer模型已经显著提升了预测的准确度，特别是在处理复杂序列任务时表现优秀。然而，尽管Transformer具有强大的注意力机制，能够捕捉到序列中的局部细节，但它们在处理长期序列预测时面临两个主要问题：计算效率低和难以把握全局趋势。
FEDformer（Frequency Enhanced Decomposed Transformer）正是为了解决这些问题而提出的。该方法融合了季节性趋势分解技术，这一技术能够将时间序列分解为季节性、趋势和其他成分，从而更好地理解数据的整体趋势。同时，FEDformer引入了一个频率增强的Transformer组件，它利用了时间序列在Fourier基中的稀疏特性。通过这种方式，模型能够更高效地学习和表示时间序列中的周期性和周期性模式。
Fourier变换是一种数学工具，可以将信号从时域转换到频域，帮助识别不同频率的成分。在FEDformer中，它被用来揭示时间序列的周期性结构，这些结构在原始数据中可能不易察觉。这种频率增强的策略使得模型能够在进行长期预测时，更准确地捕获和利用这些周期性特征。
此外，FEDformer的另一个优势是效率。与标准Transformer相比，它的线性时间复杂度意味着在处理大规模时间序列数据时，FEDformer能更快地收敛，且计算成本更低。这使得FEDformer不仅在预测精度上有所提升，而且在实际应用中更具可行性，尤其对于那些需要实时或近实时预测的场景，如能源需求预测、金融市场分析和气象预报等。
FEDformer通过结合经典的时间序列分解方法和创新的频率增强Transformer，为长期序列预测提供了一种更高效、更准确的解决方案，有望成为未来时间序列分析领域的研究热点。
Submission and Formatting Instructions for ICML 2022
(FEA) conn ecting encoder and decoder, and the Mixture
Of Experts Decomposition block (MOEDe c omp). The de-
tailed description of FEB, FEA, a nd MOEDecomp blocks
will be given in the following Section
3.2, 3.3, and 3.4 re-
spectively.
The encoder adopts a multilayer structure as: X
l
en
=
Encoder(X
l−1
en
), where l ∈ {1, ··· , N} denotes the out-
put of l-th encoder layer and X
0
en
∈ R
I×D
is the embed ded
historical series. The Enc oder(·) is formalized as
S
l,1
en
,
−
= MOEDecomp(FEB
X
l−1
en
+ X
l−1
en
),
S
l,2
en
,
−
= MOEDecomp(FeedForward
S
l,1
en
+ S
l,1
en
),
X
l
en
= S
l,2
en
,
(1)
where S
l,i
en
, i ∈ {1, 2} represents the seasonal compo-
nent a fter the i-th decomposition block in th e l-th layer r e-
spectively. For FEB module, it has two d ifferent versions
(FEB-f & FEB-w) which are implemented through Discrete
Fourier transform (DFT) and Discrete Wavelet transform
(DWT) mechanism respec tively and can seamlessly re place
the self- attention bloc k.
The decoder also a dopts a multilayer structure as:
X
l
de
, T
l
de
= D e coder(X
l−1
de
, T
l−1
de
), where l ∈ {1, ··· , M}
denotes the output of l-th decoder lay er. The Decoder(·) is
formalized as
S
l,1
de
, T
l,1
de
= MOEDecomp
FEB
X
l−1
de
+ X
l−1
de
,
S
l,2
de
, T
l,2
de
= MOEDecomp
FEA
S
l,1
de
, X
N
en
+ S
l,1
de
,
S
l,3
de
, T
l,3
de
= MOEDecomp
FeedForward
S
l,2
de
+ S
l,2
de
,
X
l
de
= S
l,3
de
,
T
l
de
= T
l−1
de
+ W
l,1
·T
l,1
de
+ W
l,2
·T
l,2
de
+ W
l,3
·T
l,3
de
,
(2)
where S
l,i
de
, T
l,i
de
, i ∈ {1, 2, 3 } represent the seaso nal and
trend component after the i-th decomposition block in the l-
th layer respectively. W
l,i
, i ∈ {1, 2, 3} represents the pro-
jector for the i-th extracted trend T
l,i
de
. Similar to FEB, FEA
has two different versions (FEA-f & FEA-w) which are im-
plemented through DFT and DWT projectio n respectively
with attention design, and can replace the cross-attention
block. The detailed description of FEA(·) will be given in
the following Section
3.3.
The final prediction is the sum of the two r e fined decom-
posed components as W
S
· X
M
de
+ T
M
de
, where W
S
is to
project the d e ep transformed seasonal component X
M
de
to
the target dimensio n.
3.2. Fourier Enhanced Structure
Discrete Fourier Transform (DFT) The proposed
Fourier Enhanced Structures u se discrete Fourier transfor m
(DFT). Le t F denotes the Fourier transform and F
−1
de-
F
F
−1
q ∈ R
L×D
Sampling
×
Q ∈ C
N ×D
C
M ×D
˜
Q ∈
R ∈
˜
Y ∈
C
M ×D
padding
Y ∈ C
N ×D
y ∈ R
L
de
×D
C
M ×D×D
X
l−1
en/de
MLP
Figure 3.
Frequency Enhanced Block with Fourier transform
(FEB-f) structure.
×
MLP
v
F + Sampling
MLP
MLP
˜
Q
˜
K
˜
V
F + Sampling
F + Sampling
σ(·)
×
Padding + F
−1
y ∈ R
L
de
×D
q ∈ R
L
de
×D
k
˜
Q
˜
K
⊤
X
l
en
S
l,1
de
Figure 4.
Frequency Enhanced Attention with Fourier transform
(FEA-f) structure, σ(·) is the activation function.
notes the inverse Fourier transform. Given a sequence of
real numbers x
n
in time domain, where n = 1, 2...N. DFT
is defined as X
l
=
P
N−1
n=0
x
n
e
−iωln
, where i is the im a g-
inary unit and X
l
, l = 1, 2...L is a sequence o f complex
numbers in the frequency domain. Similarly, the inverse
DFT is defined as x
n
=
P
L−1
l=0
X
l
e
iωln
. The complex-
ity of DFT is O(N
2
). With fast Fourier transform (FFT),
the computation complexity can be reduced to O(N log N).
Here a random subset of the Fourier basis is used and
the scale of the subset is b ounded by a scalar. When we
choose the mode index b efore DFT and reverse DFT oper-
ations, the computation complexity can be further red uced
to O(N).
Frequency Enhanced Block with Fourier Transform
(FEB-f) The FE B-f is used in both encoder and decoder
as shown in Figure
2. The input (x ∈ R
N×D
) of the
FEB-f block is first linearly projected with w ∈ R
D×D
, so
q = x·w. Then q is converted f rom the time dom ain to the
frequency domain. The Fourier transform of q is denoted
as Q ∈ C
N×D
. In frequency domain , only the randomly
selected M modes are kept so we use a select operator as
˜
Q = Select(Q) = Select(F(q)), (3)
where
˜
Q ∈ C
M×D
and M << N. Then, the FEB-f is
defined as
FEB-f(q) = F
−1
(Padding(
˜
Q ⊙ R)), (4)
where R ∈ C
D×D×M
is a parameterized kernel in itialized
randomly. Let Y = Q ⊙ C, with Y ∈ C
M×D
. The pro-
duction operator ⊙ is defined as: Y
m,d
o
=
P
D
d
i
=0
Q
m,d
i
·
R
d
i
,d
o
,m
, where d
i
= 1, 2...D is the input channel and
d
o
= 1, 2...D is the output chan nel. The result of Q ⊙ R
is then z e ro-padded to C
N×D
before performing inverse
Fourier transform bac k to the time domain. T he structure
is shown in Figure
3.
剩余18页未读，继续阅读
2023-11-15 上传
2023-06-11 上传
2024-01-05 上传
2024-04-16 上传
2023-03-09 上传
2023-07-10 上传
2023-07-16 上传
2024-11-15 上传
Chase～711
- 粉丝: 0
- 资源: 1
我的内容管理
展开
最新资源
- Java集合ArrayList实现字符串管理及效果展示
- 实现2D3D相机拾取射线的关键技术
- LiveLy-公寓管理门户：创新体验与技术实现
- 易语言打造的快捷禁止程序运行小工具
- Microgateway核心：实现配置和插件的主端口转发
- 掌握Java基本操作：增删查改入门代码详解
- Apache Tomcat 7.0.109 Windows版下载指南
- Qt实现文件系统浏览器界面设计与功能开发
- ReactJS新手实验：搭建与运行教程
- 探索生成艺术：几个月创意Processing实验
- Django框架下Cisco IOx平台实战开发案例源码解析
- 在Linux环境下配置Java版VTK开发环境
- 29街网上城市公司网站系统v1.0：企业建站全面解决方案
- WordPress CMB2插件的Suggest字段类型使用教程
- TCP协议实现的Java桌面聊天客户端应用
- ANR-WatchDog: 检测Android应用无响应并报告异常
