2022/12/6 22:00
The Annotated Transformer
nlp.seas.harvard.edu/2018/04/03/attention.html
7/34
We employ a residual connection (cite) (https://arxiv.org/abs/1512.03385) around each of the two sub-
layers, followed by layer normalization (cite) (https://arxiv.org/abs/1607.06450).
class LayerNorm(nn.Module):
"Construct a layernorm module (See citation for details)."
def __init__(self, features, eps=1e-6):
super(LayerNorm, self).__init__()
self.a_2 = nn.Parameter(torch.ones(features))
self.b_2 = nn.Parameter(torch.zeros(features))
self.eps = eps
def forward(self, x):
mean = x.mean(-1, keepdim=True)
std = x.std(-1, keepdim=True)
return self.a_2 * (x - mean) / (std + self.eps) + self.b_2
That is, the output of each sub-layer is $\mathrm{LayerNorm}(x + \mathrm{Sublayer}(x))$, where
$\mathrm{Sublayer}(x)$ is the function implemented by the sub-layer itself. We apply dropout (cite)
(http://jmlr.org/papers/v15/srivastava14a.html) to the output of each sub-layer, before it is added to the
sub-layer input and normalized.
To facilitate these residual connections, all sub-layers in the model, as well as the embedding layers,
produce outputs of dimension $d_{\text{model}}=512$.
class SublayerConnection(nn.Module):
"""
A residual connection followed by a layer norm.
Note for code simplicity the norm is first as opposed to last.
"""
def __init__(self, size, dropout):
super(SublayerConnection, self).__init__()
self.norm = LayerNorm(size)
self.dropout = nn.Dropout(dropout)
def forward(self, x, sublayer):
"Apply residual connection to any sublayer with the same size."
return x + self.dropout(sublayer(self.norm(x)))
Each layer has two sub-layers. The first is a multi-head self-attention mechanism, and the second is a
simple, position-wise fully connected feed- forward network.
class EncoderLayer(nn.Module):
"Encoder is made up of self-attn and feed forward (defined below)"
def __init__(self, size, self_attn, feed_forward, dropout):
super(EncoderLayer, self).__init__()
self.self_attn = self_attn
self.feed_forward = feed_forward
self.sublayer = clones(SublayerConnection(size, dropout), 2)
self.size = size
def forward(self, x, mask):
"Follow Figure 1 (left) for connections."
x = self.sublayer[0](x, lambda x: self.self_attn(x, x, x, mask))
return self.sublayer[1](x, self.feed_forward)
剩余33页未读，继续阅读
小李飞刀李寻欢
- 粉丝: 1w+
- 资源: 16
我的内容管理
收起
- 我的资源
快来上传第一个资源
我的收益 登录查看自己的收益
我的积分
登录查看自己的积分
我的C币
登录后查看C币余额
我的收藏 
我的下载 
下载帮助
会员权益专享
最新资源
- 电力电子系统建模与控制入门
- SQL数据库基础入门：发展历程与关键概念
- DC/DC变换器动态建模与控制方法解析
- 市***专有云IaaS服务：云主机与数据库解决方案
- 紫鸟数据魔方：跨境电商选品神器，助力爆款打造
- 电力电子技术：DC-DC变换器动态模型与控制
- 视觉与实用并重：跨境电商产品开发的六重价值策略
- VB.NET三层架构下的数据库应用程序开发
- 跨境电商产品开发：关键词策略与用户痛点挖掘
- VC-MFC数据库编程技巧与实现
- 亚马逊新品开发策略：选品与市场研究
- 数据库基础知识：从数据到Visual FoxPro应用
- 计算机专业实习经验与项目总结
- Sparkle家族轻量级加密与哈希：提升IoT设备数据安全性
- SQL数据库期末考试精选题与答案解析
- H3C规模数据融合：技术探讨与应用案例解析
资源上传下载、课程学习等过程中有任何疑问或建议，欢迎提出宝贵意见哦~我们会及时处理！
点击此处反馈
