Transformer Layer

Transformer Layer是Transformer模型中的一个基本组成部分，它由多个子层组成，每个子层都有一个残差连接和一个层归一化操作。Transformer Layer的主要作用是将输入序列映射到输出序列，其中输入序列和输出序列的维度相同。Transformer Layer的子层包括Self-Attention层和Feed-Forward层，其中Self-Attention层用于计算输入序列中每个元素与其他元素的关系，Feed-Forward层用于对Self-Attention层的输出进行非线性变换。Transformer模型中通常会堆叠多个Transformer Layer来构建深层网络。

下面是一个Transformer Layer的示例代码：

import torch
import torch.nn as nn
import torch.nn.functional as F

class TransformerLayer(nn.Module):
    def __init__(self, d_model, nhead, dim_feedforward=2048, dropout=0.1):
        super(TransformerLayer, self).__init__()
        self.self_attn = nn.MultiheadAttention(d_model, nhead, dropout=dropout)
        self.linear1 = nn.Linear(d_model, dim_feedforward)
        self.dropout = nn.Dropout(dropout)
        self.linear2 = nn.Linear(dim_feedforward, d_model)
        self.norm1 = nn.LayerNorm(d_model)
        self.norm2 = nn.LayerNorm(d_model)

    def forward(self, src, src_mask=None, src_key_padding_mask=None):
        src2 = self.self_attn(src, src, src, attn_mask=src_mask,
                              key_padding_mask=src_key_padding_mask)[0]
        src = src + self.dropout(self.linear2(F.relu(self.linear1(self.norm1(src2)))))
        src = self.norm2(src)
        return src