怎么构建Transformer Encoder

Transformer Encoder是由多个相同的Encoder层组成的，每个Encoder层包括self-attention和前馈神经网络两部分。下面是一个构建Transformer Encoder的示例代码：

import torch
import torch.nn as nn
import torch.nn.functional as F

class TransformerEncoder(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, num_heads, dropout):
        super(TransformerEncoder, self).__init__()

        # 定义多层Encoder层
        self.encoders = nn.ModuleList([EncoderLayer(input_size, hidden_size, num_heads, dropout) for _ in range(num_layers)])

    def forward(self, x, mask):
        for encoder in self.encoders:
            x = encoder(x, mask)
        return x

class EncoderLayer(nn.Module):
    def __init__(self, input_size, hidden_size, num_heads, dropout):
        super(EncoderLayer, self).__init__()

        # self-attention部分
        self.self_attn = MultiheadAttention(input_size, num_heads, dropout)

        # 前馈神经网络部分
        self.feedforward = nn.Sequential(
            nn.Linear(input_size, hidden_size),
            nn.ReLU(),
            nn.Dropout(dropout),
            nn.Linear(hidden_size, input_size),
            nn.Dropout(dropout)
        )

        # LayerNorm层
        self.norm1 = nn.LayerNorm(input_size)
        self.norm2 = nn.LayerNorm(input_size)

    def forward(self, x, mask):
        # self-attention部分
        attn_output, _ = self.self_attn(x, x, x, mask)
        x = self.norm1(x + attn_output)

        # 前馈神经网络部分
        ff_output = self.feedforward(x)
        x = self.norm2(x + ff_output)

        return x

class MultiheadAttention(nn.Module):
    def __init__(self, input_size, num_heads, dropout):
        super(MultiheadAttention, self).__init__()

        # 定义多头注意力
        self.num_heads = num_heads
        self.attention = nn.MultiheadAttention(input_size, num_heads, dropout)

        # LayerNorm层和Dropout层
        self.norm = nn.LayerNorm(input_size)
        self.dropout = nn.Dropout(dropout)

    def forward(self, x, mask=None):
        # 使用多头注意力
        attn_output, _ = self.attention(x, x, x, mask)

        # LayerNorm层和Dropout层
        x = self.norm(x + self.dropout(attn_output))

        return x

上述代码中，Transformer Encoder是由多个EncoderLayer组成的，每个EncoderLayer包括self-attention和前馈神经网络两部分。其中，Self-Attention部分使用了MultiheadAttention模型，前馈神经网络部分使用了全连接神经网络。每个EncoderLayer之间使用了LayerNorm层进行归一化处理。在EncoderLayer的forward函数中，分别对self-attention和前馈神经网络部分进行计算，然后将结果与输入x相加，再经过LayerNorm层和Dropout层输出。在TransformerEncoder的forward函数中，对每个EncoderLayer进行循环调用，输出最终的Encoder结果。