Transformer模块

Transformer模块是一种用于自然语言处理任务的深度学习模型架构。它由注意力机制和前馈神经网络组成，被广泛应用于机器翻译、文本生成和文本分类等任务[^1]。

以下是一个从零实现的Transformer解码器模块的例子：

import torch
import torch.nn as nn
import torch.nn.functional as F

class TransformerDecoder(nn.Module):
    def __init__(self, d_model, nhead, num_layers):
        super(TransformerDecoder, self).__init__()
        self.layers = clones(DecoderLayer(d_model, nhead), num_layers)
        self.norm = nn.LayerNorm(d_model)
        
    def forward(self, x, memory, src_mask, tgt_mask):
        for layer in self.layers:
            x = layer(x, memory, src_mask, tgt_mask)
        x = self.norm(x)
        return x

class DecoderLayer(nn.Module):
    def __init__(self, d_model, nhead):
        super(DecoderLayer, self).__init__()
        self.self_attn = nn.MultiheadAttention(d_model, nhead)
        self.linear1 = nn.Linear(d_model, 2048)
        self.linear2 = nn.Linear(2048, d_model)
        self.norm1 = nn.LayerNorm(d_model)
        self.norm2 = nn.LayerNorm(d_model)
        
    def forward(self, x, memory, src_mask, tgt_mask):
        x2 = self.self_attn(x, x, x, attn_mask=tgt_mask)
        x = x + x2
        x = self.norm1(x)
        x2 = self.linear2(F.relu(self.linear1(x)))
        x = x + x2
        x = self.norm2(x)
        return x

def clones(module, N):
    return nn.ModuleList([copy.deepcopy(module) for _ in range(N)])

这个例子展示了一个Transformer解码器模块的基本结构，包括多个解码层和层归一化操作。每个解码层由自注意力机制、前馈神经网络和层归一化组成。在前向传播过程中，输入通过多个解码层进行处理，最后经过层归一化操作得到输出。