计算困惑度代码

以下是使用Python计算困惑度的示例代码：

import torch
import torch.nn as nn
from torch.utils.data import DataLoader
from torch.nn.utils.rnn import pad_sequence
from dataset import MyDataset

# 加载数据集
test_dataset = MyDataset('test.txt')
test_loader = DataLoader(test_dataset, batch_size=32, shuffle=False)

# 加载模型
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = MyModel()
model.load_state_dict(torch.load('model.pth'))
model.to(device)

# 计算困惑度
criterion = nn.CrossEntropyLoss()
total_loss = 0
total_words = 0
with torch.no_grad():
    for batch in test_loader:
        inputs, targets = batch
        inputs = pad_sequence(inputs, batch_first=True).to(device)
        targets = pad_sequence(targets, batch_first=True).to(device)
        outputs = model(inputs)
        loss = criterion(outputs.reshape(-1, outputs.shape[-1]), targets.reshape(-1))
        total_loss += loss.item() * targets.numel()
        total_words += targets.numel()
perplexity = torch.exp(total_loss / total_words)
print('Perplexity:', perplexity)

其中，`MyDataset`是自定义的数据集类，`MyModel`是自定义的语言模型类。在每个batch中，使用`pad_sequence`对输入和目标序列进行填充，以保证在同一个batch中的序列长度一致。最后根据公式计算困惑度并输出结果。