首先,基于图像和数值数据作为共同输入,自己创建一个CNN回归模型,其中图像和数值数据在全连接层进行拼接;然后,对CNN模型进行训练和测试;最后,基于最后一个卷积层,实现Grad-Cam可视化。PyTorch代码实现
时间: 2023-12-25 14:04:05 浏览: 84
基于pytorch实现Grad-CAM和Grad-CAM++项目源码,可以可视化任意分类网络的CAM图
5星 · 资源好评率100%
好的,以下是一个基于PyTorch的CNN回归模型实现。假设我们的输入图像为224x224,数值数据为一个1x10的向量。我们将使用3个卷积层和2个全连接层。
```
import torch
import torch.nn as nn
class CNNRegression(nn.Module):
def __init__(self):
super(CNNRegression, self).__init__()
self.conv1 = nn.Conv2d(3, 16, kernel_size=3, padding=1)
self.bn1 = nn.BatchNorm2d(16)
self.pool1 = nn.MaxPool2d(kernel_size=2, stride=2)
self.conv2 = nn.Conv2d(16, 32, kernel_size=3, padding=1)
self.bn2 = nn.BatchNorm2d(32)
self.pool2 = nn.MaxPool2d(kernel_size=2, stride=2)
self.conv3 = nn.Conv2d(32, 64, kernel_size=3, padding=1)
self.bn3 = nn.BatchNorm2d(64)
self.pool3 = nn.MaxPool2d(kernel_size=2, stride=2)
self.fc1 = nn.Linear(64 * 28 * 28 + 10, 512)
self.fc2 = nn.Linear(512, 1)
def forward(self, x, y):
x = self.pool1(torch.relu(self.bn1(self.conv1(x))))
x = self.pool2(torch.relu(self.bn2(self.conv2(x))))
x = self.pool3(torch.relu(self.bn3(self.conv3(x))))
x = torch.flatten(x, start_dim=1)
x = torch.cat((x, y), dim=1)
x = torch.relu(self.fc1(x))
x = self.fc2(x)
return x
```
我们将使用Mean Squared Error作为我们的损失函数,并使用Adam优化器进行模型训练。
```
criterion = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters())
for epoch in range(num_epochs):
running_loss = 0.0
for i, data in enumerate(trainloader, 0):
inputs, labels = data['image'], data['value']
inputs, labels = inputs.to(device), labels.to(device)
optimizer.zero_grad()
outputs = model(inputs, labels)
loss = criterion(outputs, labels.view(-1, 1).float())
loss.backward()
optimizer.step()
running_loss += loss.item()
if i % 100 == 99:
print('[%d, %5d] loss: %.3f' %
(epoch + 1, i + 1, running_loss / 100))
running_loss = 0.0
print('Finished Training')
```
接下来,我们将实现Grad-Cam可视化,以可视化CNN模型中每个类别的重要性区域。Grad-Cam基于梯度的可视化方法,因此我们需要保存CNN模型中最后一个卷积层的梯度。
```
class GradCam:
def __init__(self, model, last_conv_layer):
self.model = model
self.last_conv_layer = last_conv_layer
self.gradient = None
self.activations = None
self.last_conv_layer.register_backward_hook(self.save_gradient)
def save_gradient(self, module, grad_input, grad_output):
self.gradient = grad_output[0]
def forward(self, x, y):
x = x.to(device)
y = y.to(device)
output = self.model(x, y)
output.backward()
activations = self.last_conv_layer(x).detach()
weight = self.gradient.mean(dim=(2, 3)).unsqueeze(2).unsqueeze(3)
cam = (weight * activations).sum(dim=1, keepdim=True)
cam = torch.relu(cam)
cam = nn.functional.interpolate(cam, size=x.shape[2:], mode='bilinear', align_corners=False)
cam = cam.squeeze()
return cam.cpu().numpy()
```
最后,我们可以使用Grad-Cam可视化模型中每个类别的重要性区域。假设我们要可视化的图像和数值数据为test_image和test_value。
```
grad_cam = GradCam(model, model.conv3)
cam = grad_cam.forward(test_image, test_value)
plt.imshow(test_image.numpy().transpose((1, 2, 0)))
plt.imshow(cam, alpha=0.5, cmap='jet')
```
我们可以将alpha值设置为0.5,以使Grad-Cam的重要性区域更清晰可见。 cmap='jet'参数可以使Grad-Cam的颜色更加明亮。
阅读全文