首先,基于图像和数值数据作为共同输入,自己创建一个CNN回归模型,其中图像和数值数据在全连接层进行拼接;然后,对CNN模型进行训练;最后,实现Grad-Cam可视化,计算最后一个卷积层中所有特征图对预测结果的权重,以热力图的形式把特征图映射到原始图片中,显示图像中用于预测的重要区域。PyTorch完整代码实现
时间: 2023-12-22 20:04:54 浏览: 303
CNN-123.rar_cnn数据_flowerkem_一维数据 CNN_一维输入 cnn_图像 cnn
5星 · 资源好评率100%
以下是基于图像和数值数据作为共同输入的CNN回归模型的PyTorch代码实现:
```python
import torch
import torch.nn as nn
import torch.optim as optim
import torchvision.transforms as transforms
class Net(nn.Module):
def __init__(self):
super(Net, self).__init__()
# Convolutional layers for image data
self.conv1 = nn.Conv2d(3, 6, 5)
self.pool = nn.MaxPool2d(2, 2)
self.conv2 = nn.Conv2d(6, 16, 5)
# Fully connected layers for numeric data and concatenated features
self.fc1 = nn.Linear(16 * 5 * 5 + 4, 120)
self.fc2 = nn.Linear(120, 84)
self.fc3 = nn.Linear(84, 1)
def forward(self, x_img, x_num):
x_img = self.pool(torch.relu(self.conv1(x_img)))
x_img = self.pool(torch.relu(self.conv2(x_img)))
x_img = x_img.view(-1, 16 * 5 * 5)
x = torch.cat((x_img, x_num), dim=1)
x = torch.relu(self.fc1(x))
x = torch.relu(self.fc2(x))
x = self.fc3(x)
return x
# Define the transform for image data
transform = transforms.Compose([
transforms.Resize(256),
transforms.CenterCrop(224),
transforms.ToTensor(),
transforms.Normalize([0.485, 0.456, 0.406], [0.229, 0.224, 0.225])
])
# Create DataLoader for the dataset
dataset = MyDataset(image_dir, labels_file, transform)
dataloader = torch.utils.data.DataLoader(dataset, batch_size=4, shuffle=True, num_workers=2)
# Initialize the model and optimizer
net = Net()
criterion = nn.MSELoss()
optimizer = optim.SGD(net.parameters(), lr=0.001, momentum=0.9)
# Train the model
for epoch in range(10): # loop over the dataset multiple times
running_loss = 0.0
for i, data in enumerate(dataloader, 0):
# Get the inputs
inputs_img, inputs_num, labels = data
# Zero the parameter gradients
optimizer.zero_grad()
# Forward + backward + optimize
outputs = net(inputs_img, inputs_num)
loss = criterion(outputs, labels)
loss.backward()
optimizer.step()
# Print statistics
running_loss += loss.item()
if i % 2000 == 1999: # print every 2000 mini-batches
print('[%d, %5d] loss: %.3f' %
(epoch + 1, i + 1, running_loss / 2000))
running_loss = 0.0
print('Finished Training')
```
在这个模型中,我们首先定义了一个Net类,它包含了图像数据和数值数据的处理过程。对于图像数据,我们使用了两个卷积层和一个最大池化层;对于数值数据,我们使用了三个全连接层。在forward函数中,我们将图像数据和数值数据拼接在一起,然后通过全连接层得到最终的回归结果。
接下来,我们定义了一个transform对象来处理图像数据,将RGB图像转换为PyTorch需要的格式,并进行了标准化处理。然后,我们创建了一个DataLoader对象来加载数据集,并将其分为小批次进行训练。
最后,我们初始化了模型、损失函数和优化器,然后循环训练模型。在每个epoch中,我们遍历整个数据集,并使用SGD优化器进行反向传播和权重更新。在每个小批次中,我们计算损失,并每隔2000个小批次打印一次平均损失。
接下来,我们实现Grad-Cam可视化,计算最后一个卷积层中所有特征图对预测结果的权重,以热力图的形式把特征图映射到原始图片中,显示图像中用于预测的重要区域。以下是实现Grad-Cam可视化的代码:
```python
import cv2
import numpy as np
import torch.nn.functional as F
# Define a function to get the Grad-CAM heatmap for a given input image and model
def get_gradcam_heatmap(img, model, layer):
# Convert the image to a PyTorch tensor
img_tensor = transform(img).unsqueeze(0)
# Get the model's prediction for the input image
outputs = model(img_tensor, ...)
_, predicted = torch.max(outputs.data, 1)
# Get the feature maps from the last convolutional layer
features = model.conv2(img_tensor)
features = F.relu(features)
# Get the gradients of the predicted class with respect to the feature maps
one_hot = torch.zeros((1, outputs.size()[-1]), dtype=torch.float32)
one_hot[0][predicted] = 1
one_hot.requires_grad = True
one_hot.backward(torch.ones_like(one_hot))
grads = model.fc2.weight.grad
pooled_grads = torch.mean(grads, dim=[0, 2, 3])
# Multiply each feature map by its corresponding gradient and take the sum
for i in range(features.size()[1]):
features[:, i, :, :] *= pooled_grads[i]
heatmap = torch.mean(features, dim=1).squeeze()
heatmap = np.maximum(heatmap.detach().numpy(), 0)
# Normalize the heatmap
heatmap /= np.max(heatmap)
# Resize the heatmap to match the original image size
heatmap = cv2.resize(heatmap, (img.shape[1], img.shape[0]))
heatmap = np.uint8(255 * heatmap)
# Apply colormap to the heatmap and overlay it on the original image
colormap = cv2.applyColorMap(heatmap, cv2.COLORMAP_JET)
overlaid = cv2.addWeighted(img, 0.7, colormap, 0.3, 0)
return overlaid
# Get an example image from the dataset
img, _, _ = dataset[0]
# Get the Grad-CAM heatmap for the example image
heatmap = get_gradcam_heatmap(img, net, net.conv2)
# Display the original image and the heatmap
cv2.imshow('Original', img)
cv2.imshow('Heatmap', heatmap)
cv2.waitKey(0)
cv2.destroyAllWindows()
```
在这个代码中,我们定义了一个get_gradcam_heatmap函数,它接受一个输入图像、一个模型和一个卷积层作为参数,并返回一个Grad-CAM热力图。首先,我们将输入图像转换为PyTorch张量,并获取模型的预测结果。然后,我们获取最后一个卷积层的特征图,并计算预测类别对特征图的梯度。接下来,我们将每个特征图乘以其对应的梯度,并求和得到一个特征图权重矩阵。我们使用这个权重矩阵对特征图进行加权平均,并对结果进行非线性处理和归一化。最后,我们将热力图重新调整大小以匹配原始图像的大小,并将其与原始图像叠加在一起。
在主函数中,我们首先从数据集中获取一个样本图像,然后调用get_gradcam_heatmap函数获取其Grad-CAM热力图,并显示原始图像和热力图。
阅读全文