根据提供的data.csv 附件(comment、star),用python语言对外卖评论进行分类,2分类,正向负向,两种方式:sklearn调用以及自己编写模型类。根据分类结果,分别画出正向词云图,负向词云图
时间: 2024-03-19 22:43:10 浏览: 192
好的,我会尽力回答你的问题。首先需要说明的是,我们需要在进行分类前对评论数据进行预处理,包括去除停用词、分词、词性标注等。这里我将使用Python中的nltk库进行预处理。
首先,我们需要安装nltk库,可以通过以下命令进行安装:
```python
!pip install nltk
```
安装完成后,我们可以通过以下代码对评论数据进行预处理:
```python
import pandas as pd
import nltk
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
from nltk.stem import WordNetLemmatizer
# 下载停用词
nltk.download('stopwords')
# 下载punkt
nltk.download('punkt')
# 下载wordnet
nltk.download('wordnet')
# 读取评论数据
data = pd.read_csv('data.csv')
# 定义停用词
stop_words = set(stopwords.words('english'))
# 初始化词性还原器
wordnet_lemmatizer = WordNetLemmatizer()
# 定义预处理函数
def preprocess(text):
# 分词
words = word_tokenize(text.lower())
# 去除停用词
words = [word for word in words if word not in stop_words]
# 词性还原
words = [wordnet_lemmatizer.lemmatize(word, pos='v') for word in words]
# 返回预处理后的评论
return " ".join(words)
# 对评论进行预处理
data['comment'] = data['comment'].apply(preprocess)
```
接下来,我们可以使用sklearn库中的TfidfVectorizer对评论进行特征提取,并使用LogisticRegression进行分类。具体代码如下:
```python
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report
# 定义特征提取器
vectorizer = TfidfVectorizer()
# 提取特征
X = vectorizer.fit_transform(data['comment'])
# 定义标签
y = data['star'].apply(lambda x: 1 if x > 3 else 0)
# 划分训练集和测试集
train_size = int(len(data) * 0.8)
X_train, X_test, y_train, y_test = X[:train_size], X[train_size:], y[:train_size], y[train_size:]
# 定义分类器
clf = LogisticRegression()
# 训练分类器
clf.fit(X_train, y_train)
# 对测试集进行预测
y_pred = clf.predict(X_test)
# 输出分类报告
print(classification_report(y_test, y_pred))
```
除了使用sklearn库外,我们也可以自己编写模型类进行分类。这里我将使用PyTorch框架来编写模型类。具体代码如下:
```python
import torch
from torch import nn
from torch.utils.data import Dataset, DataLoader
from sklearn.metrics import classification_report
# 定义评论数据集类
class CommentDataset(Dataset):
def __init__(self, data, vectorizer):
self.data = data
self.vectorizer = vectorizer
def __len__(self):
return len(self.data)
def __getitem__(self, index):
comment = self.data.iloc[index]['comment']
star = self.data.iloc[index]['star']
features = self.vectorizer.transform([comment]).toarray().squeeze()
label = 1 if star > 3 else 0
return torch.Tensor(features), torch.Tensor([label])
# 定义分类模型类
class Classifier(nn.Module):
def __init__(self, input_size):
super(Classifier, self).__init__()
self.fc1 = nn.Linear(input_size, 128)
self.fc2 = nn.Linear(128, 64)
self.fc3 = nn.Linear(64, 2)
self.relu = nn.ReLU()
self.dropout = nn.Dropout(p=0.5)
def forward(self, x):
x = self.fc1(x)
x = self.relu(x)
x = self.dropout(x)
x = self.fc2(x)
x = self.relu(x)
x = self.dropout(x)
x = self.fc3(x)
return x
# 定义特征提取器
vectorizer = TfidfVectorizer()
# 定义评论数据集
dataset = CommentDataset(data, vectorizer)
# 划分训练集和测试集
train_size = int(len(data) * 0.8)
train_dataset, test_dataset = torch.utils.data.random_split(dataset, [train_size, len(dataset) - train_size])
train_loader = DataLoader(train_dataset, batch_size=32, shuffle=True)
test_loader = DataLoader(test_dataset, batch_size=32, shuffle=False)
# 初始化分类器
classifier = Classifier(input_size=train_dataset[0][0].shape[0])
# 定义优化器和损失函数
optimizer = torch.optim.Adam(classifier.parameters(), lr=0.001)
criterion = nn.CrossEntropyLoss()
# 训练分类器
for epoch in range(10):
for features, labels in train_loader:
optimizer.zero_grad()
outputs = classifier(features)
loss = criterion(outputs, labels.long())
loss.backward()
optimizer.step()
# 在测试集上进行评估
y_true = []
y_pred = []
with torch.no_grad():
for features, labels in test_loader:
outputs = classifier(features)
_, predicted = torch.max(outputs.data, 1)
y_true.extend(labels.numpy())
y_pred.extend(predicted.numpy())
print(classification_report(y_true, y_pred))
```
接下来,我们可以根据分类结果,分别画出正向词云图和负向词云图。具体代码如下:
```python
from wordcloud import WordCloud
import matplotlib.pyplot as plt
import numpy as np
from PIL import Image
# 定义正向评论和负向评论
positive_comments = data[data['star'] > 3]['comment']
negative_comments = data[data['star'] <= 3]['comment']
# 定义正向词云图
positive_text = " ".join(positive_comments)
mask = np.array(Image.open('positive.png'))
wc = WordCloud(background_color='white', mask=mask)
wc.generate(positive_text)
plt.imshow(wc, interpolation='bilinear')
plt.axis('off')
plt.show()
# 定义负向词云图
negative_text = " ".join(negative_comments)
mask = np.array(Image.open('negative.png'))
wc = WordCloud(background_color='white', mask=mask)
wc.generate(negative_text)
plt.imshow(wc, interpolation='bilinear')
plt.axis('off')
plt.show()
```
这里需要注意的是,我们需要准备好正向词云图和负向词云图的模板图片(例如positive.png和negative.png),用于生成相应形状的词云图。
阅读全文