基于Python,对数据集neg.txt进行划分,分出训练集和测试集并利用朴素贝叶斯算法对测试集进行情感倾向预测,并计算预测准确率
时间: 2024-09-26 09:14:14 浏览: 50
在Python中,我们可以使用`sklearn`库来进行文本分类任务,包括数据预处理、划分训练集和测试集以及应用朴素贝叶斯算法。这里是一个简单的步骤概述:
1. **安装必要的库**:
首先确保已安装了`pandas`, `numpy`, `sklearn`, 和 `nltk`(自然语言工具包),如果没有,可以使用以下命令安装:
```
!pip install pandas numpy sklearn nltk
```
2. **加载和预处理数据**:
使用`pandas`读取文件:
```python
import pandas as pd
data = pd.read_csv('neg.txt', sep="\t", names=['text', 'label'])
# 如果数据包含非ASCII字符,可能需要进行编码转换
data['text'] = data['text'].str.decode('utf-8')
```
3. **数据清洗和分词**:
```python
from nltk.corpus import stopwords
from nltk.tokenize import word_tokenize
nlp = lambda x: [word for word in word_tokenize(x) if word.isalnum() and word not in stopwords.words('english')]
data['processed_text'] = data['text'].apply(nlp)
```
4. **划分数据集**:
```python
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(data['processed_text'], data['label'], test_size=0.2, random_state=42)
```
5. **创建特征向量**:
对文本数据通常使用词袋模型或TF-IDF(Term Frequency-Inverse Document Frequency):
```python
from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer()
X_train_vec = vectorizer.fit_transform(X_train)
X_test_vec = vectorizer.transform(X_test)
```
6. **朴素贝叶斯模型**:
```python
from sklearn.naive_bayes import MultinomialNB
classifier = MultinomialNB()
classifier.fit(X_train_vec, y_train)
```
7. **预测和评估**:
```python
y_pred = classifier.predict(X_test_vec)
from sklearn.metrics import accuracy_score
accuracy = accuracy_score(y_test, y_pred)
print(f"预测准确率为: {accuracy * 100}%")
```
阅读全文