python中的jieba实现对一个文件夹中的所有文档进行分词,根据停用词表去除停用词,基于文档中的非停用词及其TF-IDF值,生成该文档的词向量,具体代码...
时间: 2024-03-01 07:54:16 浏览: 93
好的,以下是实现的具体代码:
```python
import os
import jieba
import math
# 加载停用词表
stopwords = []
with open('stopwords.txt', 'r', encoding='utf-8') as f:
for line in f.readlines():
stopwords.append(line.strip())
# 计算TF-IDF值
def calculate_tf_idf(word, doc_freq, total_docs):
tf = doc_freq / total_docs
idf = math.log(total_docs / (doc_freq + 1))
return tf * idf
# 对一个文档进行分词
def segment_doc(doc_path):
with open(doc_path, 'r', encoding='utf-8') as f:
content = f.read()
words = jieba.cut(content)
words = [word for word in words if word not in stopwords]
return words
# 对一个文档生成词向量
def generate_word_vector(doc_path, words_dict, total_docs):
words = segment_doc(doc_path)
word_vector = {}
for word in words:
if word not in words_dict:
continue
if word not in word_vector:
word_vector[word] = 0
word_vector[word] += 1
for word, freq in word_vector.items():
doc_freq = words_dict[word]
tf_idf_value = calculate_tf_idf(word, doc_freq, total_docs)
word_vector[word] = tf_idf_value
return word_vector
# 获取所有文档分词后的词频
def get_words_dict(docs_dir):
words_dict = {}
total_docs = 0
for filename in os.listdir(docs_dir):
doc_path = os.path.join(docs_dir, filename)
words = segment_doc(doc_path)
for word in words:
if word not in words_dict:
words_dict[word] = 0
words_dict[word] += 1
total_docs += 1
return words_dict, total_docs
# 对整个文档集生成词向量矩阵
def generate_word_vectors_matrix(docs_dir):
words_dict, total_docs = get_words_dict(docs_dir)
word_vectors_matrix = []
for filename in os.listdir(docs_dir):
doc_path = os.path.join(docs_dir, filename)
word_vector = generate_word_vector(doc_path, words_dict, total_docs)
word_vectors_matrix.append(word_vector)
return word_vectors_matrix
```
其中,`docs_dir`是文件夹路径,表示需要对该文件夹中的所有文档进行分词和词向量的生成。停用词表文件为`stopwords.txt`,需要提前准备好。生成的词向量矩阵是一个二维列表,每一行表示一个文档的词向量。
阅读全文
相关推荐
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![pdf](https://img-home.csdnimg.cn/images/20241231044930.png)
![-](https://img-home.csdnimg.cn/images/20241231044955.png)
![-](https://img-home.csdnimg.cn/images/20241231045021.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![zip](https://img-home.csdnimg.cn/images/20241231045053.png)
![zip](https://img-home.csdnimg.cn/images/20241231045053.png)