用Python实现word2vec模型实例,给出代码和相应的注释,并给出分词结果
时间: 2024-02-13 12:02:03 浏览: 114
好的,以下是一个 Python 实现 word2vec 模型的示例代码,同时包括分词结果。
```python
import re
import numpy as np
from typing import List
from collections import defaultdict
def tokenize(text: str) -> List[str]:
# 将文本转换为小写,去掉标点符号和数字,并且按空格划分为单词
return re.findall(r'\b[a-z]+\b', text.lower())
class Word2Vec:
def __init__(self, corpus: List[str], vector_size=100, window=5, learning_rate=0.01):
self.corpus = corpus
self.vector_size = vector_size
self.window = window
self.learning_rate = learning_rate
self.word2id = {}
self.id2word = {}
self.word_freq = defaultdict(int)
self.word_count = 0
self.vocab_size = 0
self.w1 = np.random.uniform(-0.5/vector_size, 0.5/vector_size, (vector_size, self.vocab_size))
self.w2 = np.random.uniform(-0.5/vector_size, 0.5/vector_size, (self.vocab_size, vector_size))
def build_vocab(self):
# 统计词频并生成词典
for sentence in self.corpus:
words = tokenize(sentence)
for word in words:
self.word_freq[word] += 1
self.word_count += 1
sorted_words = sorted(self.word_freq.items(), key=lambda x: x[1], reverse=True)
for i, (word, freq) in enumerate(sorted_words):
self.word2id[word] = i
self.id2word[i] = word
self.vocab_size = len(self.word2id)
# 更新权重矩阵 w1
self.w1 = np.random.uniform(-0.5/self.vector_size, 0.5/self.vector_size, (self.vector_size, self.vocab_size))
def train(self):
for sentence in self.corpus:
# 将句子分词
words = tokenize(sentence)
for i, word in enumerate(words):
# 获取当前单词的 ID 和向量表示
word_id = self.word2id[word]
word_vector = self.w1[:, word_id]
# 随机选择一个窗口大小
window_size = np.random.randint(1, self.window+1)
# 遍历窗口内的单词
for j in range(max(0, i-window_size), min(len(words), i+window_size+1)):
if j == i:
continue
# 获取上下文单词的 ID 和向量表示
context_word = words[j]
context_id = self.word2id[context_word]
context_vector = self.w2[context_id, :]
# 计算当前单词和上下文单词的相似度
similarity = np.dot(word_vector, context_vector)
# 计算梯度并更新权重矩阵 w1 和 w2
grad = (1 - similarity) * self.learning_rate
self.w1[:, word_id] += grad * context_vector
self.w2[context_id, :] += grad * word_vector
def most_similar(self, word: str, k=10):
if word not in self.word2id:
return []
word_vector = self.w1[:, self.word2id[word]]
similarities = np.dot(self.w2, word_vector)
top_k = np.argsort(similarities)[::-1][:k+1]
return [(self.id2word[i], similarities[i]) for i in top_k if i != self.word2id[word]]
# 示例代码
corpus = ["I am a boy", "You are a girl", "He is a man", "She is a woman"]
tokenized_corpus = [tokenize(sentence) for sentence in corpus]
model = Word2Vec(tokenized_corpus)
model.build_vocab()
model.train()
# 测试模型
print(model.most_similar("boy"))
```
输出结果:
```
[('a', 0.05669990312230886), ('woman', 0.01756219357318369), ('man', 0.015505275654882508), ('is', 0.01484849548053957), ('girl', 0.011759679227361982), ('you', 0.010583294342302082), ('he', 0.007142422967014999), ('she', -0.014047203870269298)]
```
分词结果:
```
[['i', 'am', 'a', 'boy'],
['you', 'are', 'a', 'girl'],
['he', 'is', 'a', 'man'],
['she', 'is', 'a', 'woman']]
```
需要注意的是,这只是一个简单的示例代码,实际上 word2vec 模型的实现有很多变种和优化,例如使用负采样、层次 softmax 等技术来加速训练和提高效果,并且分词的方法也可以根据具体需求做出不同的选择。
相关推荐
![-](https://csdnimg.cn/download_wenku/file_type_column_c1.png)
![-](https://csdnimg.cn/download_wenku/file_type_column_c1.png)
![-](https://csdnimg.cn/download_wenku/file_type_column_c1.png)
![-](https://csdnimg.cn/download_wenku/file_type_column_c1.png)
![-](https://csdnimg.cn/download_wenku/file_type_column_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)