Python爬虫爬取微博用户评论代码
时间: 2024-01-02 22:14:00 浏览: 41
以下是使用Python爬虫爬取微博用户评论的代码,需要先安装requests、beautifulsoup4和lxml库:
```python
import requests
from bs4 import BeautifulSoup
# 获取指定微博的评论页数
def get_comment_pages(weibo_url):
response = requests.get(weibo_url)
soup = BeautifulSoup(response.text, 'lxml')
pages = soup.select('.card-page > ul > li')[-1].get('page')
return int(pages)
# 获取指定微博的所有评论
def get_comments(weibo_url):
pages = get_comment_pages(weibo_url)
comments = []
for page in range(1, pages+1):
url = weibo_url + f'?page={page}'
response = requests.get(url)
soup = BeautifulSoup(response.text, 'lxml')
comment_list = soup.select('.list_li')
for comment in comment_list:
text = comment.select_one('.WB_text').text.strip()
comments.append(text)
return comments
# 测试代码
weibo_url = 'https://weibo.com/1234567890123456/ABCDE6789?type=comment'
comments = get_comments(weibo_url)
for comment in comments:
print(comment)
```
需要替换weibo_url为指定微博的URL。该代码会输出指定微博的所有评论。需要注意的是,微博的评论需要登录后才能查看,所以需要先登录微博才能爬取评论。同时,该代码只能爬取普通用户的评论,无法爬取大V或官方账号的评论。