python爬取豆瓣书籍评论
时间: 2023-09-11 17:04:41 浏览: 59
可以使用Python的requests和BeautifulSoup库来爬取豆瓣书籍评论。下面是一个示例代码:
```python
import requests
from bs4 import BeautifulSoup
def get_book_comments(book_url):
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3'
}
# 发起GET请求获取页面内容
response = requests.get(book_url, headers=headers)
soup = BeautifulSoup(response.text, 'html.parser')
comments = []
# 找到所有评论所在的HTML标签
comment_tags = soup.find_all('div', class_='comment-item')
for tag in comment_tags:
# 获取评论内容
comment = tag.find('p', class_='comment-content').text.strip()
comments.append(comment)
return comments
# 调用函数并传入书籍页面的URL
book_url = 'https://book.douban.com/subject/1084336/comments/'
comments = get_book_comments(book_url)
for comment in comments:
print(comment)
```
请注意,上述代码中的headers参数是为了模拟浏览器访问而添加的,以防止被网站反爬虫机制拦截。你可以根据需要自行修改headers参数中的User-Agent值。同时,你需要将`book_url`替换为你想要爬取评论的具体豆瓣书籍页面的URL。