写python代码爬取知乎关于人生话题下的100对问答
时间: 2023-09-20 11:10:08 浏览: 58
以下是一个示例的Python代码,可以使用BeautifulSoup和requests库来爬取知乎关于人生话题下的100对问答:
```python
import requests
from bs4 import BeautifulSoup
def crawl_zhihu():
url = "https://www.zhihu.com/topic/20024670/hot"
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.182 Safari/537.36"
}
response = requests.get(url, headers=headers)
soup = BeautifulSoup(response.content, "html.parser")
question_links = soup.find_all("a", class_="question_link")
questions = []
for link in question_links:
question = link.get_text(strip=True)
questions.append(question)
return questions[:100]
if __name__ == "__main__":
data = crawl_zhihu()
for i, question in enumerate(data, start=1):
print(f"{i}. {question}")
```
请注意,这只是一个简单的示例代码,实际使用时可能需要处理反爬虫机制、翻页等问题。此外,注意尊重网站的使用规则和版权,遵守合适的爬取频率和行为准则。