帮我用python语言写一段爬虫代码
时间: 2023-03-06 22:45:37 浏览: 120
用Python语言编写一个爬虫程序
爬虫代码可以用Python语言实现,示例代码如下:import requests
from bs4 import BeautifulSoupdef get_html(url):
headers = {
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.122 Safari/537.36'
}
response = requests.get(url, headers=headers, timeout=10)
response.encoding = response.apparent_encoding
if response.status_code == 200:
return response.text
return Nonedef get_content(html):
soup = BeautifulSoup(html, 'html.parser')
items = soup.find_all('div', attrs={'class': 'item'})
content_list = []
for item in items:
content_list.append(item.find('div', attrs={'class': 'info'}).find('h2').get_text())
return content_listdef main():
url = 'https://www.example.com/'
html = get_html(url)
content = get_content(html)
for c in content:
print(c)if __name__ == '__main__':
main()
阅读全文