python爬取京东手机数据并存入数据库
时间: 2023-09-03 16:07:42 浏览: 190
基于Python爬取京东双十一商品价格曲线
好的,以下是Python爬取京东手机数据并存入MySQL数据库的示例代码:
```python
import requests
from lxml import etree
import pymysql
# 爬取页面数据
def crawl_jd():
url = 'https://list.jd.com/list.html?cat=9987,653,655&page=1'
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3'}
response = requests.get(url, headers=headers)
html = etree.HTML(response.text)
items = html.xpath('//li[@class="gl-item"]')
data = []
for item in items:
name = item.xpath('.//div[@class="p-name"]/a/em/text()')[0]
price = item.xpath('.//div[@class="p-price"]/strong/i/text()')[0]
data.append((name, price))
return data
# 存入数据库
def save_to_mysql(data):
db = pymysql.connect(host='localhost', user='root', password='123456', db='test', charset='utf8')
cursor = db.cursor()
sql = "INSERT INTO jd_mobiles(name, price) VALUES (%s, %s)"
try:
cursor.executemany(sql, data)
db.commit()
except:
db.rollback()
db.close()
if __name__ == '__main__':
data = crawl_jd()
save_to_mysql(data)
```
其中,`crawl_jd()`函数用于爬取京东手机页面数据,`save_to_mysql()`函数用于将数据存入MySQL数据库中。你可以根据自己的需求进行相应的修改。
阅读全文