帮我用python写一份爬取京东手机数据并保存至MySQL 数据库,使用pyechart 进行可视化的代码
时间: 2024-03-31 10:36:38 浏览: 98
好的,以下是代码:
```python
import requests
import pymysql
from pyecharts.charts import Bar
from pyecharts import options as opts
# 爬取京东手机数据
url = "https://search.jd.com/Search?keyword=%E6%89%8B%E6%9C%BA&enc=utf-8"
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3"}
response = requests.get(url, headers=headers)
html = response.text
# 解析数据
goods_list = []
for line in html.splitlines():
if "<li class='gl-item'>" in line:
goods = {}
sku = line.split("data-sku=")[1].split(">")[0].replace('"', '')
price_url = "https://p.3.cn/prices/mgets?skuIds=J_" + sku
price_response = requests.get(price_url, headers=headers)
price_json = price_response.json()
price = price_json[0]['p']
name = line.split("<div class='p-name p-name-type-2'>")[1].split("</div>")[0]
shop = line.split("<div class='p-shop'>")[1].split("</div>")[0]
comment_url = "https://sclub.jd.com/comment/productPageComments.action?productId=" + sku + "&score=0&sortType=3&page=0&pageSize=10"
comment_response = requests.get(comment_url, headers=headers)
comment_json = comment_response.json()
comment_count = comment_json['productCommentSummary']['commentCount']
goods['sku'] = sku
goods['price'] = price
goods['name'] = name
goods['shop'] = shop
goods['comment_count'] = comment_count
goods_list.append(goods)
# 保存至MySQL数据库
db = pymysql.connect("localhost", "root", "password", "test")
cursor = db.cursor()
cursor.execute("DROP TABLE IF EXISTS jd_phone")
sql = """CREATE TABLE jd_phone (
SKU CHAR(20) NOT NULL,
PRICE FLOAT,
NAME VARCHAR(100),
SHOP VARCHAR(100),
COMMENT_COUNT INT)"""
cursor.execute(sql)
for goods in goods_list:
sku = goods['sku']
price = goods['price']
name = goods['name']
shop = goods['shop']
comment_count = goods['comment_count']
sql = "INSERT INTO jd_phone(SKU,PRICE,NAME,SHOP,COMMENT_COUNT) VALUES (" + "'" + sku + "'," + str(
price) + ",'" + name + "','" + shop + "'," + str(comment_count) + ")"
cursor.execute(sql)
db.commit()
# 使用pyechart进行可视化
cursor.execute("SELECT NAME,PRICE FROM jd_phone ORDER BY PRICE DESC LIMIT 10")
result = cursor.fetchall()
bar = Bar()
x_data = [i[0] for i in result]
y_data = [i[1] for i in result]
bar.add_xaxis(x_data)
bar.add_yaxis("价格", y_data)
bar.set_global_opts(title_opts=opts.TitleOpts(title="京东手机价格排行TOP10"))
bar.render("jd_phone_price.html")
```
需要注意的是,此代码需要依赖以下库:
- requests
- pymysql
- pyecharts
需要预先安装,可以通过pip命令进行安装。同时,需要将代码中的"localhost"、"root"、"password"、"test"修改为自己的MySQL数据库地址、用户名、密码和数据库名。
阅读全文