scrapy连接mysql
时间: 2023-05-11 10:05:31 浏览: 101
可以使用以下代码连接Scrapy和MySQL:
1. 首先,需要在Scrapy项目的settings.py文件中添加以下代码:
```
ITEM_PIPELINES = {
'myproject.pipelines.MySQLPipeline': 300,
}
MYSQL_HOST = 'localhost'
MYSQL_DBNAME = 'mydatabase'
MYSQL_USER = 'myusername'
MYSQL_PASSWORD = 'mypassword'
```
2. 然后,在Scrapy项目的pipelines.py文件中添加以下代码:
```
import pymysql
class MySQLPipeline(object):
def __init__(self, host, dbname, user, password):
self.host = host
self.dbname = dbname
self.user = user
self.password = password
@classmethod
def from_crawler(cls, crawler):
return cls(
host=crawler.settings.get('MYSQL_HOST'),
dbname=crawler.settings.get('MYSQL_DBNAME'),
user=crawler.settings.get('MYSQL_USER'),
password=crawler.settings.get('MYSQL_PASSWORD')
)
def open_spider(self, spider):
self.conn = pymysql.connect(
host=self.host,
user=self.user,
password=self.password,
db=self.dbname,
charset='utf8mb4',
cursorclass=pymysql.cursors.DictCursor
)
def close_spider(self, spider):
self.conn.close()
def process_item(self, item, spider):
with self.conn.cursor() as cursor:
sql = "INSERT INTO mytable (column1, column2, column3) VALUES (%s, %s, %s)"
cursor.execute(sql, (item['column1'], item['column2'], item['column3']))
self.conn.commit()
return item
```
3. 最后,在Scrapy项目的items.py文件中定义你的item:
```
import scrapy
class MyItem(scrapy.Item):
column1 = scrapy.Field()
column2 = scrapy.Field()
column3 = scrapy.Field()
```
这样,当你运行Scrapy爬虫时,它会将数据存储到MySQL数据库中。