爬虫scrapy实现将数据导入Mariadb数据库

要将Scrapy爬取到的数据导入到Mariadb数据库中，需要进行以下几个步骤：

1. 安装Mariadb数据库和Python的Mariadb驱动程序pymysql

    pip install pymysql

2. 在项目的settings.py文件中添加Mariadb数据库的配置信息

    MYSQL_HOST = 'localhost'
    MYSQL_PORT = 3306
    MYSQL_USER = 'root'
    MYSQL_PASSWORD = 'password'
    MYSQL_DBNAME = 'database_name'

3. 在Scrapy项目的pipelines.py文件中编写将数据存储到Mariadb数据库的代码

    import pymysql

    class MariadbPipeline(object):
        def __init__(self, host, port, user, password, db):
            self.host = host
            self.port = port
            self.user = user
            self.password = password
            self.db = db

        @classmethod
        def from_crawler(cls, crawler):
            return cls(
                host=crawler.settings.get('MYSQL_HOST'),
                port=crawler.settings.get('MYSQL_PORT'),
                user=crawler.settings.get('MYSQL_USER'),
                password=crawler.settings.get('MYSQL_PASSWORD'),
                db=crawler.settings.get('MYSQL_DBNAME')
            )

        def open_spider(self, spider):
            self.conn = pymysql.connect(
                host=self.host,
                port=self.port,
                user=self.user,
                password=self.password,
                db=self.db,
                charset='utf8mb4',
                cursorclass=pymysql.cursors.DictCursor
            )
            self.cursor = self.conn.cursor()

        def close_spider(self, spider):
            self.conn.close()

        def process_item(self, item, spider):
            sql = """
                INSERT INTO table_name (field1, field2, field3) VALUES (%s, %s, %s)
            """
            self.cursor.execute(sql, (item['field1'], item['field2'], item['field3']))
            self.conn.commit()
            return item

4. 在Scrapy项目的settings.py文件中启用MariadbPipeline

    ITEM_PIPELINES = {
        'project_name.pipelines.MariadbPipeline': 300,
    }

这样，当Scrapy爬取到数据后，就会将数据存储到Mariadb数据库中。