【Basics】Getting Started with the Scrapy Web Scraping Framework: Structure and Basic Usage

# 1. Introduction to Scrapy: Framework and Basic Usage

Scrapy is a powerful Python framework designed for web scraping. It offers a comprehensive set of tools and components that allow developers to build scrapers with ease and efficiency. Scrapy's features include:

- **Ease of Use:** Scrapy has an intuitive and user-friendly API, enabling developers to get started quickly.
- **Scalability:** Scrapy supports various customizations and extensions, allowing developers to tailor the scraper to specific needs.
- **Performance:** Scrapy employs an asynchronous concurrency mechanism, capable of handling a large number of concurrent requests and improving scraping efficiency.

# 2. Scrapy Project Structure and Basic Usage

**2.1 Project Structure and Components**

A Scrapy project is a directory structure containing code, configuration files, and data. The fundamental components include:

- `scrapy.cfg`: The Scrapy configuration file used for setting up the scraper.
- `settings.py`: Project-specific settings file that overrides default settings in `scrapy.cfg`.
- `spiders`: The directory for spider code, containing all scraper classes.
- `pipelines`: The data pipeline directory used for processing and storing extracted data.
- `items.py`: The file where Items used in the project are defined.
- `middlewares.py`: The middleware directory for executing custom logic during the spider's request and response processing.

**2.2 Creating and Configuring a Spider**

To create a spider, create a Python file in the `spiders` directory and inherit from the `scrapy.Spider` class. The spider class must define the following methods:

- `name`: The unique name of the spider.
- `start_requests`: A generator for creating initial requests.
- `parse`: The callback function for parsing responses and extracting data.

import scrapy

class MySpider(scrapy.Spider):
     name = "my_spider"

     def start_requests(self):
         yield scrapy.Request("***")

     def parse(self, response):
         # Parse response and extract data
         pass

**2.3 Basic Process of Scraping a Web Page**

The basic process of scraping a web page with Scrapy is as follows:

1. The spider sends an HTTP request.
2. The Scrapy engine receives the request and sends it to the downloader middleware.
3. The downloader middleware processes the request and sends it to the downloader.
4. The downloader fetches the response and sends it to the response middleware.
5. The response middleware processes the response and sends it to the spider.
6. The spider parses the response and extracts data.
7. The extracted data is processed and stored through the pipeline.

**Mermaid Flowchart:**

sequenceDiagram
participant Scrapy
participant Engine
participant DownloaderMiddleware
participant Downloader
participant ResponseMiddleware
participant Spider
participant Pipeline

Scrapy->Engine: Send Request
Engine->DownloaderMiddleware: Process Request
DownloaderMiddleware->Downloader: Send Request
Downloader->DownloaderMiddleware: Process Response
DownloaderMiddleware->Engine: Send Response
Engine->ResponseMiddleware: Process Response
ResponseMiddleware->Spider: Parse Response
Spider->Pipeline: Process Data
Pipeline->Engine: Store Data

**Code Block:**

from scrapy.spiders import Spider
from scrapy.http import Request

class MySpider(Spider):
     name = "my_spider"

     def start_requests(self):
         yield Request("***", callback=self.parse)

     def parse(self, response):
         # Parse response and extract data
         pass

**Logical Analysis:**

- The `start_requests` method generates an HTTP request and specifies the `parse` method as the callback function.
- The `parse` method parses the response and extracts data.

# 3. Scrapy Crawler Practical Application

### 3.1 Web Page Parsing and Data Extraction

**Web Page Parsing**

Web page parsing is a key step in Scrapy crawlers, aiming to extract required data from web pages formatted in HTML or XML. Scrapy provides multiple parsers such as:

- `lxml`: Based on the lxml library, supports XPath and CSS selectors.
- `html.parser`: Built-in HTML parser, supports XPath and CSS selectors.
- `cssselect`: Based on the cssselect library, supports only CSS selectors.

**Data Extraction**

Data extraction follows the parsing process, aiming to extract required data from the parsed document. Scrapy offers various data extraction methods:

- `XPath`: A query language for XML, used to extract data from HTML or XML documents.
- `CSS Selectors`: A query language based on CSS, used to extract data from HTML documents.
- `Regular Expressions`: A powerful tool for matching text patterns, used to extract data from web pages.

**Code Example:**

# Extracting title using XPath
title = response.xpath('//title/text()').extract_first()

# Extracting article content using CSS selectors
content = response.css('article p::text').extract()

# Extracting email addresses using regular expressions
emails = re.findall(r'[\w\.-]+@[\w\.-]+', response.text)

### 3.2 Data Storage and Persistence

**Data Storage**

Scrapy provides various data storage options:

- `Files`: Store data in files such as CSV, JSON, or XML.
- `Databases`: Store data in relational databases such as MySQL, PostgreSQL, or Mon