[Advanced Chapter] Advanced Scrapy Practices: Custom Middleware and Pipelines

# 1. Introduction to Scrapy Framework and Custom Middleware

Scrapy is a robust web crawling framework designed for extracting data from websites. It offers extensive built-in features that enable developers to write efficient and scalable spiders with ease.

Scrapy middleware are plugins that allow developers to execute custom code at various points in the spider's lifecycle. They can be used for a variety of purposes, such as handling requests and responses, filtering data, or performing other tasks. Scrapy provides three types of middleware: Downloader Middleware, Spider Middleware, and Item Pipeline Middleware.

Downloader Middleware executes before a request is sent to the website and after a response is received. They can be used to modify request or response objects, handle redirections, or perform other request/response-related tasks.

# 2. Customizing Scrapy Middleware

### 2.1 Types and Functions of Middleware

Scrapy middleware are pluggable components that can be inserted into the Scrapy framework to perform custom actions during the request and response processing. Scrapy offers three types of middleware:

#### 2.1.1 Downloader Middleware

Downloader Middleware executes before a request is sent to the website and after a response is received by the Scrapy engine. They can be used for the following purposes:

- Modifying request headers and content
- Handling proxies and authentication
- Caching responses
- Filtering requests and responses

#### 2.1.2 Spider Middleware

Spider Middleware executes while the Scrapy spider is processing pages. They can be used for the following purposes:

- Handling page responses and extracting data
- Generating new requests
- Filtering page responses
- Monitoring the crawling process

#### 2.1.3 Item Pipeline Middleware

Item Pipeline Middleware executes at the Scrapy project level. They can be used for the following purposes:

- Configuring Scrapy settings
- Listening to Scrapy events
- Extending Scrapy functionality

### 2.2 Developing and Using Middleware

#### 2.2.1 Writing Middleware

Middleware is written using Python classes. Each middleware class must inherit from the `scrapy.middleware.Middleware` base class. Middleware classes must implement the following methods:

- `process_request(request, spider)`: Called before the request is sent to the website.
- `process_response(request, response, spider)`: Called after the response is returned to the Scrapy engine.
- `process_exception(request, exception, spider)`: Called when an exception occurs during request processing.

#### 2.2.2 Configuring and Activating Middleware

Middleware can be activated by configuring the `DOWNLOAD_MIDDLEWARES`, `SPIDER_MIDDLEWARES`, and `CLOSESPIDER_MIDDLEWARES` settings in the Scrapy project settings. These settings specify a list of middleware classes to use.

For example, to activate a downloader middleware, you can add the following line to the Scrapy project settings:

DOWNLOAD_MIDDLEWARES = {
     'myproject.middlewares.MyDownloadMiddleware': 500,
}

Here, `'myproject.middlewares.MyDownloadMiddleware'` is the full path to the middleware class, and `500` is the middleware's priority (a higher number indicates a higher priority).

**Code Block:**

class MyDownloadMiddleware(scrapy.middleware.DownloadMiddleware):

    def process_request(self, request, spider):
        # Modify request headers
        request.headers['User-Agent'] = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.45 Safari/537.36'