【Advanced Chapter】Advanced Web Crawler Practices: Crawling Dynamic Webpage Data: Real-time Data Crawling Using Websocket

## [Advanced篇] Advanced Web Crawler Practices: Websocket for Real-Time Data Scraping

# 2.1 Concept and Principle of Websocket

Websocket is a two-way communication protocol that allows for real-time, full-duplex communication between a client and a server after a single TCP connection is established. Unlike the HTTP protocol, the Websocket connection remains open after establishment, allowing clients and servers to exchange data at any time.

The principle of Websocket is based on the WebSocket Frame. A WebSocket Frame is a binary frame that contains the following information:

- **Opcode:** Indicates the frame's type, such as text frame, binary frame, or close frame.
- **Payload:** The frame's payload, which is the actual data being transmitted.
- **FIN:** Indicates whether the frame is the last one.

Clients and servers exchange WebSocket Frames to achieve real-time communication. When a client sends a WebSocket Frame, the server receives and processes it immediately. Similarly, when the server sends a WebSocket Frame, the client receives and processes it immediately. This method of real-time communication makes Websocket ideal for applications that require fast, bidirectional data transfer.

# 2. Introduction to Websocket Technology

### 2.1 Concept and Principle of Websocket

Websocket is a two-way communication technology based on the TCP protocol. It allows for full-duplex communication between a client and a server after a single connection is established. Unlike the HTTP request-response model, once a Websocket connection is established, both clients and servers can send and receive messages at any time, achieving real-time communication.

The process of establishing a Websocket connection is as follows:

1. **Handshake Phase:** The client sends an HTTP request to the server, requesting an upgrade to the Websocket protocol.
2. **Negotiation Phase:** The server responds to the client's request, negotiating the Websocket protocol version, extensions, and subprotocols.
3. **Establishing Connection:** After successful negotiation, the client and server establish a full-duplex communication channel.

### 2.2 Advantages and Application Scenarios of Websocket

Websocket technology has the following advantages:

- **Real-time Communication:** Allows clients and servers to send and receive messages at any time, achieving real-time communication.
- **Low Latency:** After the Websocket connection is established, message transmission delay is very low.
- **Bidirectional Communication:** Both clients and servers can actively send and receive messages.
- **Bandwidth Savings:** Websocket uses binary frames to transmit data, which is more bandwidth-efficient than the HTTP request-response model.

Websocket technology is widely used in scenarios that require real-time communication, such as:

- **Chat Applications:** To enable real-time message delivery between users.
- **Games:** To enable real-time interaction among players.
- **Financial Trading:** To push real-time stock information and transaction data.
- **Internet of Things:** To achieve real-time data transmission between devices and servers.

# 3. Real-Time Data Scraping Using Websocket

### 3.1 Establishing and Maintaining Websocket Connections

#### Establishing Websocket Connections

Establishing a Websocket connection requires the collaboration of both the client and the server. The client first sends an HTTP request containing the header field `Upgrade: websocket`, indicating that the client wishes to upgrade to the Websocket protocol. Upon receiving the request, if the server supports Websocket, it will respond with an HTTP 101 Switching Protocols message containing the header field `Upgrade: websocket`, indicating that the server agrees to upgrade