Scrapy框架开发指南：从入门到精通

需积分: 0 99 浏览量更新于2024-06-14 收藏 1.76MB PDF 举报

Scrapy文档概述 Scrapy是一个流行的Python爬虫框架，用于从Web页面中提取数据。下面是Scrapy文档的概述，涵盖了Scrapy的基本概念、安装指南、蜘蛛开发、项目结构、数据处理、错误处理、开发者工具等方面。 **基本概念** Scrapy是一个灵活的框架，提供了多种方式来爬取和处理数据。Scrapy的基本概念包括： * 命令行工具：Scrapy提供了一个命令行工具，用于运行蜘蛛、查看日志、调试蜘蛛等。 * 蜘蛛（Spiders）：蜘蛛是Scrapy的核心组件，负责爬取和处理数据。 * 选择器（Selectors）：选择器是Scrapy中用于提取数据的主要工具。 * 项（Items）：项是Scrapy中用于存储数据的基本单位。 * 项加载器（Item Loaders）：项加载器是Scrapy中用于加载和处理项的工具。 * Scrapy Shell：Scrapy Shell是一个交互式的命令行工具，用于调试和测试蜘蛛。 **安装指南** Scrapy的安装非常简单，用户可以使用pip安装Scrapy。Scrapy支持多种操作系统，包括Windows、Linux和Mac OS X。 **蜘蛛开发** Scrapy提供了多种方式来开发蜘蛛，包括： * 使用Scrapy的内置蜘蛛模板 * 使用Scrapy的命令行工具生成蜘蛛 * 使用Scrapy的API开发蜘蛛 **项目结构** Scrapy项目的结构包括： * 项目目录：项目目录是Scrapy项目的根目录。 * 蜘蛛目录：蜘蛛目录是Scrapy项目中的蜘蛛存储目录。 * 项目录：项目录是Scrapy项目中的项存储目录。 **数据处理** Scrapy提供了多种方式来处理数据，包括： * 使用选择器提取数据 * 使用项加载器加载和处理数据 * 使用Scrapy的内置数据处理工具 **错误处理** Scrapy提供了多种方式来处理错误，包括： * 使用Scrapy的内置错误处理工具 * 使用try-except语句捕获错误 * 使用日志工具记录错误 **开发者工具** Scrapy提供了多种开发者工具，包括： * Scrapy Shell：一个交互式的命令行工具，用于调试和测试蜘蛛。 * Logging：一个日志工具，用于记录蜘蛛的执行过程。 * Stats Collection：一个统计工具，用于收集蜘蛛的执行统计数据。 **解决问题** Scrapy提供了多种方式来解决问题，包括： * 使用FAQ解决常见问题 * 使用调试工具调试蜘蛛 * 使用蜘蛛 Contract解决蜘蛛之间的冲突 **高级主题** Scrapy还提供了多种高级主题，包括： * 使用Broad Crawls爬取大量数据 * 使用AutoThrottle extension控制爬取速度 * 使用Benchmarking工具测试蜘蛛的性能 Scrapy是一个功能强大且灵活的爬虫框架，提供了多种方式来爬取和处理数据，满足了不同用户的需求。

Scrapy Documentation, Release 2.11.1

(continued from previous page)

pipelines.py # project pipelines file

settings.py # project settings file

spiders/ # a directory where you'll later put your spiders

__init__.py

2.3.2 Our ﬁrst Spider

Spiders are classes that you deﬁne and that Scrapy uses to scrape information from a website (or a group of websites).

They must subclass Spider and deﬁne the initial requests to make, optionally how to follow links in the pages, and

how to parse the downloaded page content to extract data.

This is the code for our ﬁrst Spider. Save it in a ﬁle named quotes_spider.py under the tutorial/spiders

directory in your project:

from pathlib import Path

import scrapy

class QuotesSpider(scrapy.Spider):

name = "quotes"

def start_requests(self):

urls = [

"https://quotes.toscrape.com/page/1/",

"https://quotes.toscrape.com/page/2/",

]

for url in urls:

yield scrapy.Request(url=url, callback=self.parse)

def parse(self, response):

page = response.url.split("/")[-2]

filename = f"quotes-{page}.html"

Path(filename).write_bytes(response.body)

self.log(f"Saved file {filename}")

As you can see, our Spider subclasses scrapy.Spider and deﬁnes some attributes and methods:

• name: identiﬁes the Spider. It must be unique within a project, that is, you can’t set the same name for diﬀerent

Spiders.

• start_requests(): must return an iterable of Requests (you can return a list of requests or write a generator

function) which the Spider will begin to crawl from. Subsequent requests will be generated successively from

these initial requests.

• parse(): a method that will be called to handle the response downloaded for each of the requests made. The

response parameter is an instance of TextResponse that holds the page content and has further helpful methods

to handle it.

The parse() method usually parses the response, extracting the scraped data as dicts and also ﬁnding new URLs

to follow and creating new requests (Request) from them.

12 Chapter 2. First steps

Scrapy Documentation, Release 2.11.1

How to run our spider

To put our spider to work, go to the project’s top level directory and run:

scrapy crawl quotes

This command runs the spider with name quotes that we’ve just added, that will send some requests for the quotes.

toscrape.com domain. You will get an output similar to this:

... (omitted for brevity)

2016-12-16 21:24:05 [scrapy.core.engine] INFO: Spider opened

2016-12-16 21:24:05 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min),␣

˓→scraped 0 items (at 0 items/min)

2016-12-16 21:24:05 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.

˓→0.1:6023

2016-12-16 21:24:05 [scrapy.core.engine] DEBUG: Crawled (404) <GET https://quotes.

˓→toscrape.com/robots.txt> (referer: None)

2016-12-16 21:24:05 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://quotes.

˓→toscrape.com/page/1/> (referer: None)

2016-12-16 21:24:05 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://quotes.

˓→toscrape.com/page/2/> (referer: None)

2016-12-16 21:24:05 [quotes] DEBUG: Saved file quotes-1.html

2016-12-16 21:24:05 [quotes] DEBUG: Saved file quotes-2.html

2016-12-16 21:24:05 [scrapy.core.engine] INFO: Closing spider (finished)

...

Now, check the ﬁles in the current directory. You should notice that two new ﬁles have been created: quotes-1.html

and quotes-2.html, with the content for the respective URLs, as our parse method instructs.

Note: If you are wondering why we haven’t parsed the HTML yet, hold on, we will cover that soon.

What just happened under the hood?

Scrapy schedules the scrapy.Request objects returned by the start_requests method of the Spider. Upon receiv-

ing a response for each one, it instantiates Response objects and calls the callback method associated with the request

(in this case, the parse method) passing the response as argument.

A shortcut to the start_requests method

Instead of implementing a start_requests() method that generates scrapy.Request objects from URLs, you can

just deﬁne a start_urls class attribute with a list of URLs. This list will then be used by the default implementation

of start_requests() to create the initial requests for your spider.

from pathlib import Path

import scrapy

class QuotesSpider(scrapy.Spider):

name = "quotes"

start_urls = [

(continues on next page)

2.3. Scrapy Tutorial 13

Scrapy Documentation, Release 2.11.1

(continued from previous page)

"https://quotes.toscrape.com/page/1/",

"https://quotes.toscrape.com/page/2/",

]

def parse(self, response):

page = response.url.split("/")[-2]

filename = f"quotes-{page}.html"

Path(filename).write_bytes(response.body)

The parse() method will be called to handle each of the requests for those URLs, even though we haven’t explicitly

told Scrapy to do so. This happens because parse() is Scrapy’s default callback method, which is called for requests

without an explicitly assigned callback.

Extracting data

The best way to learn how to extract data with Scrapy is trying selectors using the Scrapy shell. Run:

scrapy shell 'https://quotes.toscrape.com/page/1/'

Note: Remember to always enclose urls in quotes when running Scrapy shell from command-line, otherwise urls

containing arguments (i.e. & character) will not work.

On Windows, use double quotes instead:

scrapy shell "https://quotes.toscrape.com/page/1/"

You will see something like:

[ ... Scrapy log here ... ]

2016-09-19 12:09:27 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://quotes.

˓→toscrape.com/page/1/> (referer: None)

[s] Available Scrapy objects:

[s] scrapy scrapy module (contains scrapy.Request, scrapy.Selector, etc)

[s] crawler <scrapy.crawler.Crawler object at 0x7fa91d888c90>

[s] item {}

[s] request <GET https://quotes.toscrape.com/page/1/>

[s] response <200 https://quotes.toscrape.com/page/1/>

[s] settings <scrapy.settings.Settings object at 0x7fa91d888c10>

[s] spider <DefaultSpider 'default' at 0x7fa91c8af990>

[s] Useful shortcuts:

[s] shelp() Shell help (print this help)

[s] fetch(req_or_url) Fetch request (or URL) and update local objects

[s] view(response) View response in a browser

Using the shell, you can try selecting elements using CSS with the response object:

>>> response.css("title")

[<Selector query='descendant-or-self::title' data='<title>Quotes to Scrape</title>'>]

The result of running response.css('title') is a list-like object called SelectorList, which represents a list

of Selector objects that wrap around XML/HTML elements and allow you to run further queries to ﬁne-grain the

selection or extract the data.

14 Chapter 2. First steps

Scrapy Documentation, Release 2.11.1

To extract the text from the title above, you can do:

>>> response.css("title::text").getall()

['Quotes to Scrape']

There are two things to note here: one is that we’ve added ::text to the CSS query, to mean we want to select only the

text elements directly inside <title> element. If we don’t specify ::text, we’d get the full title element, including

its tags:

>>> response.css("title").getall()

['<title>Quotes to Scrape</title>']

The other thing is that the result of calling .getall() is a list: it is possible that a selector returns more than one result,

so we extract them all. When you know you just want the ﬁrst result, as in this case, you can do:

>>> response.css("title::text").get()

'Quotes to Scrape'

As an alternative, you could’ve written:

>>> response.css("title::text")[0].get()

'Quotes to Scrape'

Accessing an index on a SelectorList instance will raise an IndexError exception if there are no results:

>>> response.css("noelement")[0].get()

Traceback (most recent call last):

...

IndexError: list index out of range

You might want to use .get() directly on the SelectorList instance instead, which returns None if there are no

results:

>>> response.css("noelement").get()

There’s a lesson here: for most scraping code, you want it to be resilient to errors due to things not being found on a

page, so that even if some parts fail to be scraped, you can at least get some data.

Besides the getall() and get() methods, you can also use the re() method to extract using regular expressions:

>>> response.css("title::text").re(r"Quotes.*")

['Quotes to Scrape']

>>> response.css("title::text").re(r"Q\w+")

['Quotes']

>>> response.css("title::text").re(r"(\w+) to (\w+)")

['Quotes', 'Scrape']

In order to ﬁnd the proper CSS selectors to use, you might ﬁnd it useful to open the response page from the shell in your

web browser using view(response). You can use your browser’s developer tools to inspect the HTML and come up

with a selector (see Using your browser’s Developer Tools for scraping).

Selector Gadget is also a nice tool to quickly ﬁnd CSS selector for visually selected elements, which works in many

browsers.

2.3. Scrapy Tutorial 15

Scrapy Documentation, Release 2.11.1

XPath: a brief intro

Besides CSS, Scrapy selectors also support using XPath expressions:

>>> response.xpath("//title")

[<Selector query='//title' data='<title>Quotes to Scrape</title>'>]

>>> response.xpath("//title/text()").get()

'Quotes to Scrape'

XPath expressions are very powerful, and are the foundation of Scrapy Selectors. In fact, CSS selectors are converted

to XPath under-the-hood. You can see that if you read closely the text representation of the selector objects in the shell.

While perhaps not as popular as CSS selectors, XPath expressions oﬀer more power because besides navigating the

structure, it can also look at the content. Using XPath, you’re able to select things like: select the link that contains the

text “Next Page”. This makes XPath very ﬁtting to the task of scraping, and we encourage you to learn XPath even if

you already know how to construct CSS selectors, it will make scraping much easier.

We won’t cover much of XPath here, but you can read more about using XPath with Scrapy Selectors here. To learn

more about XPath, we recommend this tutorial to learn XPath through examples, and this tutorial to learn “how to think

in XPath”.

Extracting quotes and authors

Now that you know a bit about selection and extraction, let’s complete our spider by writing the code to extract the

quotes from the web page.

Each quote in https://quotes.toscrape.com is represented by HTML elements that look like this:

<span class="text">“The world as we have created it is a process of our

thinking. It cannot be changed without changing our thinking.”</span>

<span>

by <small class="author">Albert Einstein</small>

<a href="/author/Albert-Einstein">(about)</a>

</span>

Tags:

<a class="tag" href="/tag/change/page/1/">change</a>

<a class="tag" href="/tag/deep-thoughts/page/1/">deep-thoughts</a>

<a class="tag" href="/tag/thinking/page/1/">thinking</a>

<a class="tag" href="/tag/world/page/1/">world</a>

</div>

Let’s open up scrapy shell and play a bit to ﬁnd out how to extract the data we want:

scrapy shell 'https://quotes.toscrape.com'

We get a list of selectors for the quote HTML elements with:

>>> response.css("div.quote")

[<Selector query="descendant-or-self::div[@class and contains(concat(' ', normalize-

˓→space(@class), ' '), ' quote ')]" data='<div class="quote" itemscope itemtype...'>,

(continues on next page)

16 Chapter 2. First steps

剩余424页未读，继续阅读

郭映辰

粉丝: 12
资源: 1

Scrapy框架开发指南：从入门到精通

scrapy.pdf

PythonCrawler-Scrapy-Mysql-File-Template, scrapy爬虫框架模板，将数据保存到Mysql数据库或者文件中。.zip

Scrapy：一个快速高级的网站截图和网页采集框架-python

Scrapy，一个用于 Python 的快速高级网页爬虫和数据抓取框架 .zip

funda_scraping:通过Funda +与Scrapy一起玩耍来掠夺房屋市场nl

Scrapy框架的性能优化与扩展技巧

Scrapy性能监控与日志分析：实时追踪与故障排除的艺术

scrapy都需要什么

最新资源