Scrapy 0.24.0电子书教程：Python爬虫框架指南

需积分: 35 37 浏览量更新于2024-07-22 收藏 950KB PDF 举报

"这是一份关于Python框架Scrapy的电子书开发文档，涵盖了从入门到进阶的各种主题，包括安装指南、基本概念如命令行工具、项目结构、爬虫、选择器、项加载器、项管道等，以及内置服务如日志、统计收集、邮件发送等。此外，还涉及解决特定问题的策略，如调试蜘蛛、广度优先爬取、使用Firefox和Firebug进行网页抓取等。" Scrapy是一个强大的Python爬虫框架，用于构建网络爬虫和数据提取项目。它提供了一整套工具和功能，使得开发者可以高效地抓取、处理和存储网页数据。 **Scrapy概览** Scrapy的核心组件包括： 1. **命令行工具**: 用于初始化项目、启动爬虫、管理设置等。 2. **Items**: 定义要抓取的数据结构。 3. **Spiders**: 自定义爬虫类，负责解析响应并生成新的请求或项。 4. **Selectors**: 基于XPath或CSS选择器的工具，用于从HTML或XML文档中提取数据。 5. **ItemLoaders**: 提供一种方便的方式来填充和清理Items。 6. **ItemPipeline**: 处理抓取到的项，进行数据清洗、验证、持久化等操作。 7. **LinkExtractors**: 用于从HTML页面中提取链接，控制爬虫的行为。 8. **内置服务**: 包括日志记录、统计收集、邮件通知等功能，帮助监控和调试爬虫。 **基本概念** - **安装指南**: Scrapy需要Python环境，可以通过pip进行安装。 - **Scrapy Tutorial**: 逐步引导用户创建第一个Scrapy项目。 - **Examples**: 提供实际的代码示例，展示Scrapy的用法。 **解决特定问题** - **调试Spiders**: 提供了各种调试技巧，如使用Scrapy Shell进行交互式调试。 - **Spiders Contracts**: 通过定义合约来确保爬虫行为的正确性。 - **内存泄漏调试**: 教授如何检测和解决Scrapy爬虫中的内存泄漏问题。 - **下载项图片**: 支持自动下载页面中引用的图片。 - **Scrapyd**: 一个部署和管理Scrapy爬虫的服务。 **扩展Scrapy** Scrapy设计为可扩展的，允许用户自定义中间件、下载器处理器、爬虫策略等，以满足特定需求。通过这份Scrapy开发文档，用户不仅可以学习如何使用Scrapy来构建爬虫，还能深入理解其工作原理，从而更好地利用其强大的功能来实现复杂的数据抓取任务。无论是初学者还是有经验的开发者，都能从中获益。

Scrapy Documentation, Release 0.24.0

"http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",

"http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"

]

def parse(self, response):

filename = response.url.split("/")[-2]

with open(filename, ’wb’) as f:

f.write(response.body)

Crawling

To put our spider to work, go to the project’s top level directory and run:

scrapy crawl dmoz

The crawl dmoz command runs the spider for the dmoz.org domain. You will get an output similar to this:

2014-01-23 18:13:07-0400 [scrapy] INFO: Scrapy started (bot: tutorial)

2014-01-23 18:13:07-0400 [scrapy] INFO: Optional features available: ...

2014-01-23 18:13:07-0400 [scrapy] INFO: Overridden settings: {}

2014-01-23 18:13:07-0400 [scrapy] INFO: Enabled extensions: ...

2014-01-23 18:13:07-0400 [scrapy] INFO: Enabled downloader middlewares: ...

2014-01-23 18:13:07-0400 [scrapy] INFO: Enabled spider middlewares: ...

2014-01-23 18:13:07-0400 [scrapy] INFO: Enabled item pipelines: ...

2014-01-23 18:13:07-0400 [dmoz] INFO: Spider opened

2014-01-23 18:13:08-0400 [dmoz] DEBUG: Crawled (200) <GET http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/> (referer: None)

2014-01-23 18:13:09-0400 [dmoz] DEBUG: Crawled (200) <GET http://www.dmoz.org/Computers/Programming/Languages/Python/Books/> (referer: None)

2014-01-23 18:13:09-0400 [dmoz] INFO: Closing spider (finished)

Pay attention to the lines containing [dmoz], which corresponds to our spider. You can see a log line for each URL

deﬁned in start_urls. Because these URLs are the starting ones, they have no referrers, which is shown at the end

of the log line, where it says (referer: None).

But more interesting, as our parse method instructs, two ﬁles have been created: Books and Resources, with the

content of both URLs.

What just happened under the hood?

Scrapy creates scrapy.Request objects for each URL in the start_urls attribute of the Spider, and assigns

them the parse method of the spider as their callback function.

These Requests are scheduled, then executed, and scrapy.http.Response objects are returned and then fed

back to the spider, through the parse() method.

Extracting Items

Introduction to Selectors

There are several ways to extract data from web pages. Scrapy uses a mechanism based on XPath or CSS expressions

called Scrapy Selectors. For more information about selectors and other extraction mechanisms see the Selectors

documentation.

Here are some examples of XPath expressions and their meanings:

• /html/head/title: selects the <title> element, inside the <head> element of a HTML document

12 Chapter 2. First steps

Scrapy Documentation, Release 0.24.0

• /html/head/title/text(): selects the text inside the aforementioned <title> element.

• //td: selects all the <td> elements

• //div[@class="mine"]: selects all div elements which contain an attribute class="mine"

These are just a couple of simple examples of what you can do with XPath, but XPath expressions are indeed much

more powerful. To learn more about XPath we recommend this XPath tutorial.

For working with XPaths, Scrapy provides Selector class and convenient shortcuts to avoid instantiating selectors

yourself everytime you need to select something from a response.

You can see selectors as objects that represent nodes in the document structure. So, the ﬁrst instantiated selectors are

associated with the root node, or the entire document.

Selectors have four basic methods (click on the method to see the complete API documentation):

• xpath(): returns a list of selectors, each of them representing the nodes selected by the xpath expression given

as argument.

• css(): returns a list of selectors, each of them representing the nodes selected by the CSS expression given as

argument.

• extract(): returns a unicode string with the selected data.

• re(): returns a list of unicode strings extracted by applying the regular expression given as argument.

Trying Selectors in the Shell

To illustrate the use of Selectors we’re going to use the built-in Scrapy shell, which also requires IPython (an extended

Python console) installed on your system.

To start a shell, you must go to the project’s top level directory and run:

scrapy shell "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/"

Note: Remember to always enclose urls with quotes when running Scrapy shell from command-line, otherwise urls

containing arguments (ie. & character) will not work.

This is what the shell looks like:

[ ... Scrapy log here ... ]

2014-01-23 17:11:42-0400 [default] DEBUG: Crawled (200) <GET http://www.dmoz.org/Computers/Programming/Languages/Python/Books/> (referer: None)

[s] Available Scrapy objects:

[s] crawler <scrapy.crawler.Crawler object at 0x3636b50>

[s] item {}

[s] request <GET http://www.dmoz.org/Computers/Programming/Languages/Python/Books/>

[s] response <200 http://www.dmoz.org/Computers/Programming/Languages/Python/Books/>

[s] settings <scrapy.settings.Settings object at 0x3fadc50>

[s] spider <Spider ’default’ at 0x3cebf50>

[s] Useful shortcuts:

[s] shelp() Shell help (print this help)

[s] fetch(req_or_url) Fetch request (or URL) and update local objects

[s] view(response) View response in a browser

In [1]:

After the shell loads, you will have the response fetched in a local response variable, so if you type

response.body you will see the body of the response, or you can type response.headers to see its head-

ers.

2.3. Scrapy Tutorial 13

Scrapy Documentation, Release 0.24.0

More important, if you type response.selector you will access a selector object you can use to

query the response, and convenient shortcuts like response.xpath() and response.css() mapping to

response.selector.xpath() and response.selector.css()

So let’s try it:

In [1]: response.xpath(’//title’)

Out[1]: [<Selector xpath=’//title’ data=u’<title>Open Directory - Computers: Progr’>]

In [2]: response.xpath(’//title’).extract()

Out[2]: [u’<title>Open Directory - Computers: Programming: Languages: Python: Books</title>’]

In [3]: response.xpath(’//title/text()’)

Out[3]: [<Selector xpath=’//title/text()’ data=u’Open Directory - Computers: Programming:’>]

In [4]: response.xpath(’//title/text()’).extract()

Out[4]: [u’Open Directory - Computers: Programming: Languages: Python: Books’]

In [5]: response.xpath(’//title/text()’).re(’(\w+):’)

Out[5]: [u’Computers’, u’Programming’, u’Languages’, u’Python’]

Extracting the data

Now, let’s try to extract some real information from those pages.

You could type response.body in the console, and inspect the source code to ﬁgure out the XPaths you need to

use. However, inspecting the raw HTML code there could become a very tedious task. To make this an easier task,

you can use some Firefox extensions like Firebug. For more information see Using Firebug for scraping and Using

Firefox for scraping.

After inspecting the page source, you’ll ﬁnd that the web sites information is inside a <ul> element, in fact the second

<ul> element.

So we can select each <li> element belonging to the sites list with this code:

sel.xpath(’//ul/li’)

And from them, the sites descriptions:

sel.xpath(’//ul/li/text()’).extract()

The sites titles:

sel.xpath(’//ul/li/a/text()’).extract()

And the sites links:

sel.xpath(’//ul/li/a/@href’).extract()

As we’ve said before, each .xpath() call returns a list of selectors, so we can concatenate further .xpath() calls

to dig deeper into a node. We are going to use that property here, so:

for sel in response.xpath(’//ul/li’)

title = sel.xpath(’a/text()’).extract()

link = sel.xpath(’a/@href’).extract()

desc = sel.xpath(’text()’).extract()

print title, link, desc

Note: For a more detailed description of using nested selectors, see Nesting selectors and Working with relative

14 Chapter 2. First steps

Scrapy Documentation, Release 0.24.0

Note: You can ﬁnd a fully-functional variant of this spider in the dirbot project available at

https://github.com/scrapy/dirbot

Now doing a crawl on the dmoz.org domain yields DmozItem objects:

[dmoz] DEBUG: Scraped from <200 http://www.dmoz.org/Computers/Programming/Languages/Python/Books/>

{’desc’: [u’ - By David Mertz; Addison Wesley. Book in progress, full text, ASCII format. Asks for feedback. [author website, Gnosis Software, Inc.\n],

’link’: [u’http://gnosis.cx/TPiP/’],

’title’: [u’Text Processing in Python’]}

[dmoz] DEBUG: Scraped from <200 http://www.dmoz.org/Computers/Programming/Languages/Python/Books/>

{’desc’: [u’ - By Sean McGrath; Prentice Hall PTR, 2000, ISBN 0130211192, has CD-ROM. Methods to build XML applications fast, Python tutorial, DOM and SAX, new Pyxie open source XML processing library. [Prentice Hall PTR]\n’],

’link’: [u’http://www.informit.com/store/product.aspx?isbn=0130211192’],

’title’: [u’XML Processing with Python’]}

2.3.4 Storing the scraped data

The simplest way to store the scraped data is by using the Feed exports, with the following command:

scrapy crawl dmoz -o items.json

That will generate a items.json ﬁle containing all scraped items, serialized in JSON.

In small projects (like the one in this tutorial), that should be enough. However, if you want to perform more complex

things with the scraped items, you can write an Item Pipeline. As with Items, a placeholder ﬁle for Item Pipelines

has been set up for you when the project is created, in tutorial/pipelines.py. Though you don’t need to

implement any item pipelines if you just want to store the scraped items.

2.3.5 Next steps

This tutorial covers only the basics of Scrapy, but there’s a lot of other features not mentioned here. Check the What

else? section in Scrapy at a glance chapter for a quick overview of the most important ones.

Then, we recommend you continue by playing with an example project (see Examples), and then continue with the

section Basic concepts.

2.4 Examples

The best way to learn is with examples, and Scrapy is no exception. For this reason, there is an example Scrapy project

named dirbot, that you can use to play and learn more about Scrapy. It contains the dmoz spider described in the

tutorial.

This dirbot project is available at: https://github.com/scrapy/dirbot

It contains a README ﬁle with a detailed description of the project contents.

If you’re familiar with git, you can checkout the code. Otherwise you can download a tarball or zip ﬁle of the project

by clicking on Downloads.

The scrapy tag on Snipplr is used for sharing code snippets such as spiders, middlewares, extensions, or scripts. Feel

free (and encouraged!) to share any code there.

Scrapy at a glance Understand what Scrapy is and how it can help you.

Installation guide Get Scrapy installed on your computer.

16 Chapter 2. First steps

剩余204页未读，继续阅读

谁的Utopia

粉丝: 12
资源: 5

Scrapy 0.24.0电子书教程：Python爬虫框架指南

scrapy.pdf

scrapy-1.4.pdf

学习scrapy书和代码

python scrapy一键爬取

python scrapy 项目创建

pythonscrapy和yield

利用python scrapy框架爬取“房天下”网站房源数据

python scrapy top250

python scrapy增量更新

Python Scrapy微博热搜榜

最新资源