Scrapy 1.1 官方文档详解

2星需积分: 14 89 浏览量更新于2024-07-20 4 收藏 1.06MB PDF 举报

"Scrapy1.1 帮助文档" Scrapy是一个强大的Python爬虫框架，用于高效地抓取网页内容并处理数据。该文档详细介绍了Scrapy 1.1版本的功能和用法，旨在帮助开发者快速上手并熟练掌握Scrapy的各项特性。 **第一部分：初步了解** 1. **获取帮助**：Scrapy社区提供了多种途径获取帮助，包括官方文档、邮件列表、IRC频道以及Stack Overflow等在线平台。 2. **第一步**：介绍Scrapy的基本概念，引导用户安装Scrapy并创建第一个项目。 3. **Scrapy概览**：解释Scrapy架构的核心组成部分，如引擎、调度器、下载器和蜘蛛等。 4. **安装指南**：详细说明在不同操作系统上安装Scrapy的步骤。 5. **Scrapy教程**：提供一个逐步的指南，帮助新手快速掌握Scrapy基础。 6. **示例**：包含多个实际示例，展示Scrapy在实际爬虫项目中的应用。 **第二部分：基本概念** 1. **命令行工具**：介绍`scrapy`命令行工具的使用，包括创建项目、运行蜘蛛等操作。 2. **蜘蛛(Spiders)**：蜘蛛是Scrapy的核心，用于定义爬取规则和数据解析逻辑。 3. **选择器(Selectors)**：Scrapy基于lxml库提供了XPath和CSS选择器，用于从HTML或XML文档中提取数据。 4. **Items**：定义要爬取的数据结构，便于后续处理和存储。 5. **Item Loaders**：用于填充Items，方便数据清洗和转换。 6. **Scrapy Shell**：交互式环境，可快速测试选择器和数据提取。 7. **Item Pipeline**：处理从蜘蛛中提取的Item数据，实现数据清洗、验证和持久化。 8. **Feed Exports**：将爬取结果导出到各种格式，如CSV、JSON等。 9. **请求(Requests)与响应(Responses)**：请求用于发起网络请求，响应包含了服务器返回的数据。 10. **链接提取器(Link Extractors)**：自动从页面中提取链接，帮助蜘蛛跟踪页面。 11. **设置(Settings)**：配置Scrapy项目的各种选项，如中间件、代理、下载延迟等。 12. **异常(Exceptions)**：列举了Scrapy中可能出现的异常类型及其处理。 **第三部分：内置服务** 1. **日志(Logging)**：配置和使用Scrapy的日志系统，监控爬虫运行状态。 2. **统计收集(Stats Collection)**：收集关于爬虫运行的统计信息，如爬取的URL数量、时间消耗等。 3. **发送邮件**：配置Scrapy在特定事件发生时发送电子邮件通知。 4. **Telnet控制台(Telnet Console)**：通过telnet连接到Scrapy引擎，实时查看和调试爬虫状态。 5. **Web Service**：提供一个基于HTTP的接口，用于远程监控和控制爬虫。 **第四部分：解决特定问题** 1. **常见问题**：解答用户在使用Scrapy时可能遇到的问题。 2. **调试蜘蛛**：提供调试技巧和工具，帮助找出代码中的错误。 3. **蜘蛛契约(Spider Contracts)**：用于验证蜘蛛行为的断言，确保其按预期工作。 4. **最佳实践**：推荐的编码和项目管理规范，提高代码质量和效率。 5. **广度优先爬取**：指导如何配置Scrapy进行广度优先的网页抓取。 6. **使用Firefox进行爬取**：使用Firefox浏览器的开发者工具辅助爬虫开发。 7. **使用Firebug进行爬取**：介绍如何利用Firebug插件辅助解析和调试网页。 8. **调试内存泄漏**：针对Scrapy可能出现的内存问题提供解决方法。 9. **下载和处理文件及图片**：讲解如何抓取和处理非文本内容，如图片和文件。 10. **Ubuntu包**：在Ubuntu系统中安装Scrapy的方法。 11. **部署蜘蛛**：介绍如何将Scrapy项目部署到生产环境。 12. **AutoThrottle扩展**：自动调整请求速率，避免对目标网站造成过大压力。 13. **基准测试**：进行性能评估，优化爬虫速度。 14. **Job Board**：可能是指Scrapy社区提供的工作板，分享和寻找与Scrapy相关的职位。 Scrapy 1.1的帮助文档涵盖了从安装、使用到高级特性和问题解决的全面内容，是学习和使用Scrapy进行网络爬虫开发的重要参考资料。

Scrapy Documentation, Release 1.1.0

"http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",

"http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"

]

def parse(self, response):

filename = response.url.split("/")[-2] + '.html'

with open(filename, 'wb') as f:

f.write(response.body)

Crawling

To put our spider to work, go to the project’s top level directory and run:

scrapy crawl dmoz

This command runs the spider with name dmoz that we’ve just added, that will send some requests for the dmoz.org

domain. You will get an output similar to this:

2014-01-23 18:13:07-0400 [scrapy] INFO: Scrapy started (bot: tutorial)

2014-01-23 18:13:07-0400 [scrapy] INFO: Optional features available: ...

2014-01-23 18:13:07-0400 [scrapy] INFO: Overridden settings: {}

2014-01-23 18:13:07-0400 [scrapy] INFO: Enabled extensions: ...

2014-01-23 18:13:07-0400 [scrapy] INFO: Enabled downloader middlewares: ...

2014-01-23 18:13:07-0400 [scrapy] INFO: Enabled spider middlewares: ...

2014-01-23 18:13:07-0400 [scrapy] INFO: Enabled item pipelines: ...

2014-01-23 18:13:07-0400 [scrapy] INFO: Spider opened

2014-01-23 18:13:08-0400 [scrapy] DEBUG: Crawled (200) <GET http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/> (referer: None)

2014-01-23 18:13:09-0400 [scrapy] DEBUG: Crawled (200) <GET http://www.dmoz.org/Computers/Programming/Languages/Python/Books/> (referer: None)

2014-01-23 18:13:09-0400 [scrapy] INFO: Closing spider (finished)

Note: At the end you can see a log line for each URL deﬁned in start_urls. Because these URLs are the starting

ones, they have no referrers, which is shown at the end of the log line, where it says (referer: None).

Now, check the ﬁles in the current directory. You should notice two new ﬁles have been created: Books.html and

Resources.html, with the content for the respective URLs, as our parse method instructs.

What just happened under the hood?

Scrapy creates scrapy.Request objects for each URL in the start_urls attribute of the Spider, and assigns

them the parse method of the spider as their callback function.

These Requests are scheduled, then executed, and scrapy.http.Response objects are returned and then fed

back to the spider, through the parse() method.

Extracting Items

Introduction to Selectors

There are several ways to extract data from web pages. Scrapy uses a mechanism based on XPath or CSS expressions

called Scrapy Selectors. For more information about selectors and other extraction mechanisms see the Selectors

documentation.

Here are some examples of XPath expressions and their meanings:

12 Chapter 2. First steps

Scrapy Documentation, Release 1.1.0

• /html/head/title: selects the <title> element, inside the <head> element of an HTML document

• /html/head/title/text(): selects the text inside the aforementioned <title> element.

• //td: selects all the <td> elements

• //div[@class="mine"]: selects all div elements which contain an attribute class="mine"

These are just a couple of simple examples of what you can do with XPath, but XPath expressions are indeed much

more powerful. To learn more about XPath, we recommend this tutorial to learn XPath through examples, and this

tutorial to learn “how to think in XPath”.

Note: CSS vs XPath: you can go a long way extracting data from web pages using only CSS selectors. However,

XPath offers more power because besides navigating the structure, it can also look at the content: you’re able to select

things like: the link that contains the text ‘Next Page’. Because of this, we encourage you to learn about XPath even if

you already know how to construct CSS selectors.

For working with CSS and XPath expressions, Scrapy provides Selector class and convenient shortcuts to avoid

instantiating selectors yourself every time you need to select something from a response.

You can see selectors as objects that represent nodes in the document structure. So, the ﬁrst instantiated selectors are

associated with the root node, or the entire document.

Selectors have four basic methods (click on the method to see the complete API documentation):

• xpath(): returns a list of selectors, each of which represents the nodes selected by the xpath expression given

as argument.

• css(): returns a list of selectors, each of which represents the nodes selected by the CSS expression given as

argument.

• extract(): returns a unicode string with the selected data.

• re(): returns a list of unicode strings extracted by applying the regular expression given as argument.

Trying Selectors in the Shell

To illustrate the use of Selectors we’re going to use the built-in Scrapy shell, which also requires IPython (an extended

Python console) installed on your system.

To start a shell, you must go to the project’s top level directory and run:

scrapy shell "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/"

Note: Remember to always enclose urls in quotes when running Scrapy shell from command-line, otherwise urls

containing arguments (ie. & character) will not work.

This is what the shell looks like:

[ ... Scrapy log here ... ]

2014-01-23 17:11:42-0400 [scrapy] DEBUG: Crawled (200) <GET http://www.dmoz.org/Computers/Programming/Languages/Python/Books/> (referer: None)

[s] Available Scrapy objects:

[s] crawler <scrapy.crawler.Crawler object at 0x3636b50>

[s] item {}

[s] request <GET http://www.dmoz.org/Computers/Programming/Languages/Python/Books/>

[s] response <200 http://www.dmoz.org/Computers/Programming/Languages/Python/Books/>

2.3. Scrapy Tutorial 13

Scrapy Documentation, Release 1.1.0

[s] settings <scrapy.settings.Settings object at 0x3fadc50>

[s] spider <Spider 'default' at 0x3cebf50>

[s] Useful shortcuts:

[s] shelp() Shell help (print this help)

[s] fetch(req_or_url) Fetch request (or URL) and update local objects

[s] view(response) View response in a browser

In [1]:

After the shell loads, you will have the response fetched in a local response variable, so if you type

response.body you will see the body of the response, or you can type response.headers to see its head-

ers.

More importantly response has a selector attribute which is an instance of Selector

class, instantiated with this particular response. You can run queries on response by calling

response.selector.xpath() or response.selector.css(). There are also some convenience short-

cuts like response.xpath() or response.css() which map directly to response.selector.xpath()

and response.selector.css().

So let’s try it:

In [1]: response.xpath('//title')

Out[1]: [<Selector xpath='//title' data=u'<title>Open Directory - Computers: Progr'>]

In [2]: response.xpath('//title').extract()

Out[2]: [u'<title>Open Directory - Computers: Programming: Languages: Python: Books</title>']

In [3]: response.xpath('//title/text()')

Out[3]: [<Selector xpath='//title/text()' data=u'Open Directory - Computers: Programming:'>]

In [4]: response.xpath('//title/text()').extract()

Out[4]: [u'Open Directory - Computers: Programming: Languages: Python: Books']

In [5]: response.xpath('//title/text()').re('(\w+):')

Out[5]: [u'Computers', u'Programming', u'Languages', u'Python']

Extracting the data

Now, let’s try to extract some real information from those pages.

You could type response.body in the console, and inspect the source code to ﬁgure out the XPaths you need to

use. However, inspecting the raw HTML code there could become a very tedious task. To make it easier, you can

use Firefox Developer Tools or some Firefox extensions like Firebug. For more information see Using Firebug for

scraping and Using Firefox for scraping.

After inspecting the page source, you’ll ﬁnd that the web site’s information is inside a <ul> element, in fact the

second <ul> element.

So we can select each <li> element belonging to the site’s list with this code:

response.xpath('//ul/li')

And from them, the site’s descriptions:

response.xpath('//ul/li/text()').extract()

The site’s titles:

14 Chapter 2. First steps

Scrapy Documentation, Release 1.1.0

class DmozSpider(scrapy.Spider):

name = "dmoz"

allowed_domains = ["dmoz.org"]

start_urls = [

"http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",

"http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"

]

def parse(self, response):

for sel in response.xpath('//ul/li'):

item = DmozItem()

item['title'] = sel.xpath('a/text()').extract()

item['link'] = sel.xpath('a/@href').extract()

item['desc'] = sel.xpath('text()').extract()

yield item

Note: You can ﬁnd a fully-functional variant of this spider in the dirbot project available at

https://github.com/scrapy/dirbot

Now crawling dmoz.org yields DmozItem objects:

[scrapy] DEBUG: Scraped from <200 http://www.dmoz.org/Computers/Programming/Languages/Python/Books/>

{'desc': [u' - By David Mertz; Addison Wesley. Book in progress, full text, ASCII format. Asks for feedback. [author website, Gnosis Software, Inc.\n],

'link': [u'http://gnosis.cx/TPiP/'],

'title': [u'Text Processing in Python']}

[scrapy] DEBUG: Scraped from <200 http://www.dmoz.org/Computers/Programming/Languages/Python/Books/>

{'desc': [u' - By Sean McGrath; Prentice Hall PTR, 2000, ISBN 0130211192, has CD-ROM. Methods to build XML applications fast, Python tutorial, DOM and SAX, new Pyxie open source XML processing library. [Prentice Hall PTR]\n'],

'link': [u'http://www.informit.com/store/product.aspx?isbn=0130211192'],

'title': [u'XML Processing with Python']}

2.3.4 Following links

Let’s say, instead of just scraping the stuff in Books and Resources pages, you want everything that is under the Python

directory.

Now that you know how to extract data from a page, why not extract the links for the pages you are interested, follow

them and then extract the data you want for all of them?

Here is a modiﬁcation to our spider that does just that:

import scrapy

from tutorial.items import DmozItem

class DmozSpider(scrapy.Spider):

name = "dmoz"

allowed_domains = ["dmoz.org"]

start_urls = [

"http://www.dmoz.org/Computers/Programming/Languages/Python/",

]

def parse(self, response):

for href in response.css("ul.directory.dir-col > li > a::attr('href')"):

url = response.urljoin(href.extract())

yield scrapy.Request(url, callback=self.parse_dir_contents)

16 Chapter 2. First steps

剩余247页未读，继续阅读

flyinghorse_2012

粉丝: 4
资源: 7

Scrapy 1.1 官方文档详解

scrapy中文教程（官方）

scrapy-0.24中文文档|中文教程

scrapy 中文教程

scrapy1.1参考手册

Scrapy 1.1新手指南与核心概念详解

win10系统中安装scrapy-1.1

pythonscrapy爬虫实例Python爬虫Scrapy实例

20B_基于Scrapy的WebUI开发_项目设计文档v1.0.01

scrapy爬虫教程

scrapy.pdf

最新资源