帮我写个scrapy爬虫的python代码,爬取baidu.com
时间: 2023-06-03 22:03:55 浏览: 147
基于Python的Scrapy爬虫框架
当然可以,以下是示例代码:
```python
import scrapy
class BaiduSpider(scrapy.Spider):
name = 'baidu'
allowed_domains = ['baidu.com']
start_urls = ['http://www.baidu.com/s?wd=python']
def parse(self, response):
for result in response.css('.c-container'):
title = result.css('.t a::text').get()
link = result.css('.t a::attr(href)').get()
yield {
'title': title,
'link': link,
}
next_page = response.css('#page a:last-of-type::attr(href)').get()
if next_page is not None:
yield response.follow(next_page, self.parse)
```
这只是一个简单的例子,你需要根据你的实际需求进行修改。
阅读全文