Python爬虫实战：数据科学中的最佳实践与案例解析

需积分: 10 198 浏览量更新于2024-07-18 1 收藏 4.79MB PDF 举报

身份认证购VIP最低享 7 折!

领优惠券(最高得80元）

"《Practical Web Scraping for Data Science》是由Seppe vanden Broucke和Bart Baesens合著的一本关于使用Python进行网络爬虫的书籍，旨在为数据科学家提供现代且全面的指导。书中深入浅出地介绍了网络爬虫技术，并结合实际案例，同时探讨了网络技术的大环境，确保读者能够充分理解。作者强调网络爬虫在数据科学项目中的重要性，因为许多项目都始于获取合适的数据集。书中的内容包括基础的HTTP、HTML、CSS概念，Python简介，Selenium用于处理JavaScript丰富的网站，以及详细的网络爬虫技术。最后，书中总结了最佳实践并提供了多个实例，展示如何将所学应用于不同的数据科学场景。这本书适合已经熟悉Python或其他编程语言或分析工具（如R、SAS、SPSS等）的数据科学爱好者，同时也适用于大学课程的学生和教师。对于不熟悉Python的读者，第一章还提供了快速入门指南，帮助他们掌握基础知识。" 通过本书，你可以学习到： 1. 如何利用已建立的最佳实践和常用的Python包进行网络爬虫。 2. 应对现代网络挑战，包括处理JavaScript、管理cookies以及对抗常见的网页防爬策略。 3. 了解网络爬虫的法律和管理问题，确保你的行为在法律允许的范围内。本书面向的读者群体是： 1. 对数据科学有一定背景，可能已经熟悉Python或其他编程语言或分析工具的专业人士。 2. 高校学生和教师，特别是在数据分析或相关领域学习的人群。 3. 对Python不太熟悉的读者，可以通过书中的快速Python入门部分来补充基础知识。书中的部分内容提到了商标、名字和图片的使用，但主要关注的是网络爬虫技术及其在数据科学中的应用。通过阅读此书，读者不仅可以掌握网络爬虫技术，还能了解到如何在实际操作中遵循法律和道德规范，从而安全有效地进行数据收集和分析。

资源详情

资源推荐

1.1.1 Why Web Scraping forData Science?

When surfing the web using a normal web browser, you’ve probably encountered

multiple sites where you considered the possibility of gathering, storing, and analyzing

the data presented on the site’s pages. Especially for data scientists, whose “raw

material” is data, the web exposes a lot of interesting opportunities:

• There might be an interesting table on a Wikipedia page (or pages)

you want to retrieve to perform some statistical analysis.

• Perhaps you want to get a list of reviews from a movie site to perform

text mining, create a recommendation engine, or build a predictive

model to spot fake reviews.

• You might wish to get a listing of properties on a real-estate site to

build an appealing geo-visualization.

• You’d like to gather additional features to enrich your data set based

on information found on the web, say, weather information to

forecast, for example, soft drink sales.

• You might be wondering about doing social network analytics using

prole data found on a web forum.

• It might be interesting to monitor a news site for trending new stories

on a particular topic of interest.

The web contains lots of interesting data sources that provide a treasure trove for all

sorts of interesting things. Sadly, the current unstructured nature of the web does not

always make it easy to gather or export this data in an easy manner. Web browsers are

very good at showing images, displaying animations, and laying out websites in a way

that is visually appealing to humans, but they do not expose a simple way to export their

data, at least not in most cases. Instead of viewing the web page by page through your

web browser’s window, wouldn’t it be nice to be able to automatically gather a rich data

set? This is exactly where web scraping enters the picture.

If you know your way around the web a bit, you’ll probably be wondering: “Isn’t

this exactly what Application Programming Interface (APIs) are for?” Indeed, many

websites nowadays provide such an API that provides a means for the outside world to

access their data repository in a structured way— meant to be consumed and accessed

CHAPTER 1 INTRODUCTION

• Scraping is being applied a lot in HR and employee analytics. e San

Francisco-based hiQ startup specializes in selling employee analyses

by collecting and examining public prole information, for instance,

from LinkedIn (who was not happy about this but was so far unable

to prevent this practice following a court case; see https://www.

bloomberg.com/news/features/2017-11-15/the-brutal-ﬁght-to-

mine- your-data-and-sell-it-to-your-boss).

• Digital marketeers and digital artists often use data from the web

for all sorts of interesting and creative projects. “We Feel Fine” by

Jonathan Harris and Sep Kamvar, for instance, scraped various blog

sites for phrases starting with “I feel,” the results of which could then

visualize how the world was feeling throughout the day.

• In another study, messages scraped from Twitter, blogs, and other

social media were scraped to construct a data set that was used to

build a predictive model toward identifying patterns of depression

and suicidal thoughts. is might be an invaluable tool for aid

providers, though of course it warrants a thorough consideration

of privacy related issues as well (see https://www.sas.com/en_ca/

insights/articles/analytics/using-big-data-to-predict-

suicide-risk-canada.html).

• Emmanuel Sales also scraped Twitter, though here with the goal

to make sense of his own social circle and time line of posts (see

https://emsal.me/blog/4). An interesting observation here is

that the author rst considered using Twitter’s API, but found that

“Twitter heavily rate limits doing this: if you want to get a user’s

follow list, then you can only do so 15 times every 15 minutes, which

is pretty unwieldy to work with.”

• In a paper titled “e Billion Prices Project: Using Online Prices for

Measurement and Research” (see http://www.nber.org/papers/

w22111), web scraping was used to collect a data set of online price

information that was used to construct a robust daily price index for

multiple countries.

CHAPTER 1 INTRODUCTION

• Banks and other nancial institutions are using web scraping

for competitor analysis. For example, banks frequently scrape

competitors’ sites to get an idea of where branches are being opened

or closed, or to track loan rates oered— all of which is interesting

information that can be incorporated in their internal models and

forecasting. Investment rms also often use web scraping, for instance,

to keep track of news articles regarding assets in their portfolio.

• Sociopolitical scientists are scraping social websites to track

population sentiment and political orientation. A famous article

called “Dissecting Trump’s Most Rabid Online Following”

(see https://ﬁvethirtyeight.com/features/dissecting-trumps-

most-rabid-online-following/) analyzes user discussions on

Reddit using semantic analysis to characterize the online followers

and fans of Donald Trump.

• One researcher was able to train a deep learning model based on

scraped images from Tinder and Instagram together with their “likes”

to predict whether an image would be deemed “attractive”

(see http://karpathy.github.io/2015/10/25/selﬁe/).

Smartphone makers are already incorporating such models in their

photo apps to help you brush up your pictures.

• In “e Girl with the Brick Earring,” Lucas Woltmann sets out to

scrape Lego brick information from https://www.bricklink.com

to determine the best selection of Lego pieces (see http://

lucaswoltmann.de/art'n'images/2017/04/08/the-girl-with-the-

brick- earring.html) to represent an image (one of the co-authors of

this book is an avid Lego fan, so we had to include this example).

• In “Analyzing 1000+ Greek Wines With Python,” Florents Tselai

scrapes information about a thousand wine varieties from a Greek

wine shop (see https://tselai.com/greek-wines-analysis.html)

to analyze their origin, rating, type, and strength (one of the co-

authors of this book is an avid wine fan, so we had to include this

example).

CHAPTER 1 INTRODUCTION

剩余312页未读，继续阅读

mengweilil

粉丝: 104
资源: 66

Python爬虫实战：数据科学中的最佳实践与案例解析

python爬虫外文文献

import requests from lxml import etree import pandas as pd

python的爬虫教程你有推荐嘛

python utils

python爬虫的参考文献

Error scraping for collect.slave_status: Error 1227: Access denied; you need (at least one of) the SUPER, REPLICATION CLIENT privilege(s) for this operation" source="exporter.go:171"

解释一下from selenium import webdriver

watir-webdriver ssh

用python爬取豆瓣图书网并进行可视化分析的相关参考文献

beautiful soup soup

我想从大众点评上批量获取商家信息该怎么实现

二手房数据爬取参考文献

Python automation

列举一个例子，使用Anaconda爬虫抓取数据代码

pip search

classify titles by their similarities using Python and pleasue using Clustering

python3.10导入数据表

解析html并提取所需信息的代码

关于python爬虫的文献

python 数据分析师面试常见问题

最新资源