Python内置URL库urllib3的全面指南与实践

需积分: 20 132 浏览量更新于2024-07-20 1 收藏 347KB PDF 举报

Python 自带的 URL 处理库文档提供了全面的指南，用于在 Python 开发中高效地处理 URL、发送 HTTP 请求、管理响应内容以及处理各种网络通信问题。该库主要由 `urllib3` 组件构成，它包含多个子模块和功能，如 `urllib3.PoolManager` 用于创建连接池进行批量请求，`urllib3.exceptions` 提供错误和异常处理机制，`urllib3.connection` 和 `urllib3.response` 用于建立和解析网络连接。以下是文档中的关键知识点： 1. **用户指南**： - **发送请求**：文档介绍了如何使用 `urllib3.request` 来构建请求对象，并通过 `PoolManager` 发送 HTTP 请求，包括 GET、POST 等方法。 - **响应内容**：处理返回的响应，包括解析 JSON、XML 或文本内容，以及错误码和状态码检查。 - **数据传输**：包括请求头的设置和请求体的管理，支持多种数据格式如表单数据、文件上传等。 2. **高级用法**： - **自定义连接池行为**：允许开发者根据项目需求调整连接池的大小、超时时间、重试策略等。 - **流式处理和 I/O**：支持分块读取响应数据，减少内存占用，适合处理大文件或持续传输。 - **代理支持**：处理通过代理服务器进行的网络请求，适用于公司内网或限制访问的情况。 - **SSL证书管理**：包括定制 SSL 证书、客户端证书以及验证规则，确保安全连接。 - **证书验证和 MacOS X**：针对特定平台的证书处理，解决兼容性问题。 - **SSL警告处理**：处理可能出现的 SSL 错误和警告，提高程序健壮性。 - **Google App Engine**：与 Google App Engine 的集成，考虑了特定环境下的配置和优化。 3. **参考**： - **子包和模块**：列举了 `urllib3` 包下的各个模块，如 `connection`, `poolmanager`, `request`, `response` 等，每个模块都有详细的接口说明。 - **异常模块**：展示了遇到网络问题时可能抛出的异常类型和处理方式。 - **字段模块**：涉及请求和响应中使用的元数据和参数。 4. **贡献**： - **运行测试**：指导如何运行官方提供的测试用例，确保代码质量和稳定性。 - **赞助和支持**：鼓励社区成员通过赞助或申请项目资助来支持库的维护和发展。 5. **安装与使用**：文档提供了安装 `urllib3` 的步骤，以及如何将其与其他 Python 库结合使用。 6. **用途**：说明了 `urllib3` 库在实际开发中的应用，如网站爬虫、API 调用、自动化测试等场景。 7. **许可证**：阐述了库的开源许可协议，确保了代码的可用性和可复制性。 8. **贡献指南**：详细列出如何参与项目开发，包括提交 bug 报告、修复、编写文档等。 Python 自带的 `urllib3` URL 处理库文档是 Python 开发者处理网络请求、实现高效并发网络操作和确保安全通信的重要工具，它提供了丰富的功能和易用的 API，使得网络编程变得更加便捷。

urllib3 Documentation, Release dev

Any new requests will block until a connection is available from the pool. This is a great way to prevent ﬂooding a

host with too many connections in multi-threaded applications.

Streaming and IO

When dealing with large responses it’s often better to stream the response content:

>>> import urllib3

>>> http = urllib3.PoolManager()

>>> r = http.request(

... 'GET',

... 'http://httpbin.org/bytes/1024',

... preload_content=False)

>>> for chunk in r.stream(32):

... print(chunk)

b'...'

...

>>> r.release_conn()

Setting preload_content to False means that urllib3 will stream the response content. stream() lets you

iterate over chunks of the response content.

Note: When using preload_content=False, you should call release_conn() to release the http connec-

tion back to the connection pool so that it can be re-used.

However, you can also treat the HTTPResponse instance as a ﬁle-like object. This allows you to do buffering:

>>> r = http.request(

... 'GET',

... 'http://httpbin.org/bytes/1024',

... preload_content=False)

>>> r.read(4)

b'\x88\x1f\x8b\xe5'

Calls to read() will block until more response data is available.

>>> import io

>>> reader = io.BufferedReader(r, 8)

>>> reader.read(4)

>>> r.release_conn()

You can use this ﬁle-like object to do things like decode the content using codecs:

>>> import codecs

>>> reader = codecs.getreader('utf-8')

>>> r = http.request(

... 'GET',

... 'http://httpbin.org/ip',

... preload_content=False)

>>> json.load(reader(r))

{'origin': '127.0.0.1'}

>>> r.release_conn()

10 Chapter 2. Advanced Usage

urllib3 Documentation, Release dev

Proxies

You can use ProxyManager to tunnel requests through an HTTP proxy:

>>> import urllib3

>>> proxy = urllib3.ProxyManager('http://localhost:3128/')

>>> proxy.request('GET', 'http://google.com/')

The usage of ProxyManager is the same as PoolManager.

You can use SOCKSProxyManager to connect to SOCKS4 or SOCKS5 proxies. In order to use SOCKS proxies

you will need to install PySocks or install urllib3 with the socks extra:

pip install urllib3[socks]

Once PySocks is installed, you can use SOCKSProxyManager:

>>> from urllib3.contrib.socks import SOCKSProxyManager

>>> proxy = SOCKSProxyManager('socks5://localhost:8889/')

>>> proxy.request('GET', 'http://google.com/')

Custom SSL certiﬁcates and client certiﬁcates

Instead of using certiﬁ you can provide your own certiﬁcate authority bundle. This is useful for cases where you’ve

generated your own certiﬁcates or when you’re using a private certiﬁcate authority. Just provide the full path to the

certiﬁcate bundle when creating a PoolManager:

>>> import urllib3

>>> http = urllib3.PoolManager(

... cert_reqs='CERT_REQUIRED',

... ca_certs='/path/to/your/certificate_bundle')

When you specify your own certiﬁcate bundle only requests that can be veriﬁed with that bundle will succeed. It’s

recommended to use a separate PoolManager to make requests to URLs that do not need the custom certiﬁcate.

You can also specify a client certiﬁcate. This is useful when both the server and the client need to verify each other’s

identity. Typically these certiﬁcates are issued from the same authority. To use a client certiﬁcate, provide the full path

when creating a PoolManager:

>>> http = urllib3.PoolManager(

... cert_file='/path/to/your/client_cert.pem',

... cert_reqs='CERT_REQUIRED',

... ca_certs='/path/to/your/certificate_bundle')

Certiﬁcate validation and Mac OS X

Apple-provided Python and OpenSSL libraries contain a patches that make them automatically check the system

keychain’s certiﬁcates. This can be surprising if you specify custom certiﬁcates and see requests unexpectedly succeed.

For example, if you are specifying your own certiﬁcate for validation and the server presents a different certiﬁcate you

would expect the connection to fail. However, if that server presents a certiﬁcate that is in the system keychain then

the connection will succeed.

This article has more in-depth analysis and explanation.

2.3. Proxies 11

urllib3 Documentation, Release dev

SSL Warnings

urllib3 will issue several different warnings based on the level of certiﬁcate veriﬁcation support. These warning

indicate particular situations and can resolved in different ways.

• InsecureRequestWarning This happens when an request is made to an HTTPS URL without certiﬁcate

veriﬁcation enabled. Follow the certiﬁcate veriﬁcation guide to resolve this warning.

• InsecurePlatformWarning This happens on Python 2 platforms that have an outdated ssl module.

These older ssl modules can cause some insecure requests to succeed where they should fail and secure

requests to fail where they should succeed. Follow the pyOpenSSL guide to resolve this warning.

• SNIMissingWarning This happens on Python 2 versions older than 2.7.9. These older versions lack SNI

support. This can cause servers to present a certiﬁcate that the client thinks is invalid. Follow the py-

OpenSSL guide to resolve this warning.

Making unveriﬁed HTTPS requests is strongly discouraged, however, if you understand the risks and wish to disable

these warnings, you can use disable_warnings():

>>> import urllib3

>>> urllib3.disable_warnings()

Alternatively you can capture the warnings with the standard logging module:

>>> logging.captureWarnings(True)

Finally, you can suppress the warnings at the interpreter level by setting the PYTHONWARNINGS environment variable

or by using the -W ﬂag.

Google App Engine

urllib3 supports Google App Engine with some caveats.

If you’re using the Flexible environment, you do not have to do any conﬁguration- urllib3 will just work. How-

ever, if you’re using the Standard environment then you either have to use urllib3.contrib.appengine‘s

AppEngineManager or use the Sockets API

To use AppEngineManager:

>>> from urllib3.contrib.appengine import AppEngineManager

>>> http = AppEngineManager()

>>> http.request('GET', 'https://google.com/')

To use the Sockets API, add the following to your app.yaml and use PoolManager as usual:

env_variables:

GAE_USE_SOCKETS_HTTPLIB : 'true'

For more details on the limitations and gotchas, see urllib3.contrib.appengine.

12 Chapter 2. Advanced Usage

剩余80页未读，继续阅读

普通网友

粉丝: 222
资源:
28

Python内置URL库urllib3的全面指南与实践

python url 参数修改方法

python根据url地址下载小文件的实例

python 解析url

常用Python第三方库介绍

Python 自带有哪些库

请告诉我，python的http库的详细用法

python 解析html

通过python代码实现爬取税屋网站https://www.shui5.cn/article/BeiJingShiCaiShuiFaGui/10页数据并写入txt文档

python匹配网页内容

python爬虫怎么使用

最新资源