利用python的socket发送http(s)请求方法示例_pythonhttp通信

1星 171 浏览量更新于2023-03-03 评论收藏 66KB PDF 举报

身份认证购VIP最低享 7 折!

领优惠券(最高得80元）

资源详情

资源评论

资源推荐

利用利用python的的socket发送发送http(s)请求方法示例请求方法示例

主要给大家介绍了关于利用python的socket发送http(s)请求的相关资料，文中通过示例代码介绍的非常详细，对大家学习或者使用python具有一定的参考学习价值，需要的朋友们下

面来一起看看吧

前言前言

这是个在写计算机网络课设的时候碰到的问题，卡了我一天，所以总结一下。

其实在之前就有用requests写过python爬虫，但是计算机网络要求更底层的实现，刚好我看到了[这篇文章]1结果发现他就是用socket来实现的请求，所以就学习了。

本来也觉得应该不难，毕竟就是建立tcp连接。

原网站的例子如下：原网站的例子如下：

def fetch(url):

sock = socket.socket() # 建立socket

sock.connect(('xkcd.com', 80)) # 远程连接

request = 'GET {} HTTP/1.0Host: xkcd.com'.format(url) # 构建请求

sock.send(request.encode('ascii')) # 向socket发送数据

response = b''

chunk = sock.recv(4096) # 从socket接收数据

while chunk:

response += chunk

chunk = sock.recv(4096)

# Page is now downloaded.

links = parse_links(response)

q.add(links)

我选择爬取的网站是链家，当然也看了很多其他例子,还用fiddler抓包，把headers整个放了上去，首先是参考了这个文章：https://segmentfault.com/a/1190000005126160 ，该文章介绍到：

Python通过socket发送http请求

我们以访问百度主页为案例，使用socket发送http请求。

import socket

s=socket.socket(socket.AF_INET,socket.SOCK_STREAM)

s.connect(('www.baidu.com',80))

s.send('''GET https://www.baidu.com/ HTTP/1.1

Host: www.baidu.com

Connection: keep-alive

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8

Upgrade-Insecure-Requests: 1

User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36

Accept-Language: zh-CN,zh;q=0.8

''')

buf=s.recv(1024)

while len(buf):

print buf

buf = s.recv(1024)

基于socket进行http编程在于请求参数的可控性更好，但是难度相应就更大了。以上发送的数据是直接从fiddler抓包器里面复制的。

基于以上，写出如下代码：

s=socket.socket(socket.AF_INET,socket.SOCK_STREAM)

s.connect(('www.baidu.com',80))

s.send('''GET / HTTP/1.1

Host: zh.lianjia.com

Connection: keep-alive

Cache-Control: max-age=0

Upgrade-Insecure-Requests: 1

User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8

Referer: https://www.baidu.com/link?url=4J5Kx--GLdLFESJhkfRePU8Ac_0agnTcOtB-b3kfnX8VNdZ_6TPqOyJGKVXkTczg&ck=6140.3.83.296.315.287.208.155&shh=www.baidu.com&sht=94886267_hao_pg&wd=&eqid=af98b98700060b77000000065aef0524

Accept-Encoding: gzip, deflate, br

Accept-Language: zh-CN,zh;q=0.9,en-CA;q=0.8,en;q=0.7

Cookie: lianjia_uuid=ce61c41c-25b0-46d6-a0a0-d57a75ee8706; UM_distinctid=1631f588055f9-0286722badd3ec-b34356b-1fa400-1631f58805657f; _ga=GA1.2.43397143.1525239286; _smt_uid=5ae94e02.558be516; _jzqx=1.1525248800.1525335927.1.jzqsr=zh%2Elianjia%2Ecom|jzqct=/ershoufang/xiangzhouqu/.-; _jzqc=1; _jzqckmp=1; _gid=GA1.2.1028411676.1525594529; select_city=440400; all-lj=c60bf575348a3bc08fb27ee73be8c666; _qzjc=1; CNZZDATA1254525948=963210960-1525238218-https%253A%252F%252Fwww.lianjia.com%252F%7C1525608956; CNZZDATA1255633284=1054798284-1525238580-https%253A%252F%252Fwww.lianjia.com%252F%7C1525608969; lianjia_ssid=c046ddb3-3e66-4809-998a-52ade335fdfc; _qzja=1.1070225156.1525239298260.1525603274282.1525613866775.1525609113492.1525613866775.0.0.0.92.9; _qzjto=29.3.0; _jzqa=1.3750161754444366000.1525239284.1525603274.1525613867.9; _jzqy=1.1525239284.1525613867.3.jzqsr=baidu.jzqsr=baidu; Hm_lvt_9152f8221cb6243a53c83b956842be8a=1525607433,1525607626,1525609113,1525613867; Hm_lpvt_9152f8221cb6243a53c83b956842be8a=1525613867; _qzjb=1.1525613866775.1.0.0.0; _jzqb=1.1.10.1525613867.1; CNZZDATA1255604082=964175865-1525237915-https%253A%252F%252Fwww.lianjia.com%252F%7C1525612833

''')

结果总是报400(Bad Request) ，这个地方卡了很久，最后解决办法是一条一条发送，每一条后面加。

sock = socket.socket()

sock.connect(('zh.lianjia.com', 80))

sock.send('GET /ershoufang/ HTTP/1.1'.encode())

sock.send('Host: zh.lianjia.com'.encode())

sock.send('Connection: keep-alive'.encode())

sock.send('Cache-Control: no-cache'.encode())

sock.send('Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8'.encode())

sock.send('Upgrade-Insecure-Requests: 1'.encode())

sock.send('User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.139 Safari/537.36'.encode())

sock.send('Accept-Encoding: gzip, deflate, br'.encode())

sock.send('Cookie: lianjia_uuid=ce61c41c-25b0-46d6-a0a0-d57a75ee8706; UM_distinctid=1631f588055f9-0286722badd3ec-b34356b-1fa400-1631f58805657f; _ga=GA1.2.43397143.1525239286; _smt_uid=5ae94e02.558be516; _jzqx=1.1525248800.1525335927.1.jzqsr=zh%2Elianjia%2Ecom|jzqct=/ershoufang/xiangzhouqu/.-; _jzqc=1; _jzqy=1.1525239284.1525594526.2.jzqsr=baidu.jzqsr=baidu|jzqct=%E9%93%BE%E5%AE%B6; _jzqckmp=1; _gid=GA1.2.1028411676.1525594529; Hm_lvt_9152f8221cb6243a53c83b956842be8a=1525594526,1525594536,1525594804,1525595210; select_city=440400; all-lj=c60bf575348a3bc08fb27ee73be8c666; _qzjc=1; lianjia_ssid=99306d63-8ee5-a53c-a740-2d3021f3db2f; CNZZDATA1255604082=964175865-1525237915-https%253A%252F%252Fwww.lianjia.com%252F%7C1525602095; _jzqa=1.3750161754444366000.1525239284.1525594526.1525603274.8; CNZZDATA1254525948=963210960-1525238218-https%253A%252F%252Fwww.lianjia.com%252F%7C1525603556; CNZZDATA1255633284=1054798284-1525238580-https%253A%252F%252Fwww.lianjia.com%252F%7C1525603557; Hm_lpvt_9152f8221cb6243a53c83b956842be8a=1525606057; _jzqb=1.9.10.1525603274.1; _qzja=1.1070225156.1525239298260.1525597069547.1525603274282.1525605398368.1525606071025.0.0.0.86.8; _qzjb=1.1525603274282.9.0.0.0; _qzjto=23.2.0\r\n\r\n'.encode())