fetch设置referer示例
时间: 2023-07-22 22:06:17 浏览: 249
要在 Fetch API 中设置 Referer,你可以使用 Headers 对象来添加自定义的请求头信息。以下是一个示例:
```javascript
function loadFile(url) {
var headers = new Headers();
headers.append('Referer', 'http://example.com');
return fetch(url, {
headers: headers
})
.then(response => response.arrayBuffer())
.then(arrayBuffer => {
// 在这里处理 ArrayBuffer
console.log(arrayBuffer);
});
}
var m3u8Url = 'path/to/your/m3u8.ts';
loadFile(m3u8Url);
```
在这个示例中,我们创建了一个 Headers 对象,并使用 `append` 方法添加了一个名为 "Referer" 的自定义请求头,将其值设置为你想要的 Referer URL。然后,在 Fetch 请求的配置对象中,通过 `headers` 属性将自定义的 Headers 对象传递给 Fetch API。
这样,在发送 Fetch 请求时,会携带自定义的 Referer 请求头信息。请确保将 "http://example.com" 替换为你实际需要设置的 Referer URL。
相关问题
import requests import os from bs4 import BeautifulSoup cookie = os.getenv('MY_COOKIE') # 保存到环境变量中 def get_with_cookie(url): headers = { 'cookie': cookie, 'pragma': 'no-cache', 'referer': 'https://uland.taobao.com/sem/tbsearch?refpid=mm_26632258_3504122_32538762&keyword=%E7%88%B1%E6%B7%98%E5%AE%9D%E4%B9%B0%E4%B8%9C%E8%A5%BF&clk1=066356e5e429004e6730976351cc1afc&upsId=066356e5e429004e6730976351cc1afc', 'sec-ch-ua': '"Microsoft Edge";v="113", "Chromium";v="113", "Not-A.Brand";v="24"', 'sec-ch-ua-mobile': '?0', 'sec-ch-ua-platform': "Windows", 'sec-fetch-dest': 'document', 'sec-fetch-mode': 'navigate', 'sec-fetch-site': 'same-origin', 'sec-fetch-user': '?1', 'upgrade-insecure-requests': '1', 'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36 Edg/113.0.1774.50' } try: resp = requests.get(url=url, headers=headers) return resp.text except requests.exceptions.RequestException as e: print("请求异常:", e) return None # 解析HTML内容,提取商品名称、价格和销量信息 def parse_html(html): try: soup = BeautifulSoup(html, 'html.parser') title = soup.select_one('h3.tb-main-title[data-title]') return title except Exception as e: print("解析异常:", e) return None # 测试接口: url = 'https://item.taobao.com/item.htm?id=699089474285&ali_refid=a3_430673_1006:1102522391:N:%2BAkz14GC%2BXi%2Bma9dw7y9M4p5Yhq5iT8I:f9e826228fbc823934848d20e28dff81&ali_trackid=162_f9e826228fbc823934848d20e28dff81&spm=a2e0b.20350158.31919782.5' resp_text = get_with_cookie(url) if resp_text is not None: title = parse_html(resp_text) if title is not None: # 打印商品信息 print(title.text.strip()) else: print("解析失败") else: print("请求失败")将cookie存储在session中
可以使用requests库中的Session类来存储cookie,示例代码如下:
```python
import requests
from bs4 import BeautifulSoup
session = requests.Session()
cookie = 'your_cookie_here'
session.headers.update({'cookie': cookie})
def get_with_cookie(url):
try:
resp = session.get(url)
return resp.text
except requests.exceptions.RequestException as e:
print("请求异常:", e)
return None
# 解析HTML内容,提取商品名称、价格和销量信息
def parse_html(html):
try:
soup = BeautifulSoup(html, 'html.parser')
title = soup.select_one('h3.tb-main-title[data-title]')
return title
except Exception as e:
print("解析异常:", e)
return None
# 测试接口:
url = 'https://item.taobao.com/item.htm?id=699089474285&ali_refid=a3_430673_1006:1102522391:N:%2BAkz14GC%2BXi%2Bma9dw7y9M4p5Yhq5iT8I:f9e826228fbc823934848d20e28dff81&ali_trackid=162_f9e826228fbc823934848d20e28dff81&spm=a2e0b.20350158.31919782.5'
resp_text = get_with_cookie(url)
if resp_text is not None:
title = parse_html(resp_text)
if title is not None:
# 打印商品信息
print(title.text.strip())
else:
print("解析失败")
else:
print("请求失败")
```
在这个示例代码中,我们使用了requests.Session()来创建一个session对象,并将cookie存储在session对象的headers中。在请求时,我们直接调用session.get()方法即可实现带cookie请求。这样,我们就可以不用在每次请求时都带上cookie了。
VBA写一个爬虫其中包含Host,Referer,User-Agent,Cookie请求头
以下是使用VBA编写基本的爬虫代码,并包含了Host、Referer、User-Agent和Cookie请求头:
```
Sub webScraping()
Dim xmlHttp As Object
Dim htmlDoc As Object
Dim url As String
Dim headers As String
' 设置请求URL和请求头信息
url = "https://example.com"
headers = "Host: example.com" & vbCrLf & _
"Referer: https://www.google.com/" & vbCrLf & _
"User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3" & vbCrLf & _
"Cookie: SESSIONID=1234567890abcdef"
' 创建XMLHttpRequest对象
Set xmlHttp = CreateObject("MSXML2.XMLHTTP")
' 发送GET请求
xmlHttp.Open "GET", url, False
xmlHttp.setRequestHeader "Content-Type", "text/plain;charset=UTF-8"
xmlHttp.setRequestHeader "Connection", "keep-alive"
xmlHttp.setRequestHeader "Accept-Language", "en-US,en;q=0.9"
xmlHttp.setRequestHeader "Accept-Encoding", "gzip, deflate, br"
xmlHttp.setRequestHeader "Cache-Control", "max-age=0"
xmlHttp.setRequestHeader "Upgrade-Insecure-Requests", "1"
xmlHttp.setRequestHeader "Pragma", "no-cache"
xmlHttp.setRequestHeader "DNT", "1"
xmlHttp.setRequestHeader "Sec-Fetch-Site", "none"
xmlHttp.setRequestHeader "Sec-Fetch-Mode", "navigate"
xmlHttp.setRequestHeader "Sec-Fetch-User", "?1"
xmlHttp.setRequestHeader "Sec-Fetch-Dest", "document"
xmlHttp.setRequestHeader "Sec-Ch-Ua", """Google Chrome"";v=""93"", "" Not;A Brand"";v=""99"", ""Chromium"";v=""93"""
xmlHttp.setRequestHeader "Sec-Ch-Ua-Mobile", "?0"
' 添加自定义请求头
xmlHttp.setRequestHeader "Host", "example.com"
xmlHttp.setRequestHeader "Referer", "https://www.google.com/"
xmlHttp.setRequestHeader "User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3"
xmlHttp.setRequestHeader "Cookie", "SESSIONID=1234567890abcdef"
' 发送请求并获取响应
xmlHttp.send
' 解析响应内容
Set htmlDoc = CreateObject("HTMLfile")
htmlDoc.body.innerHTML = xmlHttp.responseText
' 输出响应结果
Debug.Print htmlDoc.body.innerHTML
End Sub
```
请注意,上述示例代码仅供参考,并且具体的请求头信息需要根据实际情况进行调整。同时,还需要注意网站是否允许爬虫访问,否则可能会触发反爬虫机制。
阅读全文