同时爬取五个函数并输出html
时间: 2024-09-16 12:04:18 浏览: 34
在Python中,如果你需要同时爬取五个不同的URL并输出它们的HTML内容,可以使用线程池或者协程(asyncio)来提高效率。下面是一个使用`concurrent.futures`模块(线程池)的例子:
```python
import concurrent.futures
import requests
from bs4 import BeautifulSoup
def fetch_html(url):
try:
response = requests.get(url)
if response.status_code == 200:
return response.text
else:
print(f"Failed to fetch {url}, status code: {response.status_code}")
return None
except Exception as e:
print(f"Error occurred while fetching {url}: {str(e)}")
return None
urls = ["http://example1.com", "http://example2.com", "http://example3.com",
"http://example4.com", "http://example5.com"]
with concurrent.futures.ThreadPoolExecutor() as executor:
future_to_url = {executor.submit(fetch_html, url): url for url in urls}
for future in concurrent.futures.as_completed(future_to_url):
url = future_to_url[future]
html = future.result()
if html is not None:
print(f"Succeeded in fetching {url}:")
print(html)
阅读全文