可以帮我用Python写一个生产者消费者模式的多线程下载程序吗?
时间: 2023-05-18 21:03:41 浏览: 200
当然可以,以下是一个简单的生产者消费者模式的多线程下载程序的Python代码:
```python
import threading
import queue
import requests
class Downloader:
def __init__(self, url_queue, result_queue):
self.url_queue = url_queue
self.result_queue = result_queue
def download(self):
while True:
url = self.url_queue.get()
try:
response = requests.get(url)
self.result_queue.put(response.content)
except Exception as e:
print(f"Error downloading {url}: {e}")
finally:
self.url_queue.task_done()
class Producer:
def __init__(self, url_queue, urls):
self.url_queue = url_queue
self.urls = urls
def produce(self):
for url in self.urls:
self.url_queue.put(url)
def main():
urls = [
"https://www.example.com",
"https://www.google.com",
"https://www.python.org",
"https://www.github.com",
"https://www.stackoverflow.com"
]
url_queue = queue.Queue()
result_queue = queue.Queue()
downloader = Downloader(url_queue, result_queue)
downloader_threads = [threading.Thread(target=downloader.download) for _ in range(4)]
producer = Producer(url_queue, urls)
producer_thread = threading.Thread(target=producer.produce)
producer_thread.start()
for downloader_thread in downloader_threads:
downloader_thread.start()
url_queue.join()
while not result_queue.empty():
result = result_queue.get()
print(len(result))
if __name__ == "__main__":
main()
```
这个程序使用了Python的queue模块来实现生产者消费者模式,其中Downloader类是消费者,Producer类是生产者。程序首先创建了一个包含多个URL的列表,然后创建了一个URL队列和一个结果队列。接着创建了一个Downloader实例和多个Downloader线程,以及一个Producer实例和一个Producer线程。Producer线程将URLs放入URL队列中,Downloader线程从URL队列中取出URL并下载对应的内容,将结果放入结果队列中。最后程序等待URL队列中的所有URL都被处理完毕,然后从结果队列中取出结果并打印结果的长度。
阅读全文