没有合适的资源?快使用搜索试试~ 我知道了~
首页Celery 4.1.0官方文档:入门与特性概述
Celery 4.1.0官方文档:入门与特性概述
需积分: 10 3 下载量 48 浏览量
更新于2024-07-18
收藏 2.63MB PDF 举报
Celery 4.1.0 是一个功能强大且灵活的分布式任务队列和工作流管理系统,专注于实时处理和任务调度。它设计用于处理大量消息,同时提供必要的工具来维护这样一个系统。作为一款开源软件,Celery基于 BSD 许可证,这意味着它的源代码是公开可用并鼓励社区参与。 对于初学者而言,Celery 4.1.0文档提供了详细的入门指南,包括“First Steps with Celery”教程,这是一条清晰的路径,帮助新用户快速了解如何设置、配置和使用Celery。教程覆盖了从安装、配置到创建和管理任务的基本步骤,确保用户能够轻松上手。 此外,文档还包含了常见问题解答(FAQ),这是解决初次使用过程中遇到问题的重要资源。FAQ通常涵盖了各种常见的问题和解决方案,可以帮助用户避免许多初期陷阱,并加速学习过程。 整个文档结构清晰,分为两个主要章节:第一章“Getting Started”是入门章节,适合新手入门,而第二章则可能涵盖了更深入的内容,如版权信息、Celery User Manual 的版权声明以及所有者AskSolem对材料使用的权利限制。Python模块索引和参考文献列表也提供了进一步的技术支持和学术支持,使得用户在遇到特定功能或技术细节时能迅速找到所需的信息。 Celery 4.1.0文档旨在为用户提供全面的指导,无论是从零开始的用户还是经验丰富的开发人员,都能从中获取到有价值的资源。通过这个版本,开发者可以构建高效、可靠的分布式任务处理系统,同时享受到活跃的社区支持和持续的更新维护。
资源详情
资源推荐
![](https://csdnimg.cn/release/download_crawler_static/10413808/bg10.jpg)
Celery Documentation, Release 4.1.0
With git
Please the Contributing section.
Brokers
Release 4.1
Date Jul 24, 2017
Celery supports several message transport alternatives.
Broker Instructions
Using RabbitMQ
• Installation & Configuration
• Installing the RabbitMQ Server
– Setting up RabbitMQ
– Installing RabbitMQ on macOS
*
Configuring the system host name
*
Starting/Stopping the RabbitMQ server
Installation & Configuration
RabbitMQ is the default broker so it doesn’t require any additional dependencies or initial configuration, other than
the URL location of the broker instance you want to use:
broker_url = 'amqp://myuser:mypassword@localhost:5672/myvhost'
For a description of broker URLs and a full list of the various broker configuration options available to Celery, see
Broker Settings, and see below for setting up the username, password and vhost.
Installing the RabbitMQ Server
See Installing RabbitMQ over at RabbitMQ’s website. For macOS see Installing RabbitMQ on macOS.
Note: If you’re getting nodedown errors after installing and using rabbitmqctl then this blog post can help you
identify the source of the problem:
http://www.somic.org/2009/02/19/on-rabbitmqctl-and-badrpcnodedown/
12 Chapter 2. Contents
![](https://csdnimg.cn/release/download_crawler_static/10413808/bg11.jpg)
Celery Documentation, Release 4.1.0
Setting up RabbitMQ
To use Celery we need to create a RabbitMQ user, a virtual host and allow that user access to that virtual host:
$ sudo rabbitmqctl add_user myuser mypassword
$ sudo rabbitmqctl add_vhost myvhost
$ sudo rabbitmqctl set_user_tags myuser mytag
$ sudo rabbitmqctl set_permissions -p myvhost myuser ".
*
" ".
*
" ".
*
"
Substitute in appropriate values for myuser, mypassword and myvhost above.
See the RabbitMQ Admin Guide for more information about access control.
Installing RabbitMQ on macOS
The easiest way to install RabbitMQ on macOS is using Homebrew the new and shiny package management system
for macOS.
First, install Homebrew using the one-line command provided by the Homebrew documentation:
ruby -e "$(curl -fsSL https://raw.github.com/Homebrew/homebrew/go/install)"
Finally, we can install RabbitMQ using brew:
$ brew install rabbitmq
After you’ve installed RabbitMQ with brew you need to add the following to your path to be able to start and stop
the broker: add it to the start-up file for your shell (e.g., .bash_profile or .profile).
PATH=$PATH:/usr/local/sbin
Configuring the system host name
If you’re using a DHCP server that’s giving you a random host name, you need to permanently configure the host
name. This is because RabbitMQ uses the host name to communicate with nodes.
Use the scutil command to permanently set your host name:
$ sudo scutil --set HostName myhost.local
Then add that host name to /etc/hosts so it’s possible to resolve it back into an IP address:
127.0.0.1 localhost myhost myhost.local
If you start the rabbitmq-server, your rabbit node should now be rabbit@myhost, as verified by rabbitmqctl:
$ sudo rabbitmqctl status
Status of node rabbit@myhost ...
[{running_applications,[{rabbit,"RabbitMQ","1.7.1"},
{mnesia,"MNESIA CXC 138 12","4.4.12"},
{os_mon,"CPO CXC 138 46","2.2.4"},
2.2. Getting Started 13
![](https://csdnimg.cn/release/download_crawler_static/10413808/bg12.jpg)
Celery Documentation, Release 4.1.0
{sasl,"SASL CXC 138 11","2.1.8"},
{stdlib,"ERTS CXC 138 10","1.16.4"},
{kernel,"ERTS CXC 138 10","2.13.4"}]},
{nodes,[rabbit@myhost]},
{running_nodes,[rabbit@myhost]}]
...done.
This is especially important if your DHCP server gives you a host name starting with an IP address, (e.g.,
23.10.112.31.comcast.net). In this case RabbitMQ will try to use rabbit@23: an illegal host name.
Starting/Stopping the RabbitMQ server
To start the server:
$ sudo rabbitmq-server
you can also run it in the background by adding the -detached option (note: only one dash):
$ sudo rabbitmq-server -detached
Never use kill (kill(1)) to stop the RabbitMQ server, but rather use the rabbitmqctl command:
$ sudo rabbitmqctl stop
When the server is running, you can continue reading Setting up RabbitMQ.
Using Redis
Installation
For the Redis support you have to install additional dependencies. You can install both Celery and these dependencies
in one go using the celery[redis] bundle:
$ pip install -U "celery[redis]"
Configuration
Configuration is easy, just configure the location of your Redis database:
app.conf.broker_url = 'redis://localhost:6379/0'
Where the URL is in the format of:
redis://:password@hostname:port/db_number
all fields after the scheme are optional, and will default to localhost on port 6379, using database 0.
If a Unix socket connection should be used, the URL needs to be in the format:
redis+socket:///path/to/redis.sock
Specifying a different database number when using a Unix socket is possible by adding the virtual_host param-
eter to the URL:
14 Chapter 2. Contents
![](https://csdnimg.cn/release/download_crawler_static/10413808/bg13.jpg)
Celery Documentation, Release 4.1.0
redis+socket:///path/to/redis.sock?virtual_host=db_number
Visibility Timeout
The visibility timeout defines the number of seconds to wait for the worker to acknowledge the task before the message
is redelivered to another worker. Be sure to see Caveats below.
This option is set via the broker_transport_options setting:
app.conf.broker_transport_options = {'visibility_timeout': 3600} # 1 hour.
The default visibility timeout for Redis is 1 hour.
Results
If you also want to store the state and return values of tasks in Redis, you should configure these settings:
app.conf.result_backend = 'redis://localhost:6379/0'
For a complete list of options supported by the Redis result backend, see Redis backend settings
Caveats
Fanout prefix
Broadcast messages will be seen by all virtual hosts by default.
You have to set a transport option to prefix the messages so that they will only be received by the active virtual host:
app.conf.broker_transport_options = {'fanout_prefix': True}
Note that you won’t be able to communicate with workers running older versions or workers that doesn’t have this
setting enabled.
This setting will be the default in the future, so better to migrate sooner rather than later.
Fanout patterns
Workers will receive all task related events by default.
To avoid this you must set the fanout_patterns fanout option so that the workers may only subscribe to worker
related events:
app.conf.broker_transport_options = {'fanout_patterns': True}
Note that this change is backward incompatible so all workers in the cluster must have this option enabled, or else they
won’t be able to communicate.
This option will be enabled by default in the future.
2.2. Getting Started 15
![](https://csdnimg.cn/release/download_crawler_static/10413808/bg14.jpg)
Celery Documentation, Release 4.1.0
Visibility timeout
If a task isn’t acknowledged within the Visibility Timeout the task will be redelivered to another worker and executed.
This causes problems with ETA/countdown/retry tasks where the time to execute exceeds the visibility timeout; in fact
if that happens it will be executed again, and again in a loop.
So you have to increase the visibility timeout to match the time of the longest ETA you’re planning to use.
Note that Celery will redeliver messages at worker shutdown, so having a long visibility timeout will only delay the
redelivery of ‘lost’ tasks in the event of a power failure or forcefully terminated workers.
Periodic tasks won’t be affected by the visibility timeout, as this is a concept separate from ETA/countdown.
You can increase this timeout by configuring a transport option with the same name:
app.conf.broker_transport_options = {'visibility_timeout': 43200}
The value must be an int describing the number of seconds.
Key eviction
Redis may evict keys from the database in some situations
If you experience an error like:
InconsistencyError: Probably the key ('_kombu.binding.celery') has been
removed from the Redis database.
then you may want to configure the redis-server to not evict keys by setting the timeout parameter to 0 in the
redis configuration file.
Using Amazon SQS
Installation
For the Amazon SQS support you have to install the boto library using pip:
$ pip install -U boto
Configuration
You have to specify SQS in the broker URL:
broker_url = 'sqs://ABCDEFGHIJKLMNOPQRST:ZYXK7NiynGlTogH8Nj+P9nlE73sq3@'
where the URL format is:
sqs://aws_access_key_id:aws_secret_access_key@
you must remember to include the “@” at the end.
The login credentials can also be set using the environment variables AWS_ACCESS_KEY_ID and
AWS_SECRET_ACCESS_KEY, in that case the broker URL may only be sqs://.
16 Chapter 2. Contents
剩余713页未读,继续阅读
![pdf](https://img-home.csdnimg.cn/images/20210720083512.png)
![pdf](https://img-home.csdnimg.cn/images/20210720083512.png)
![pdf](https://img-home.csdnimg.cn/images/20210720083512.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://csdnimg.cn/download_wenku/file_type_ask_c1.png)
![](https://profile-avatar.csdnimg.cn/default.jpg!1)
perfectionhello
- 粉丝: 0
- 资源: 2
上传资源 快速赚钱
我的内容管理 展开
我的资源 快来上传第一个资源
我的收益
登录查看自己的收益我的积分 登录查看自己的积分
我的C币 登录后查看C币余额
我的收藏
我的下载
下载帮助
![](https://csdnimg.cn/release/wenkucmsfe/public/img/voice.245cc511.png)
最新资源
- Vue实现iOS原生Picker组件:详细解析与实现思路
- Arduino蓝牙小车:参数调试与功能控制
- 百度Java面试精华:200页精选资源涵盖核心知识点
- Swift使用CoreData填坑指南:CoreData在Swift 3.0的变化
- 微距离无线充电器创新设计及其实验探索
- MTK Android平台开发全攻略:44步详解流程
- RecyclerView全面解析:替代ListView的新选择
- Android开发:自动适配中英文键盘解决方案
- Android调用WebService接口教程
- Android开发:BitmapUtil图片处理全解析与实例
- Android多线程断点续传实现详解
- PCA算法在人脸识别会议签到系统中的应用
- EventBus 3.0:Android事件总线详解与实战应用
- Android FileUtil:全面解析文件操作实用技巧与实例
- RecyclerView添加头部和尾部实战教程
- Android实现微博滑动固定顶部栏实战与优化
资源上传下载、课程学习等过程中有任何疑问或建议,欢迎提出宝贵意见哦~我们会及时处理!
点击此处反馈
![](https://img-home.csdnimg.cn/images/20220527035711.png)
![](https://img-home.csdnimg.cn/images/20220527035711.png)
![](https://img-home.csdnimg.cn/images/20220527035111.png)
安全验证
文档复制为VIP权益,开通VIP直接复制
![](https://csdnimg.cn/release/wenkucmsfe/public/img/green-success.6a4acb44.png)