python监听提交到yarn任务的状态和进度,并实时写入redis
时间: 2023-05-18 22:03:22 浏览: 114
可以使用Python的yarn-api-client库来监听提交到yarn任务的状态和进度,并实时写入redis。以下是一个示例代码:
```python
from yarn_api_client import ApplicationMaster, HistoryServer
import redis
# Connect to Redis
r = redis.Redis(host='localhost', port=6379, db=0)
# Connect to YARN Application Master
am = ApplicationMaster('http://yarn-app-master:8088')
# Get list of running applications
apps = am.cluster_applications(state='RUNNING')
# Loop through running applications
for app in apps:
# Get application ID
app_id = app['id']
# Get application state and progress
state = am.application_state(app_id)
progress = am.application_progress(app_id)
# Write state and progress to Redis
r.set(app_id + ':state', state)
r.set(app_id + ':progress', progress)
```
这段代码会连接到Redis,并使用yarn-api-client库连接到YARN Application Master。然后,它会获取正在运行的应用程序列表,并循环遍历每个应用程序。对于每个应用程序,它会获取应用程序的ID、状态和进度,并将它们写入Redis。
阅读全文