Dask 0.17.5文档：灵活的并行计算库指南

需积分: 20 11 浏览量更新于2024-07-18 收藏 2.55MB PDF 举报

Dask文档，版本0.17.5，由Dask开发团队于2018年5月22日发布，是一个面向分析计算的灵活并行计算库。它由两个主要组件构成：动态任务调度器和大数据集合。 1. **动态任务调度**：Dask的任务调度器专为交互式计算工作负载优化，与Airflow、Luigi、Celery或Make类似，但更加适应实时处理。这种设计允许用户在本地笔记本电脑上轻松扩展到分布式集群，提供高效的工作流管理和执行。 2. **大数据集合支持**：Dask集成了与NumPy和Pandas等库相似的功能，提供了并行数组（parallel arrays）和数据帧（dataframes），使用户能够在内存超出范围或分布式环境中处理大规模数据。这些集合操作底层运行在动态任务调度器之上，确保了在扩展计算能力的同时保持接口的熟悉性。 **优点**： - **易用性**（Familiar）：Dask提供与NumPy和Pandas兼容的对象，使得数据分析人员能够无缝地进行并行化处理，无需对底层实现有深入了解。 - **灵活性**（Flexible）：除了内置的库支持，Dask还提供了一个任务调度接口，允许用户自定义工作负载，并与其他项目进行集成，满足多样化的应用场景需求。 - **纯Python原生性**（Native）：Dask完全基于Python编写，这意味着开发者可以利用丰富的PyData生态系统，包括科学计算库和工具。 - **性能高效**（Fast）：Dask设计上追求低开销和低延迟，确保在处理大规模数据时能保持高性能，减少了额外的运行时间和系统资源消耗。总结来说，Dask文档详细介绍了如何利用这个强大的工具进行并行计算，无论是为了加速数据处理任务还是构建可扩展的数据处理流程。通过结合动态任务调度和大数据集合，Dask成为现代数据分析和机器学习项目中的关键基础设施。用户可以根据其特性选择合适的使用场景，如批处理、实时分析或模型训练，以提升工作效率和性能。

dask Documentation, Release 0.17.5

– Python API (advanced): Create Scheduler and Worker objects from Python as part of a dis-

tributed Tornado TCP application. This page is useful for those building custom frameworks.

– Docker containers are available and may be useful in some of the solutions above.

– Cloud for current recommendations on how to deploy Dask and Jupyter on common cloud providers

like Amazon, Google, or Microsoft Azure.

4.2.1 Single-Machine Scheduler

The default Dask scheduler provides parallelism on a single machine by using either threads or processes. It is the

default choice used by Dask because it requires no setup. You don’t need to make any choices or set anything up to

use this scheduler, however you do have a choice between threads and processes:

1. Threads: Use multiple threads in the same process. This option is good for numeric code that releases the GIL

(like NumPy, Pandas, Scikit-Learn, Numba, . . . ) because data is free to share. This is the default scheduler for

dask.array, dask.dataframe, and dask.delayed

2. Processes: Send data to separate processes for processing. This option is good when operating on pure Python

objects like strings or JSON-like dictionary data that holds onto the GIL but not very good when operating on

numeric data like Pandas dataframes or NumPy arrays. Using processes avoids GIL issues but can also result

in a lot of inter-process communication, which can be slow. This is the default scheduler for dask.bag and is

sometimes useful with dask.dataframe.

Note that the dask.distributed scheduler is often a better choice when working with GIL-bound code. See

Dask.distributed on a single machine.

3. Single-threaded: Execute computations in a single thread. This option provides no parallelism, but is useful

when debugging or proﬁling. Turning your parallel execution into a sequential one can be a convenient option

in many situations where you want to better understand what is going on.

Selecting Threads, Processes, or Single Threaded

Currently these options are available by selecting different get functions:

• dask.threaded.get: The threaded scheduler

• dask.multiprocessing.get: The multiprocessing scheduler

• dask.local.get_sync: The single-threaded scheduler

You can specify these functions in any of the following ways:

• When calling .compute()

x.compute(scheduler='threads')

• With a context manager

with dask.config.set(scheduler='threads'):

x.compute()

y.compute()

• As a global setting

dask.config.set(scheduler='threads')

12 Chapter 4. Index

dask Documentation, Release 0.17.5

Use the Distributed Scheduler

The newer dask.distributed scheduler also works well on a single machine and offers more features and diagnostics.

See this page for more information.

4.2.2 Single Machine: Dask.distributed

The dask.distributed scheduler works well on a single machine. It is sometimes preferred over the default scheduler

for the following reasons:

1. It provides access to asynchronous API, notably Futures

2. It provides a diagnostic dashboard that can provide valuable insight on performance and progress

3. It handles data locality with more sophistication, and so can be more efﬁcient than the multiprocessing scheduler

on workloads that require multiple processes.

You can create a dask.distributed scheduler by importing and creating a Client with no arguments. This overrides

whatever default was previously set.

from dask.distributed import Client

client = Client()

You can navigate to http://localhost:8787/status to see the diagnostic dashboard if you have Bokeh installed.

Client

You can trivially set up a local cluster on your machine by instantiating a Dask Client with no arguments

from dask.distributed import Client

client = Client()

This sets up a scheduler in your local process and several processes running single-threaded Workers.

If you want to run workers in your same process you can pass the processes=False keyword argument.

client = Client(processes=False)

This is sometimes preferable if you want to avoid inter-worker communication and your computations release the GIL.

This is common when primarily using NumPy or Dask.array.

LocalCluster

The Client() call described above is shorthand for creating a LocalCluster and then passing that to your client.

from dask.distributed import Client, LocalCluster

cluster = LocalCluster()

client = Client(cluster)

This is equivalent, but somewhat more explicit. You may want to look at the keyword arguments available on Local-

Cluster to understand the options available to you on handling the mixture of threads and processes, specifying explicit

ports, and so on.

4.2. Setup 13

dask Documentation, Release 0.17.5

Then launch dask-worker on the rest of the nodes, providing the address to the node that hosts

dask-scheduler:

$ dask-worker tcp://192.0.0.100:8786

Start worker at: tcp://192.0.0.1:12345

Registered to: tcp://192.0.0.100:8786

$ dask-worker tcp://192.0.0.100:8786

Start worker at: tcp://192.0.0.2:40483

Registered to: tcp://192.0.0.100:8786

$ dask-worker tcp://192.0.0.100:8786

Start worker at: tcp://192.0.0.3:27372

Registered to: tcp://192.0.0.100:8786

The workers connect to the scheduler, which then sets up a long-running network connection back to the worker. The

workers will learn the location of other workers from the scheduler.

Handling Ports

The scheduler and workers both need to accept TCP connections on an open port. By default the scheduler binds

to port 8786 and the worker binds to a random open port. If you are behind a ﬁrewall then you may have to open

particular ports or tell Dask to listen on particular ports with the --port and --worker-port keywords.:

dask-scheduler --port 8000

dask-worker --bokeh-port 8000 --nanny-port 8001

Nanny Processes

Dask workers are run within a Nanny process that monitors the worker process and restarts it if necessary.

Diagnostic Web Servers

Additionally Dask schedulers and workers host interactive diagnostic web servers using Bokeh. These are optional,

but generally useful to users. The diagnostic server on the scheduler is particularly valuable, and is served on port

8787 by default (conﬁgurable with the --bokeh-port keyword).

• More information about relevant ports is available by looking at the help pages with dask-scheduler

--help and dask-worker --help

Automated Tools

There are various mechanisms to deploy these executables on a cluster, ranging from manually SSH-ing into all of the

machines to more automated systems like SGE/SLURM/Torque or Yarn/Mesos. Additionally, cluster SSH tools exist

to send the same commands to many machines. We recommend searching online for “cluster ssh” or “cssh”..

API

These may be out-dated. We recommend referring to the --help text of your installed version.

16 Chapter 4. Index

剩余720页未读，继续阅读

escaiyi

粉丝: 1
资源: 9

Dask 0.17.5文档：灵活的并行计算库指南

Python库 | dask_cuda-21.6.0-py3-none-any.whl

dask-ml:使用Dask进行可扩展的机器学习

AutomationDesk中文操作文档

Firibird Documentation

NTFS Documentation

Linux boot time documentation

Turbo C 2.00[DISK]

Turbo C++ 3.00[DISK]

Turbo C 2.01[DISK]

Floppy Disk Operating System-开源

最新资源