Python分布式计算实战指南

需积分: 10 35 浏览量更新于2024-07-17 收藏 7.67MB PDF 举报

"本书《Distributed Computing With Python》是一本快速、信息丰富的指南，由Francesco Pierfederici撰写，旨在教你如何利用Python构建分布式系统。书中详细讲解了使用Python进行分布式计算的方法和技术，适合对分布式系统感兴趣的Python开发者阅读。" 在分布式计算的世界里，Python作为一种强大的编程语言，因其易读性、丰富的库支持和庞大的社区而被广泛用于构建大规模、高性能的系统。通过学习《Distributed Computing With Python》，读者可以了解到以下关键知识点： 1. **分布式系统基础**：了解分布式系统的基本概念，包括节点、通信协议、数据分布、负载均衡、容错机制等，这些都是构建分布式系统的基础。 2. **Python分布式计算框架**：探讨Python中的各种分布式计算框架，如Apache Hadoop、Apache Spark、Celery、Dask等，以及它们在处理大规模数据时的优势和应用场景。 3. **网络通信与并发**：学习Python的网络编程，包括socket通信、HTTP协议、异步I/O（如asyncio库）等，这些是实现多节点间通信的关键技术。 4. **数据分布与存储**：研究如何在分布式环境中存储和管理数据，如使用分布式数据库（如Cassandra、HBase）或键值存储（如Redis）等。 5. **任务调度与协调**：掌握如何使用任务调度器（如Kubernetes、YARN）来管理和协调分布式系统中的任务，以及如何使用工具（如Zookeeper、Etcd）实现服务发现和配置管理。 6. **错误处理与容错**：学习如何设计和实现具有弹性的系统，包括故障检测、故障恢复策略、数据复制和备份，以及如何利用Python的异常处理机制来提高系统的健壮性。 7. **性能优化**：深入理解分布式系统性能优化技巧，包括并行计算、数据压缩、缓存策略、负载预测等，以提升系统的整体效率。 8. **案例研究**：通过具体的项目实例，例如大数据处理、实时流处理、机器学习分布式训练等，展示如何将理论知识应用到实际项目中。 9. **最佳实践**：获取作者Francesco Pierfederici在构建分布式系统方面的实践经验，包括设计模式、工具选择、调试技巧等，帮助读者避免常见陷阱，提升开发效率。 10. **持续集成与部署**：了解如何在分布式环境中进行持续集成和持续部署（CI/CD），使用Jenkins、Docker等工具来确保代码质量和系统稳定性。通过学习这本书，读者将具备使用Python构建和管理分布式系统的能力，能够应对日益复杂的数据处理需求，提升软件系统的可扩展性和性能。

[ 1 ]

An Introduction to Parallel

and Distributed Computing

The rst modern digital computer was invented in the late 30s and early 40s (that is,

arguably, the Z1 from Konrad Zuse in 1936), probably before most of the readers of

this book—let alone the author—were born. These last seventy odd years have seen

computers become faster and cheaper at an amazing rate, which was unique across

industries. Just think that today's smartphones (for example, the latest iPhones or

Android phones) are faster than the fastest computer in the world from just 20 years

ago. Not to mention, the amazing feat of miniaturization: those supercomputers used

to take up entire rooms; now they t in our pockets.

These years have also seen, among others, two key inventions relevant to the topic

at hand. One is the ability to cram more than one processor on a single motherboard

(and even multiple CPU cores on a single processor). This development was crucial in

allowing computations to be performed truly concurrently. As we know, processors

are able to perform only one task at a time; however, as we will see later on in the

chapter, they are fast enough to give the illusion of being able to run multiple tasks at

the same time. To be able to perform more than one action exactly at the same time,

you need access to more than one processor.

The other critical invention is high-speed computer networking. This allowed, for the

rst time, a potentially enormous number of computers to communicate with each

other. These networked machines can either be located in the same ofce/building

(the so-called Local Area Network (LAN)) or be spread out across different buildings,

cities, or even across the planet (that is, WAN or wide area networking).

An Introduction to Parallel and Distributed Computing

[ 2 ]

By now, most of us are familiar with multiprocessor/multicore computers, and

indeed, the chances are pretty high that the phone in our pocket, the tablet in our

hands, or the laptop we take on the road has a handful of cores already. The graphics

card, also called Graphics Processing Unit (GPU) in these devices is more often

than not massively parallel, with hundreds or even thousands of processing units.

Computer networks too are all around us, starting from the most famous of them all:

the Internet, to the Wi-Fi in our homes and coffee shops and the 4G mobile networks

our phones use.

In the rest of this chapter, we will lay some working denitions of the topics that we

will explore in the rest of the book. We will be introducing the concepts of parallel

and distributed computing. We will give some examples of each that are taken from

familiar topics and technologies. Some general advantages and disadvantages of

each architecture and programming paradigm will be discussed as well.

Before proceeding with our denitions and a little bit of theory, let's clarify a possible

source of confusion. In this and the following chapters, we will use the term processor

and the term CPU core (or even simply core) interchangeably, unless otherwise

specied. This is, of course, technically incorrect; a processor has one or more

cores, and a computer has one or more processors as cores do not exist in isolation.

Depending on the algorithm and its performance requirements, running on multiple

processors or on a single processor using multiple cores can make a bit of difference in

speed, assuming, of course, that the algorithm can be parallelized in the rst place. For

our intents and purposes, however, we will ignore these differences and refer to more

advanced texts for further exploration of this topic.

Parallel computing

Denitions of parallel computing abound. However, for the purpose of this book,

a simple denition will sufce, which is as follows:

Parallel computing is the simultaneous use of more than one processor to solve

a problem.

Typically, this denition is further specialized by requiring that the processors reside

on the same motherboard. This is mostly to distinguish parallel computing from

distributed computing (which is discussed in the next section).

The idea of splitting work among many workers is as old as human civilization, is

not restricted to the digital world, and nds an immediate and obvious application

in modern computers equipped with higher and higher numbers of compute units.

Chapter 1

[ 3 ]

There are, of course, many reasons why parallel computing might be useful and

even necessary. The simplest one is performance; if we can indeed break up a

long-running computation into smaller chunks and parcel them out to different

processors, then we can do more work in the same amount of time.

Other times, and just as often, parallel computing techniques are used to present

users with responsive interfaces while the system is busy with some other task.

Remember that one processor executes just one task at the time. Applications with

GUIs need to ofoad work to a separate thread of execution running on another

processor so that one processor is free to update the GUI and respond to user inputs.

The following gure illustrates this common architecture, where the main thread

is processing user and system inputs using what is called an event loop. Tasks that

require a long time to execute and those that would otherwise block the GUI are

ofoaded to a background or worker thread:

Main Thread Worker Thread

Long-running Task

Event Loop

GUI

A simple real-world example of this parallel architecture could be a photo

organization application. When we connect a digital camera or a smartphone to our

computers, the photo application needs to perform a number of actions; all the while

its user interface needs to stay interactive. For instance, our application needs to copy

images from the device to the internal disk, create thumbnails, extract metadata (for

example, date and time of the shot), index the images, and nally update the image

gallery. While all of this happens, we are still able to browse images that are already

imported, open them, edit them, and so on.

Of course, all these actions could very well be performed sequentially on a single

processor—the same processor that is handling the GUI. The drawback would be a

sluggish interface and an extremely slow overall application. Performing these steps

in parallel keeps the application snappy and its users happy.

An Introduction to Parallel and Distributed Computing

[ 4 ]

The astute reader might jump up at this point and rightfully point out that older

computers, with a single processor and a single core, could already perform multiple

things at the same time (by way of multitasking). What happened back then (and

even today, when we launch more tasks than there are processors and cores on our

computers) was that the one running task gave up the CPU (either voluntarily or

forcibly by the OS, for example, in response to an IO event) so that another task could

run in its place. These interrupts would happen over and over again, with various

tasks acquiring and giving up the CPU many times over the course of the application's

life. In those cases, users had the impression of multiple tasks running concurrently, as

the switches were extremely fast. In reality, however, only one task was running at any

given time.

The typical tools used in parallel applications are threads. On systems such as

Python (as we will see in Chapter 3, Parallelism in Python) where threads have

signicant limitations, programmers resort to launching (oftentimes, by means of

forking) subprocesses instead. These subprocesses replace (or complement) threads

and run alongside the main application process.

The rst technique is called multithreaded programming. The second is called

multiprocessing. It is worth noting that multiprocessing should not be seen as

inferior or as a workaround with respect to using multiple threads.

There are many situations where multiprocessing is preferable to multiple threads.

Interestingly, even though they both run on a single computer, a multithreaded

application is an example of shared-memory architecture, whereas a multiprocess

application is an example of distributed memory architecture (refer to the following

section to know more).

Distributed computing

For the remainder of this book, we will adopt the following working denition of

distributed computing:

Distributed computing is the simultaneous use of more than one computer to solve

a problem.

Typically, as in the case of parallel computing, this denition is oftentimes further

restricted. The restriction usually is the requirement that these computers appear

to their users as a single machine, therefore hiding the distributed nature of the

application. In this book, we will be happy with the more general denition.

Chapter 1

[ 5 ]

Distributing computation across multiple computers is again a pretty obvious

strategy when using systems that are able to speak to each other over the (local or

otherwise) network. In many respects, in fact, this is just a generalization of the

concepts of parallel computing that we saw in the previous section.

Reasons to build distributed systems abound. Oftentimes, the reason is the ability

to tackle a problem so big that no individual computer could handle it at all, or at

least, not in a reasonable amount of time. An interesting example from a eld that

is probably familiar to most of us is the rendering of 3D animation movies, such as

those from Pixar and DreamWorks.

Given the sheer number of frames to render for a full-length feature (30 frames per

second on a two-hour movie is a lot!), movie studios need to spread the full-rendering

job to large numbers of computers (computer farms as they are called).

Other times, the very nature of the application being developed requires a

distributed system. This is the case, for instance, for instant messaging and video

conferencing applications. For these pieces of software, performance is not the main

driver. It is just that the problem that the application solves is itself distributed.

In the following gure, we see a very common web application architecture

(another example of a distributed application), where multiple users connect to

the website over the network. At the same time, the application itself communicates

with systems (such as a database server) running on different machines in its LAN:

Web User

Web User Web User Web User

Database Server

Another interesting example of distributed systems, which might be a bit

counterintuitive, is the CPU-GPU combination. These days, graphics cards are

very sophisticated computers in their own right. They are highly parallel and offer

compelling performance for a large number of compute-intensive problems, not just

for displaying images on screen. Tools and libraries exist to allow programmers to

make use of GPUs for general-purpose computing (for example CUDA from NVIDIA,

OpenCL, and OpenAcc among others).

剩余170页未读，继续阅读

gaoxinmutha

粉丝: 8
资源: 19

Python分布式计算实战指南

Distributed Computing with Python [2016]

Distributed Computing with Python

Distributed Computing with Python azw3

feature engineering python

Please initialize `TimeDistributed` layer with a `tf.keras.layers.Layer` instance. Received: 10

idea Failed to execute job 'Flink Streaming Job'

/opt/anaconda3/envs/yolov5/bin/python: No module named torch.distributed.run

XGBOOST python

AttributeError: module 'tensorflow.python.keras' has no attribute 'TimeDistributed'

最新资源