MPI-2高级特性的并行计算指南：实践与推荐

5星 · 超过95%的资源需积分: 9 32 浏览量更新于2024-08-02 收藏 1.95MB PDF 举报

《Using MPI-2：高级特性》是一本专为并行计算领域读者设计的权威指南，由麻省理工学院出版社出版。这本书深入讲解了Message Passing Interface（MPI）的第二版，MPI是广泛应用于高性能计算中的标准通信库，特别是科学与工程计算中的并行处理。作者以其丰富的经验和详实的内容，让读者掌握如何有效地利用MPI的高级特性和最佳实践，以提升多处理器环境下的数据并行程序性能。在书中，作者不仅介绍了MPI-2的基本概念，还涵盖了多个关键主题。首先，通过《Data-Parallel Programming on MIMD Computers》一书，读者可以了解到如何在大规模并行系统（MIMD，Multiple Instruction Multiple Data）上进行数据并行编程。该部分强调了并行计算中的数据组织和分发策略。其次，书籍列举了《Unstructured Scientific Computation on Scalable Multiprocessors》、《Parallel Computational Fluid Dynamics: Implementation and Results》等著作，这些篇章着重于实际应用中的案例研究，如科学模拟和流体动力学，展示了MPI在解决复杂问题时的灵活性和效率。《Enterprise Integration Modeling》则讨论了MPI在企业级集成中的角色，展示了MPI如何促进不同系统间的高效通信，这对于分布式计算和云计算环境中的应用具有重要意义。《The High Performance Fortran Handbook》则将MPI与高级编程语言如Fortran结合，提供了优化并行代码的技巧，帮助开发者编写出性能优越的程序。 PVM（Parallel Virtual Machine）的相关章节详细解释了分布式计算框架，它是一种网络并行计算工具，展示了MPI与其他并行技术的互补性，如《PVM: Parallel Virtual Machine - A Users' Guide and Tutorial for Network Parallel Computing》。对于更为实战的指导，《Practical Parallel Programming》由Gregory V. Wilson撰写，书中包含了丰富的实例和实用的编程技巧，适合初学者和经验丰富的程序员提升并行编程技能。最后，书中的《Enabling Technologies for Petaflops Computing》探讨了随着计算需求的提升，MPI如何适应和推动实现每秒千万亿次浮点运算（Petaflops）级的超级计算机，以及未来高性能计算的发展趋势。总而言之，《Using MPI-2 - Advanced Features》是一本涵盖了MPI核心原理、应用案例、编程实践和技术前沿的综合指南，对于任何希望在并行计算领域深化理解或寻求技术突破的读者来说，都是一份宝贵的资源。

In the year following its release, MPI was taken up enthusiastically by users, and a 1995 survey by the Ohio Supercomputer

Center showed that even its more esoteric features found users. The MPICH portable implementation [30], layered on top of

existing vendor systems, was available immediately, since it had evolved along with the standard. Other portable

implementations appeared, particularly LAM [7], and then vendor implementations in short order, some of them leveraging

MPICH. The first edition of Using MPI [31] appeared in the fall of 1994, and we like to think that it helped win users to the

new Standard.

But the very success of MPI-1 drew attention to what was not there. PVM users missed dynamic process creation, and several

users needed parallel I/O. The success of the Cray shmem library on the Cray T3D and the active-message library on the CM-5

made users aware of the advantages of "one-sided" operations in algorithm design. The MPI Forum would have to go back to

work.

Page 3

1.1.3—

The MPI-2 Forum

The modern history of MPI begins in the spring of 1995, when the Forum resumed its meeting schedule, with both veterans of

MPI-1 and about an equal number of new participants. In the previous three years, much had changed in parallel computing,

and these changes would accelerate during the two years the MPI-2 Forum would meet.

On the hardware front, a consolidation of MPP vendors occurred, with Thinking Machines Corp., Meiko, and Intel all leaving

the marketplace. New entries such as Convex (now absorbed into Hewlett-Packard) and SGI (now having absorbed Cray

Research) championed a shared-memory model of parallel computation although they supported MPI (passing messages

through shared memory), and many applications found that the message-passing model was still well suited for extracting peak

performance on shared-memory (really NUMA) hardware. Small-scale shared-memory multiprocessors (SMPs) became

available from workstation vendors and even PC manufacturers. Fast commodity-priced networks, driven by the PC

marketplace, became so inexpensive that clusters of PCs combined with inexpensive networks, started to appear as "home-

brew" parallel supercomputers. A new federal program, the Accelerated Strategic Computing Initiative (ASCI), funded the

development of the largest parallel computers ever built, with thousands of processors. ASCI planned for its huge applications

to use MPI.

On the software front, MPI, as represented by MPI-1, became ubiquitous as the application programming interface (API) for

the message-passing model. The model itself remained healthy. Even on flat shared-memory and NUMA (nonuniform memory

access) machines, users found the message-passing model a good way to control cache behavior and thus performance. The

perceived complexity of programming with the message-passing model was alleviated by two developments. The first was the

convenience of the MPI interface itself, once programmers became more comfortable with it as the result of both experience

and tutorial presentations. The second was the appearance of libraries that hide much of the MPI-level complexity from the

application programmer. Examples are PETSc [3], ScaLAPACK [12], and PLAPACK [94]. This second development is

especially satisfying because it was an explicit design goal for the MPI Forum to encourage the development of libraries by

including features that libraries particularly needed.

At the same time, non-message-passing models have been explored. Some of these may be beneficial if actually adopted as

portable standards; others may still require interaction with MPI to achieve scalability. Here we briefly summarize two

promising, but quite different approaches.

Page 4

Explicit multithreading is the use of an API that manipulates threads (see [32] for definitions within a single address space.

This approach may be sufficient on systems that can devote a large number of CPUs to servicing a single process, but

interprocess communication will still need to be used on scalable systems. The MPI API has been designed to be thread safe.

However, not all implementations are thread safe. An MPI-2 feature is to allow applications to request and MPI

implementations to report their level of thread safety (see Chapter 8).

In some cases the compiler generates the thread parallelism. In such cases the application or library uses only the MPI API, and

additional parallelism is uncovered by the compiler and expressed in the code it generates. Some compilers do this unaided;

others respond to directives in the form of specific comments in the code.

OpenMP is a proposed standard for compiler directives for expressing parallelism, with particular emphasis on loop-level

parallelism. Both C [68] and Fortran [67] versions exist.

Thus the MPI-2 Forum met during time of great dynamism in parallel programming models. What did the Forum do, and what

did it come up with?

1.2—

What's New in MPI-2?

The MPI-2 Forum began meeting in March of 1995. Since the MPI-1 Forum was judged to have been a successful effort, the

new Forum procedures were kept the same as for MPI-1. Anyone was welcome to attend the Forum meetings, which were held

every six weeks. Minutes of the meetings were posted to the Forum mailing lists, and chapter drafts were circulated publicly

for comments between meetings. At meetings, subcommittees for various chapters met and hammered out details, and the final

version of the standard was the result of multiple votes by the entire Forum.

The first action of the Forum was to correct errors and clarify a number of issues that had caused misunderstandings in the

original document of July 1994, which was retroactively labeled MPI-1.0. These minor modifications, encapsulated as MPI-

1.1, were released in May 1995. Corrections and clarifications, to MPI-1 topics continued during the next two years, and the

MPI-2 document contains MPI-1.2 as a chapter (Chapter 3) of the MPI-2 release, which is the current version of the MPI

standard. MPI-1.2 also contains a number of topics that belong in spirit to the MPI-1 discussion, although they were added by

the MPI-2 Forum.

Page 5

MPI-2 has three "large," completely new areas, which represent extensions of the MPI programming model substantially

beyond the strict message-passing model represented by MPI-1. These areas are parallel I/O, remote memory operations, and

dynamic process management. In addition, MPI-2 introduces a number of features designed to make all of MPI more robust

and convenient to use, such as external interface specifications, C++ and Fortran-90 bindings, support for threads, and mixed-

language programming.

1.2.1—

Parallel I/O

The parallel I/O part of MPI-2, sometimes just called MPI-IO, originated independently of the Forum activities, as an effort

within IBM to explore the analogy between input/output and message passing. After all, one can think of writing to a file as

analogous to sending a message to the file system and reading from a file as receiving a message from it. Furthermore, any

parallel I/O system is likely to need collective operations, ways of defining noncontiguous data layouts both in memory and in

files, and nonblocking operations. In other words, it will need a number of concepts that have already been satisfactorily

specified and implemented in MPI. The first study of the MPI-IO idea was carried out at IBM Research [71]. The effort was

expanded to include a group at NASA Ames, and the resulting specification appeared in [15]. After that, an open e-mail

discussion group was formed, and this group released a series of proposals, culminating in [90]. At that point the group merged

with the MPI Forum, and I/O became a part of MPI-2. The I/O specification evolved further over the course of the Forum

meetings, until MPI-2 was finalized in July 1997.

In general, I/O in MPI-2 can be thought of as Unix I/O plus quite a lot more. That is, MPI does include analogues of the basic

operations of open, close, seek, read, and write. The arguments for these functions are similar to those of the

corresponding Unix I/O operations, making an initial port of existing programs to MPI relatively straightforward. The purpose

of parallel I/O in MPI, however, is to achieve much higher performance than the Unix API can deliver, and serious users of

MPI must avail themselves of the more advanced features, which include

• noncontiguous access in both memory and file,

• collective I/O operations,

• use of explicit offsets to avoid separate seeks,

• both individual and shared file pointers,

• nonblocking I/O,

• portable and customized data representations, and

Page 6

• hints for the implementation and file system.

We will explore in detail in Chapter 3 exactly how to exploit these features. We will find out there just how the I/O API

defined by MPI enables optimizations that the Unix I/O API precludes.

1.2.2—

Remote Memory Operations

The hallmark of the message-passing model is that data is moved from the address space of one process to that of another by

means of a cooperative operation such as a send/receive pair. This restriction sharply distinguishes the message-passing

model from the shared-memory model, in which processes have access to a common pool of memory and can simply perform

ordinary memory operations (load from, store into) on some set of addresses.

In MPI-2, an API is defined that provides elements of the shared-memory model in an MPI environment. These are called

MPI's "one-sided" or "remote memory" operations. Their design was governed by the need to

• balance efficiency and portability across several classes of architectures, including shared-memory multiprocessors (SMPs),

nonuniform memory access (NUMA) machines, distributed-memory massively parallel processors (MPPs), SMP clusters, and

even heterogeneous networks;

• retain the "look and feel" of MPI-1;

• deal with subtle memory behavior issues, such as cache coherence and sequential consistency; and

• separate synchronization from data movement to enhance performance.

The resulting design is based on the idea of remote memory access windows: portions of each process's address space that it

explicitly exposes to remote memory operations by other processes defined by an MPI communicator. Then the one-sided

operations put, get, and accumulate can store into, load from, and update, respectively, the windows exposed by other

processes. All remote memory operations are nonblocking, and synchronization operations are necessary to ensure their

completion. A variety of such synchronizations operations are provided, some for simplicity, some for precise control, and

some for their analogy with shared-memory synchronization operations. In Chapter 4, we explore some of the issues of

synchronization between senders and receivers of data. Chapters 5 and 6 describe the remote memory operations of MPI-2 in

detail.

Page 7

1.2.3—

Dynamic Process Management

The third major departure from the programming model defined by MPI-1 is the ability of an MPI process to participate in the

creation of new MPI processes or to establish communication with MPI processes that have been started separately. The main

issues faced in designing an API for dynamic process management are

• maintaining simplicity and flexibility;

• interacting with the operating system, the resource manager, and the process manager in a complex system software

environment; and

• avoiding race conditions that compromise correctness.

The key to correctness is to make the dynamic process management operations collective, both among the processes doing the

creation of new processes and among the new processes being created. The resulting sets of processes are represented in an

intercommunicator. Intercommunicators (communicators containing two groups of processes rather than one) are an esoteric

feature of MPI-1, but are fundamental for the MPI-2 dynamic process operations. The two families of operations defined in

MPI-2, both based on intercommunicators, are creating of new sets of processes, called spawning, and establishing

communications with pre-existing MPI programs, called connecting. The latter capability allows applications to have parallel-

client/parallel-server structures of processes. Details of the dynamic process management operations can be found in Chapter 7.

1.2.4—

Odds and Ends

Besides the above ''big three," the MPI-2 specification covers a number of issues that were not discussed in MPI-1.

Extended Collective Operations

Extended collective operations in MPI-2 are analogous to the collective operations of MPI-1, but are defined for use on

intercommunicators. (In MPI-1, collective operations are restricted to intracommunicators.) MPI-2 also extends the MPI-1

intracommunicator collective operations to allow an "in place" option, in which the send and receive buffers are the same.

C++ and Fortran 90

In MPI-1, the only languages considered were C and Fortran, where Fortran was construed as Fortran 77. In MPI-2, all

functions (including MPI-1 functions) have C++ bindings, and Fortran means Fortran 90 (or Fortran 95 [1]). For C++, the MPI-

2 Forum chose a "minimal" approach in which the C++ versions of MPI functions are quite similar to the C versions, with

classes defined

Page 8

for most of the MPI objects (such as MPI::Request for the C MPI_Request. Most MPI functions are member functions

of MPI classes (easy to do because MPI has an object-oriented design), and others are in the MPI namespace.

MPI can't take advantage of some Fortran-90 features, such as array sections, and some MPI functions, particularly ones like

MPI-Send that use a "choice" argument, can run afoul of Fortran's compile-time type checking for arguments to routines. This

is usually harmless but can cause warning messages. However, the use of choice arguments does not match the letter of the

Fortran standard; some Fortran compilers may require the use of a compiler option to relax this restriction in the Fortran

language.

"Basic" and "extended" levels of support for Fortran 90 are provided in MPI-2. Essentially, basic support requires

that mpif.h be valid in both fixed-and free-form format, and "extended" support includes an MPI module and some new

functions that use parameterized types. Since these language extensions apply to all of MPI, not just MPI-2, they are covered in

detail in the second edition of Using MPI [32] rather than in this book.

Language Interoperability

Language interoperability is a new feature in MPI-2. MPI-2 defines features, both by defining new functions and by specifying

the behavior of implementations, that enable mixed-language programming, an area ignored by MPI-1.

External Interfaces

The external interfaces part of MPI makes it easy for libraries to extend MPI by accessing aspects of the implementation that

are opaque in MPI-1. It aids in the construction of integrated tools, such as debuggers and performance analyzers, and is

already being used in the early implementations of the MPI-2 I/O functionality [88].

Threads

MPI-1, other than designing a thread-safe interface, ignored the issue of threads. In MPI-2, threads are recognized as a potential

part of an MPI programming environment. Users can inquire of an implementation at run time what

Because Fortran uses compile-time data-type matching rather than run-time data-type matching, it is invalid to make two calls

to the same routine in which two different data types are used in the same argument position. This affects the "choice" arguments

in the MPI Standard. For example, calling MPI-Send with a first argument of type integer and then with a first argument of

type real is invalid in Fortran 77. In Fortran 90, when using the extended Fortran support, it is possible to allow arguments of

different types by specifying the appropriate interfaces in the MPI module. However, this requires a different interface for each

type and is not a practical approach for Fortran 90 derived types. MPI does provide for data-type checking, but does so at run

time through a separate argument, the MPI datatype argument.

Page 9

its level of thread-safety is. In cases where the implementation supports multiple levels of thread-safety, users can select the

level that meets the application's needs while still providing the highest possible performance.

1.3—

Reading This Book

This book is not a complete reference book for MPI-2. We leave that to the Standard itself [59] and to the two volumes of

MPI—The Complete Reference [27, 79]. This book, like its companion Using MPI focusing on MPI-1, is organized around

using the concepts of MPI-2 in application programs. Hence we take an iterative approach. In the preceding section we

presented a very high level overview of the contents of MPI-2. In the next chapter we demonstrate the use of several of these

concepts in simple example programs. Then in the following chapters we go into each of the major areas of MPI-2 in detail.

We start with the parallel I/O capabilities of MPI in Chapter 3, since that has proven to be the single most desired part of MPI-

2. In Chapter 4 we explore some of the issues of synchronization between senders and receivers of data. The complexity and

importance of remote memory operations deserve two chapters, Chapters 5 and 6. The next chapter, Chapter 7, is on dynamic

process management. We follow that with a chapter on MPI and threads, Chapter 8, since the mixture of multithreading and

message passing is likely to become a widely used programming model. In Chapter 9 we consider some advanced features of

MPI-2 that are particularly useful to library writers. We conclude in Chapter 10 with an assessment of possible future directions

for MPI.

In each chapter we focus on example programs to illustrate MPI as it is actually used. Some miscellaneous minor topics will

just appear where the example at hand seems to be a good fit for them. To find a discussion on a given topic, you can consult

either the subject index or the function and term index, which is organized by MPI function name.

Finally, you may wish to consult the companion volume, Using MPI: Portable Parallel Programming with the Message-

passing Interface [32]. Some topics considered by the MPI-2 Forum are small extensions to MPI-1 topics and are covered in

the second edition (1999) of Using MPI. Although we have tried to make this volume self-contained, some of the examples

have their origins in the examples of Using MPI.

Now, let's get started!

Page 11

2—

Getting Started with MPI-2

In this chapter we demonstrate what MPI-2 "looks like," while deferring the details to later chapters. We use relatively simple

examples to give a flavor of the new capabilities provided by MPI-2. We focus on the main areas of parallel I/O, remote

memory operations, and dynamic process management, but along the way demonstrate MPI in its new language bindings, C++

and Fortran 90, and touch on a few new features of MPI-2 as they come up.

2.1—

Portable Process Startup

One small but useful new feature of MPI-2 is the recommendation of a standard method for starting MPI programs. The

simplest version of this is

mpiexec -n 16 myprog

to run the program myprog with 16 processes.

剩余274页未读，继续阅读

yorua007

粉丝: 1
资源: 2

MPI-2高级特性的并行计算指南：实践与推荐

Using.MPI-2.Advanced.Features

'train': ( "{cmd_mpi:s} nnp-scaling 100 > nnp-scaling-stdout.log 2> nnp-scaling-stdout.err; " "{cmd_mpi:s} nnp-train > nnp-train-stdout.log 2> nnp-train-stdout.err"), 'predict': '{cmd_mpi:s} nnp-dataset 0 > nnp-dataset-stdout.log 2> nnp-dataset-stdout.err'

S7-300 MPI/DP通讯 Python

进程0->进程1->进程2->进程3...->进程n->进程0

cross-mpi: cross-scale stereo for image super-resolution using multiplane images

详细说说MPI.COMM_WORLD

vscode无法打开mpi.h源文件

conda install -c conda-forge glpk-mpi

最新资源