CUDA编程最佳实践指南

需积分: 10 123 浏览量更新于2024-07-19 收藏 2.1MB PDF 举报

"CUDA实践指南" CUDA（Compute Unified Device Architecture）是NVIDIA推出的一种并行计算平台和编程模型，主要用于利用图形处理单元（GPU）进行高性能计算。本实践指南旨在为开发者提供CUDA编程的最佳实践，帮助他们高效地利用GPU的计算能力。在开始CUDA编程之前，了解文档的目标群体至关重要。这本指南适合对并行计算有兴趣，希望通过CUDA技术优化应用性能的软件开发者，无论他们是初学者还是有经验的程序员。文档的核心思想是“评估、并行化、优化、部署”（Assess, Parallelize, Optimize, Deploy）这一流程，旨在引导读者逐步提升CUDA程序的效率和性能。 1. **评估（Assess）**：这部分介绍了如何分析现有应用程序的性能瓶颈，以确定哪些部分可以受益于GPU加速。开发者需要理解代码执行的时间消耗，识别出那些计算密集型的任务，这些任务通常是并行化的理想候选。 2. **并行化（Parallelize）**：并行化是CUDA编程的关键，本章会探讨如何将任务分解到多个GPU核心上执行。内容包括了CPU与GPU之间的差异，以及GPU能够执行的任务类型，如计算密集型的数学运算、数据处理等。 3. **应用调优（Optimize）**：调优是提升CUDA程序性能的重要步骤。这里会讨论如何通过性能分析工具（如NVIDIA的nvprof）来创建和解析性能报告，找出程序中的热点（hotspots），理解并行化后性能的强弱缩放原则，如Amdahl定律和Gustafson定律，以及如何根据这些理论调整代码。 4. **开始CUDA编程（Getting Started）**：对于初学者，此章节提供了入门指导，包括使用CUDA库、并行编译器，以及如何编写暴露并行性的代码。CUDA库如cuBLAS、cuFFT等可以大大简化计算任务，而并行编译器如NVCC则可以帮助生成GPU可执行的代码。 5. **获取正确结果（Getting the Right Answer）**：在追求性能的同时，确保程序的正确性同样重要。本章讲解了验证方法，如通过对比参考实现或进行单元测试来检查结果的准确性。同时，调试技巧也在此部分介绍，这对于查找并修复CUDA程序中的错误是必不可少的。 6. **设备选择与配置**：在部署阶段，选择合适的GPU设备和配置是关键，要考虑硬件的兼容性、内存需求、功耗等因素，以确保应用程序能在目标环境中高效运行。 "CUDA实践指南"是一本深入浅出的教程，涵盖了从评估应用到优化CUDA程序的全过程，为开发者提供了丰富的实战经验和策略，以充分利用CUDA的潜力，实现高性能计算。

Heterogeneous Computing

www.nvidia.com

CUDA C Best Practices Guide DG-05603-001_v9.0|4

data element transferred. For the preceding procedure, assuming matrices of

size NxN, there are N

operations (additions) and 3N

elements transferred,

so the ratio of operations to elements transferred is 1:3 or O(1). Performance

benefits can be more readily achieved when this ratio is higher. For example,

a matrix multiplication of the same matrices requires N

operations (multiply-

add), so the ratio of operations to elements transferred is O(N), in which case the

larger the matrix the greater the performance benefit. The types of operations

are an additional factor, as additions have different complexity profiles than,

for example, trigonometric functions. It is important to include the overhead

of transferring data to and from the device in determining whether operations

should be performed on the host or on the device.

‣

Data should be kept on the device as long as possible. Because transfers

should be minimized, programs that run multiple kernels on the same data

should favor leaving the data on the device between kernel calls, rather than

transferring intermediate results to the host and then sending them back to the

device for subsequent calculations. So, in the previous example, had the two

matrices to be added already been on the device as a result of some previous

calculation, or if the results of the addition would be used in some subsequent

calculation, the matrix addition should be performed locally on the device. This

approach should be used even if one of the steps in a sequence of calculations

could be performed faster on the host. Even a relatively slow kernel may be

advantageous if it avoids one or more PCIe transfers. Data Transfer Between

Host and Device provides further details, including the measurements of

bandwidth between the host and the device versus within the device proper.

Application Profiling

www.nvidia.com

CUDA C Best Practices Guide DG-05603-001_v9.0|6

$ gcc -O2 -g -pg myprog.c

$ gprof ./a.out > profile.txt

Each sample counts as 0.01 seconds.

% cumulative self self total

time seconds seconds calls ms/call ms/call name

33.34 0.02 0.02 7208 0.00 0.00 genTimeStep

16.67 0.03 0.01 240 0.04 0.12 calcStats

16.67 0.04 0.01 8 1.25 1.25 calcSummaryData

16.67 0.05 0.01 7 1.43 1.43 write

16.67 0.06 0.01 mcount

0.00 0.06 0.00 236 0.00 0.00 tzset

0.00 0.06 0.00 192 0.00 0.00 tolower

0.00 0.06 0.00 47 0.00 0.00 strlen

0.00 0.06 0.00 45 0.00 0.00 strchr

0.00 0.06 0.00 1 0.00 50.00 main

0.00 0.06 0.00 1 0.00 0.00 memcpy

0.00 0.06 0.00 1 0.00 10.11 print

0.00 0.06 0.00 1 0.00 0.00 profil

0.00 0.06 0.00 1 0.00 50.00 report

3.1.2.Identifying Hotspots

In the example above, we can clearly see that the function genTimeStep() takes one-

third of the total running time of the application. This should be our first candidate

function for parallelization. Understanding Scaling discusses the potential benefit we

might expect from such parallelization.

It is worth noting that several of the other functions in the above example also take

up a significant portion of the overall running time, such as calcStats() and

calcSummaryData(). Parallelizing these functions as well should increase our speedup

potential. However, since APOD is a cyclical process, we might opt to parallelize these

functions in a subsequent APOD pass, thereby limiting the scope of our work in any

given pass to a smaller set of incremental changes.

3.1.3.Understanding Scaling

The amount of performance benefit an application will realize by running on CUDA

depends entirely on the extent to which it can be parallelized. Code that cannot be

sufficiently parallelized should run on the host, unless doing so would result in

excessive transfers between the host and the device.

High Priority: To get the maximum benefit from CUDA, focus first on finding ways to

parallelize sequential code.

By understanding how applications can scale it is possible to set expectations and plan

an incremental parallelization strategy. Strong Scaling and Amdahl's Law describes

strong scaling, which allows us to set an upper bound for the speedup with a fixed

problem size. Weak Scaling and Gustafson's Law describes weak scaling, where the

speedup is attained by growing the problem size. In many applications, a combination

of strong and weak scaling is desirable.

Application Profiling

www.nvidia.com

CUDA C Best Practices Guide DG-05603-001_v9.0|7

3.1.3.1.Strong Scaling and Amdahl's Law

Strong scaling is a measure of how, for a fixed overall problem size, the time to solution

decreases as more processors are added to a system. An application that exhibits linear

strong scaling has a speedup equal to the number of processors used.

Strong scaling is usually equated with Amdahl's Law, which specifies the maximum

speedup that can be expected by parallelizing portions of a serial program. Essentially, it

states that the maximum speedup S of a program is:

Here P is the fraction of the total serial execution time taken by the portion of code that

can be parallelized and N is the number of processors over which the parallel portion of

the code runs.

The larger N is(that is, the greater the number of processors), the smaller the P/

N fraction. It can be simpler to view N as a very large number, which essentially

transforms the equation into . Now, if 3/4 of the running time of a sequential

program is parallelized, the maximum speedup over serial code is 1 / (1 - 3/4) = 4.

In reality, most applications do not exhibit perfectly linear strong scaling, even if they

do exhibit some degree of strong scaling. For most purposes, the key point is that the

larger the parallelizable portion P is, the greater the potential speedup. Conversely, if

P is a small number (meaning that the application is not substantially parallelizable),

increasing the number of processors N does little to improve performance. Therefore,

to get the largest speedup for a fixed problem size, it is worthwhile to spend effort on

increasing P, maximizing the amount of code that can be parallelized.

3.1.3.2.Weak Scaling and Gustafson's Law

Weak scaling is a measure of how the time to solution changes as more processors are

added to a system with a fixed problem size per processor; i.e., where the overall problem

size increases as the number of processors is increased.

Weak scaling is often equated with Gustafson's Law, which states that in practice, the

problem size scales with the number of processors. Because of this, the maximum

speedup S of a program is:

Here P is the fraction of the total serial execution time taken by the portion of code that

can be parallelized and N is the number of processors over which the parallel portion of

the code runs.

Another way of looking at Gustafson's Law is that it is not the problem size that remains

constant as we scale up the system but rather the execution time. Note that Gustafson's

Law assumes that the ratio of serial to parallel execution remains constant, reflecting

additional cost in setting up and handling the larger problem.

剩余84页未读，继续阅读

HitManReborn

粉丝: 0
资源: 2

CUDA编程最佳实践指南

CUDA编程指南5.0中文版

CUDA编程指南

CUDA编程实践指南

GPU编程基础：CUDA Fortran实践指南

CUDA最佳实践指南：从评估到优化

NVIDIA CUDA C最佳实践指南

CUDA编程最佳实践指南：CUDA4.0官方文档详解

CUDA C编程最佳实践指南

CUDA Fortran高效编程实践指南

CUDA编程指南：入门到实践

最新资源