PyTorch 1.5.1官方文档：离线PDF版与Python API详解

5星 · 超过95%的资源需积分: 24 173 浏览量更新于2024-07-15 1 收藏 11.76MB PDF 举报

"这是一份包含了PyTorch 1.5.1版本的官方英文文档，专注于Python API和库的详细信息。文档包含了书签，便于离线查阅和高效学习。文档内容覆盖了PyTorch的核心功能、自动混合精度、自动微分机制、广播语义、CPU线程和TorchScript推理、CUDA语义、分布式数据并行处理、扩展PyTorch、常见问题解答、大型部署特性、多进程最佳实践、可重复性、序列化语义、Windows平台常见问题，以及C++、Java和Python等语言接口。此外，还涉及到了量化、分布式RPC框架、随机数生成、稀疏张量、存储、数据集工具、模型动物园、ONNX导出、优化器、命名张量等扩展功能和库，如torchaudio、torchtext和torchvis。" PyTorch是一个用于深度学习的高性能张量库，支持GPU和CPU计算。文档中的"Automatic Mixed Precision examples"部分介绍如何利用混合精度训练提高模型在GPU上的速度和内存效率。"Autograd mechanics"解释了PyTorch的自动微分系统，这是实现反向传播的关键。"Broadcasting semantics"则描述了张量操作时形状不匹配时的广播规则。 "CPU threading and TorchScript inference"讨论了PyTorch如何处理CPU线程以及TorchScript如何进行推理。"CUDA semantics"涵盖了在GPU上运行计算的细节。"Distributed Data Parallel"部分介绍了如何在多GPU或多节点上进行分布式训练。 "Extending PyTorch"章节提供了自定义模块和操作符的指导，对于开发人员来说非常有用。"Frequently Asked Questions (FAQs)"包含了对使用过程中遇到问题的解答，有助于解决常见问题。"Multiprocessing best practices"提供了多进程编程的最佳实践，以提高计算效率。 "Serialization semantics"讲述了如何保存和加载模型，确保可重复性。"Windows FAQ"针对Windows用户，提供了特定平台的解决方法。"Language Bindings"包括了C++和Java接口，以及主要的Python API。 "torch"、"torch.nn"和"torch.nn.functional"是PyTorch的核心模块，分别提供基本张量操作、神经网络构建模块和函数。"torch.Tensor"和"Tensor Attributes"详述了张量的属性和操作。"torch.autograd"是自动微分的主要部分，"torch.cuda"则涉及CUDA支持。 "torch.distributed"支持分布式训练，"torch.distributions"提供了概率分布函数，"torch.hub"用于加载和分享预训练模型。"torch.jit"是TorchScript的接口，用于模型编译和推理优化。"torch.optim"包含了常见的优化算法。文档还涵盖了量化（Quantization）、分布式RPC框架、torch.random（随机数生成）、torch.sparse（稀疏张量）和torch.Storage（存储管理）。"torch.utils"子模块提供了各种实用工具，如数据集处理、性能瓶颈分析、C++扩展编写、模型动物园、ONNX导出等。"TypeInfo"部分介绍了PyTorch的类型系统，"Named Tensors"则引入了具有名称的张量，以增强代码的可读性。最后，文档列出了几个库，如torchaudio用于音频处理，torchtext用于自然语言处理，以及torchvis用于可视化。这些库扩展了PyTorch的功能，使其在特定领域更具实用性。通过这份详尽的文档，开发者可以深入理解PyTorch，更好地利用其功能进行深度学习项目。

CUDA streams
A CUDA stream is a linear sequence of execution that belongs to a specific device. You normally do not need to create one explicitly: by default, each device uses its own “default” stream.
Operations inside each stream are serialized in the order they are created, but operations from different streams can execute concurrently in any relative order, unless explicit
synchronization functions (such as  synchronize()  or  wait_stream() ) are used. For example, the following code is incorrect:
cuda = torch.device('cuda')
s = torch.cuda.Stream()  
# Create a new stream.
A = torch.empty((100, 100), device=cuda).normal_(0.0, 1.0)
with torch.cuda.stream(s): 
    
# sum() may start execution before normal_() finishes!
 
    B = torch.sum(A)
When the “current stream” is the default stream, PyTorch automatically performs necessary synchronization when data is moved around, as explained above. However, when using non-
default streams, it is the user’s responsibility to ensure proper synchronization.
Memory management
PyTorch uses a caching memory allocator to speed up memory allocations. This allows fast memory deallocation without device synchronizations. However, the unused memory managed by
the allocator will still show as if used in  nvidia-smi . You can use  memory_allocated()  and  max_memory_allocated()  to monitor memory occupied by tensors, and use
memory_reserved()  and  max_memory_reserved()  to monitor the total amount of memory managed by the caching allocator. Calling  empty_cache()  releases all unused
cached memory from PyTorch so that those can be used by other GPU applications. However, the occupied GPU memory by tensors will not be freed so it can not increase the amount of
GPU memory available for PyTorch.
For more advanced users, we offer more comprehensive memory benchmarking via  memory_stats() . We also offer the capability to capture a complete snapshot of the memory
allocator state via  memory_snapshot() , which can help you understand the underlying allocation patterns produced by your code.
cuFFT plan cache
For each CUDA device, an LRU cache of cuFFT plans is used to speed up repeatedly running FFT methods (e.g.,  torch.fft() ) on CUDA tensors of same geometry with same
configuration. Because some cuFFT plans may allocate GPU memory, these caches have a maximum capacity.
You may control and query the properties of the cache of current device with the following APIs:
torch.backends.cuda.cufft_plan_cache.max_size  gives the capacity of the cache (default is 4096 on CUDA 10 and newer, and 1023 on older CUDA versions). Setting
this value directly modifies the capacity.
torch.backends.cuda.cufft_plan_cache.size  gives the number of plans currently residing in the cache.
torch.backends.cuda.cufft_plan_cache.clear()  clears the cache.
To control and query plan caches of a non-default device, you can index the  torch.backends.cuda.cufft_plan_cache  object with either a  torch.device  object or a device
index, and access one of the above attributes. E.g., to set the capacity of the cache for device  1 , one can write  torch.backends.cuda.cufft_plan_cache[1].max_size =
10 .
Best practices
Device-agnostic code
Due to the structure of PyTorch, you may need to explicitly write device-agnostic (CPU or GPU) code; an example may be creating a new tensor as the initial hidden state of a recurrent
neural network.
The first step is to determine whether the GPU should be used or not. A common pattern is to use Python’s  argparse  module to read in user arguments, and have a flag that can be used
to disable CUDA, in combination with  is_available() . In the following,  args.device  results in a  torch.device  object that can be used to move tensors to CPU or CUDA.
import argparse
import torch 
 
parser = argparse.ArgumentParser(description='PyTorch Example')
parser.add_argument('--disable-cuda', action='store_true', 
                    help='Disable CUDA')
args = parser.parse_args()
args.device = None
if not args.disable_cuda and torch.cuda.is_available(): 
    args.device = torch.device('cuda')
else: 
    args.device = torch.device('cpu')
Now that we have  args.device , we can use it to create a Tensor on the desired device.
x = torch.empty((8, 42), device=args.device)
net = Network().to(device=args.device)
This can be used in a number of cases to produce device agnostic code. Below is an example when using a dataloader:

Next 
cuda0 = torch.device('cuda:0')  
# CUDA GPU 0
for i, x in enumerate(train_loader): 
    x = x.to(cuda0)
When working with multiple GPUs on a system, you can use the  CUDA_VISIBLE_DEVICES  environment flag to manage which GPUs are available to PyTorch. As mentioned above, to
manually control which GPU a tensor is created on, the best practice is to use a  torch.cuda.device  context manager.
print("Outside device is 0")  
# On device 0 (default in most scenarios)
with torch.cuda.device(1): 
    print("Inside device is 1")  
# On device 1
print("Outside device is still 0")  
# On device 0
If you have a tensor and would like to create a new tensor of the same type on the same device, then you can use a  torch.Tensor.new_*  method (see  torch.Tensor ). Whilst the
previously mentioned  torch.*  factory functions (Creation Ops) depend on the current GPU context and the attributes arguments you pass in,  torch.Tensor.new_*  methods
preserve the device and other attributes of the tensor.
This is the recommended practice when creating modules in which new tensors need to be created internally during the forward pass.
cuda = torch.device('cuda')
x_cpu = torch.empty(2)
x_gpu = torch.empty(2, device=cuda)
x_cpu_long = torch.empty(2, dtype=torch.int64) 
 
y_cpu = x_cpu.new_full([3, 2], fill_value=0.3)
print(y_cpu) 
 
    tensor([[ 0.3000,  0.3000], 
            [ 0.3000,  0.3000], 
            [ 0.3000,  0.3000]]) 
 
y_gpu = x_gpu.new_full([3, 2], fill_value=-5)
print(y_gpu) 
 
    tensor([[-5.0000, -5.0000], 
            [-5.0000, -5.0000], 
            [-5.0000, -5.0000]], device='cuda:0') 
 
y_cpu_long = x_cpu_long.new_tensor([[1, 2, 3]])
print(y_cpu_long) 
 
    tensor([[ 1,  2,  3]])
If you want to create a tensor of the same type and size of another tensor, and fill it with either ones or zeros,  ones_like()  or  zeros_like()  are provided as convenient helper
functions (which also preserve  torch.device  and  torch.dtype  of a Tensor).
x_cpu = torch.empty(2, 3)
x_gpu = torch.empty(2, 3) 
 
y_cpu = torch.ones_like(x_cpu)
y_gpu = torch.zeros_like(x_gpu)
Use pinned memory buffers
Host to GPU copies are much faster when they originate from pinned (page-locked) memory. CPU tensors and storages expose a  pin_memory()  method, that returns a copy of the
object, with data put in a pinned region.
Also, once you pin a tensor or storage, you can use asynchronous GPU copies. Just pass an additional  non_blocking=True  argument to a  to()  or a  cuda()  call. This can be used
to overlap data transfers with computation.
You can make the  DataLoader  return batches placed in pinned memory by passing  pin_memory=True  to its constructor.
Use nn.parallel.DistributedDataParallel instead of multiprocessing or nn.DataParallel
Most use cases involving batched inputs and multiple GPUs should default to using  DistributedDataParallel  to utilize more than one GPU.
There are significant caveats to using CUDA models with  multiprocessing ; unless care is taken to meet the data handling requirements exactly, it is likely that your program will have
incorrect or undefined behavior.
It is recommended to use  DistributedDataParallel , instead of  DataParallel  to do multi-GPU training, even if there is only a single node.
The difference between  DistributedDataParallel  and  DataParallel  is:  DistributedDataParallel  uses multiprocessing where a process is created for each GPU,
while  DataParallel  uses multithreading. By using multiprocessing, each GPU has its dedicated process, this avoids the performance overhead caused by GIL of Python interpreter.
If you use  DistributedDataParallel , you could use torch.distributed.launch utility to launch your program, see Launch utility.
 Previous

Table of Contents

DISTRIBUTED DATA PARALLEL

The implementation of torch.nn.parallel.DistributedDataParallel evolves over time. This design note is written based on the state as of v1.4.

torch.nn.parallel.DistributedDataParallel (DDP) transparently performs distributed data parallel training. This page describes how it works and reveals implementation

details.

Example

Let us start with a simple torch.nn.parallel.DistributedDataParallel example. This example uses a torch.nn.Linear as the local model, wraps it with DDP, and then

runs one forward pass, one backward pass, and an optimizer step on the DDP model. After that, parameters on the local model will be updated, and all models on different processes should

be exactly the same.

import torch

import torch.distributed as dist

import torch.multiprocessing as mp

import torch.nn as nn

import torch.optim as optim

from torch.nn.parallel import DistributedDataParallel as DDP

def example(rank, world_size):

# create default process group

dist.init_process_group("gloo", rank=rank, world_size=world_size)

# create local model

model = nn.Linear(10, 10).to(rank)

# construct DDP model

ddp_model = DDP(model, device_ids=[rank])

# define loss function and optimizer

loss_fn = nn.MSELoss()

optimizer = optim.SGD(ddp_model.parameters(), lr=0.001)

# forward pass

outputs = ddp_model(torch.randn(20, 10).to(rank))

labels = torch.randn(20, 10).to(rank)

# backward pass

loss_fn(outputs, labels).backward()

# update parameters

optimizer.step()

def main():

world_size = 2

mp.spawn(example,

args=(world_size,),

nprocs=world_size,

join=True)

if __name__=="__main__":

main()

Internal Design

This section reveals how it works under the hood of torch.nn.parallel.DistributedDataParallel by diving into details of every step in one iteration.

Prerequisite: DDP relies on c10d ProcessGroup for communications. Hence, applications must create ProcessGroup instances before constructing DDP.

Construction: The DDP constructor takes a reference to the local module, and broadcasts state_dict() from the process with rank 0 to all other processes in the group to make

sure that all model replicas start from the exact same state. Then, each DDP process creates a local Reducer , which later will take care of the gradients synchronization during the

backward pass. To improve communication efficiency, the Reducer organizes parameter gradients into buckets, and reduces one bucket at a time. Bucket size can be configured by

setting the bucket_cap_mb argument in DDP constructor. The mapping from parameter gradients to buckets is determined at the construction time, based on the bucket size limit and

parameter sizes. Model parameters are allocated into buckets in (roughly) the reverse order of Model.parameters() from the given model. The reason for using the reverse

order is because DDP expects gradients to become ready during the backward pass in approximately that order. The figure below shows an example. Note that, the grad0 and

grad1 are in bucket1 , and the other two gradients are in bucket0 . Of course, this assumption might not always be true, and when that happens it could hurt DDP backward

speed as the Reducer cannot kick off the communication at the earliest possible time. Besides bucketing, the Reducer also registers autograd hooks during construction, one

hook per parameter. These hooks will be triggered during the backward pass when the gradient becomes ready.

WARNING

•

Forward Pass: The DDP takes the input and passes it to the local model, and then analyzes the output from the local model if find_unused_parameters is set to True . This

mode allows running backward on a subgraph of the model, and DDP finds out which parameters are involved in the backward pass by traversing the autograd graph from the model

output and marking all unused parameters as ready for reduction. During the backward pass, the Reducer would only wait for unready parameters, but it would still reduce all

buckets. Marking a parameter gradient as ready does not help DDP skip buckets as for now, but it will prevent DDP from waiting for absent gradients forever during the backward pass.

Note that traversing the autograd graph introduces extra overheads, so applications should only set find_unused_parameters to True when necessary.

Backward Pass: The backward() function is directly invoked on the loss Tensor , which is out of DDP’s control, and DDP uses autograd hooks registered at construction time to

trigger gradients synchronizations. When one gradient becomes ready, its corresponding DDP hook on that grad accumulator will fire, and DDP will then mark that parameter gradient

as ready for reduction. When gradients in one bucket are all ready, the Reducer kicks off an asynchronous allreduce on that bucket to calculate mean of gradients across all

processes. When all buckets are ready, the Reducer will block waiting for all allreduce operations to finish. When this is done, averaged gradients are written to the

param.grad field of all parameters. So after the backward pass, the grad field on the same corresponding parameter across different DDP processes should be the same.

Optimizer Step: From the optimizer’s perspective, it is optimizing a local model. Model replicas on all DDP processes can keep in sync because they all start from the same state and

they have the same averaged gradients in every iteration.

DDP requires Reducer instances on all processes to invoke allreduce in exactly the same order, which is done by always running allreduce in the bucket index order

instead of actual bucket ready order. Mismatched allreduce order across processes can lead to wrong results or DDP backward hang.

Implementation

Below are pointers to the DDP implementation components. The stacked graph shows the structure of the code.

ProcessGroup

ProcessGroup.hpp: contains the abstract API of all process group implementations. The c10d library provides 4 implementations out of the box, namely, ProcessGroupGloo,

ProcessGroupNCCL, ProcessGroupMPI, and ProcessGroupRoundRobin, where ProcessGroupRoundRobin is a composition of multiple process group instances and launches collective

communications in a round-robin manner. DistributedDataParallel uses ProcessGroup::broadcast() to send model states from the process with rank 0 to others

during initialization and ProcessGroup::allreduce() to sum gradients.

Store.hpp: assists the rendezvous service for process group instances to find each other.

DistributedDataParallel

distributed.py: is the Python entry point for DDP. It implements the initialization steps and the forward function for the nn.parallel.DistributedDataParallel module

which call into C++ libraries. Its _sync_param function performs intra-process parameter synchronization when one DDP process works on multiple devices, and it also broadcasts

model buffers from the process with rank 0 to all other processes. The inter-process parameter synchronization happens in Reducer.cpp .

comm.h: implements the coalesced broadcast helper function which is invoked to broadcast model states during initialization and synchronize model buffers before the forward pass.

reducer.h: provides the core implementation for gradient synchronization in the backward pass. It has three entry point functions:

Reducer : The constructor is called in distributed.py which registers Reducer::autograd_hook() to gradient accumulators.

autograd_hook() function will be invoked by the autograd engine when a gradient becomes ready.

prepare_for_backward() is called at the end of DDP forward pass in distributed.py . It traverses the autograd graph to find unused parameters when

find_unused_parameters is set to True in DDP constructor.

NOTE

•

剩余746页未读，继续阅读

银河舰长88

粉丝: 3
资源: 15

PyTorch 1.5.1官方文档：离线PDF版与Python API详解

pytorch中文文档-API查询与使用.pdf

Pytorch中文文档.pdf

pytorch1.5官方英文文档PythonAPI(包含书签)

pytorch1.5.1镜像

python3.7.16支持的pytorch版本

pytorch1.5.1安装

pytorch和python对应版本

pytorch官方文档

pytorch和python版本对应

python和pytorch版本对应

最新资源