Transformer架构的扩散模型：Sora与图像生成新突破

需积分: 0 61 浏览量更新于2024-06-16 收藏 41.79MB PDF 举报

"这篇资源是关于Open AI的Sora的原始论文，主要探讨了一种基于Transformer架构的新式扩散模型，即Diffusion Transformers (DiTs)。该模型在图像生成领域表现出卓越的质量，尤其是在处理ImageNet数据集时。" 在当前的计算机科学领域，人工智能（AI）的发展已经引领了诸多创新，尤其是在深度学习和生成模型方面。扩散模型作为其中的一种，通过逐步“扩散”噪声来生成高逼真的图像，已经在图像生成任务中取得了显著的进步。这篇论文的核心在于介绍了一种新的扩散模型变体——使用Transformer作为基础架构的模型。 Transformer是一种最初用于自然语言处理（NLP）的序列模型，由Vaswani等人在2017年提出，因其自注意力机制而广受赞誉。在本研究中，作者将Transformer的这种优势应用到图像生成领域，构建了所谓的Latent Diffusion Models，它们以潜在空间的补丁为输入，替代了传统的U-Net结构。U-Net是一种常用于图像分割和生成任务的卷积神经网络，其双路径结构便于信息的传递和恢复。论文中提到，Diffusion Transformers（DiTs）在前向传播复杂度（以GFLops衡量）上具有良好的可扩展性。随着Transformer的深度、宽度增加或输入令牌数量的增多，模型的计算复杂度增加，但其FID（Fréchet Inception Distance）分数却降低。FID是评估生成图像质量的一个重要指标，分数越低表示生成的图像与真实图像越接近。实验结果显示，最大的DiT-XL/2模型在类条件下的ImageNet 512×512和256×256基准测试中超越了所有先前的扩散模型，特别是在256×256分辨率下达到了2.27的FID，这是目前的最优水平。这表明，基于Transformer的扩散模型在图像生成任务上展现出强大的潜力，且有很好的扩展性，有望在未来的AI研究和应用中扮演重要角色。此外，论文还深入分析了这些模型在不同规模下的性能，以及如何通过调整模型参数来优化生成结果。这不仅对AI研究人员有极大的启发，也为开发者提供了新的工具和技术，以创建更高质量的图像生成系统。这项工作为人工智能领域的扩散模型和Transformer架构的应用开辟了新的方向。

100K 200K 300K 400K

Training Steps

100

FID-50K

XL/2 In-Context

XL/2 Cross-Attention

XL/2 adaLN

XL/2 adaLN-Zero

Figure 5. Comparing different conditioning strategies. adaLN-

Zero outperforms cross-attention and in-context conditioning at all

stages of training.

– Cross-attention block. We concatenate the embeddings

of t and c into a length-two sequence, separate from

the image token sequence. The transformer block is

modiﬁed to include an additional multi-head cross-

attention layer following the multi-head self-attention

block, similar to the original design from Vaswani et

al. [60], and also similar to the one used by LDM for

conditioning on class labels. Cross-attention adds the

most Gﬂops to the model, roughly a 15% overhead.

– Adaptive layer norm (adaLN) block. Following

the widespread usage of adaptive normalization lay-

ers [40] in GANs [2, 28] and diffusion models with U-

Net backbones [9], we explore replacing standard layer

norm layers in transformer blocks with adaptive layer

norm (adaLN). Rather than directly learn dimension-

wise scale and shift parameters γ and β, we regress

them from the sum of the embedding vectors of t and

c. Of the three block designs we explore, adaLN adds

the least Gﬂops and is thus the most compute-efﬁcient.

It is also the only conditioning mechanism that is re-

stricted to apply the same function to all tokens.

– adaLN-Zero block. Prior work on ResNets has found

that initializing each residual block as the identity

function is beneﬁcial. For example, Goyal et al. found

that zero-initializing the ﬁnal batch norm scale factor γ

in each block accelerates large-scale training in the su-

pervised learning setting [13]. Diffusion U-Net mod-

els use a similar initialization strategy, zero-initializing

the ﬁnal convolutional layer in each block prior to any

residual connections. We explore a modiﬁcation of

the adaLN DiT block which does the same. In addi-

tion to regressing γ and β, we also regress dimension-

wise scaling parameters α that are applied immediately

prior to any residual connections within the DiT block.

Model Layers N Hidden size d Heads Gﬂops (I=32, p=4)

DiT-S 12 384 6 1.4

DiT-B 12 768 12 5.6

DiT-L 24 1024 16 19.7

DiT-XL 28 1152 16 29.1

Table 1. Details of DiT models. We follow ViT [10] model con-

ﬁgurations for the Small (S), Base (B) and Large (L) variants; we

also introduce an XLarge (XL) conﬁg as our largest model.

We initialize the MLP to output the zero-vector for all

α; this initializes the full DiT block as the identity

function. As with the vanilla adaLN block, adaLN-

Zero adds negligible Gﬂops to the model.

We include the in-context, cross-attention, adaptive layer

norm and adaLN-Zero blocks in the DiT design space.

Model size. We apply a sequence of N DiT blocks, each

operating at the hidden dimension size d. Following ViT,

we use standard transformer conﬁgs that jointly scale N,

d and attention heads [10, 63]. Speciﬁcally, we use four

conﬁgs: DiT-S, DiT-B, DiT-L and DiT-XL. They cover a

wide range of model sizes and ﬂop allocations, from 0.3

to 118.6 Gﬂops, allowing us to gauge scaling performance.

Table 1 gives details of the conﬁgs.

We add B, S, L and XL conﬁgs to the DiT design space.

Transformer decoder. After the ﬁnal DiT block, we need

to decode our sequence of image tokens into an output noise

prediction and an output diagonal covariance prediction.

Both of these outputs have shape equal to the original spa-

tial input. We use a standard linear decoder to do this; we

apply the ﬁnal layer norm (adaptive if using adaLN) and lin-

early decode each token into a p×p×2C tensor, where C is

the number of channels in the spatial input to DiT. Finally,

we rearrange the decoded tokens into their original spatial

layout to get the predicted noise and covariance.

The complete DiT design space we explore is patch size,

transformer block architecture and model size.

4. Experimental Setup

We explore the DiT design space and study the scaling

properties of our model class. Our models are named ac-

cording to their conﬁgs and latent patch sizes p; for exam-

ple, DiT-XL/2 refers to the XLarge conﬁg and p = 2.

Training. We train class-conditional latent DiT models at

256 × 256 and 512 × 512 image resolution on the Ima-

geNet dataset [31], a highly-competitive generative mod-

eling benchmark. We initialize the ﬁnal linear layer with

zeros and otherwise use standard weight initialization tech-

niques from ViT. We train all models with AdamW [29,33].

剩余24页未读，继续阅读

WEL测试

粉丝: 4151
资源: 47

Transformer架构的扩散模型：Sora与图像生成新突破

爆火Sora背后的技术，UC伯克利最新《扩散模型》课程视频，210页ppt .pdf

电子行业周报：OpenAI重磅发布文生视频AI大模型Sora，持续关注AI产业链.pdf

人工智能Sora提示词案例

Sora大模型网络构成

chatgpt sora

sora如何下载使用

sora的使用和安装

国内怎么使用sora

rabbitMQ 什么是订阅模型

rabbitMQ 订阅模型分类是什么

最新资源