AIGC驱动的StreamingT2V：2分钟长视频生成，兼容SVD与animatediff，推动开源生态

人工智能

需积分: 1 175 浏览量更新于2024-06-14 收藏 16.25MB PDF 举报

身份认证购VIP最低享 7 折!

领优惠券(最高得80元）

资源详情

资源推荐

Figure 3. Method overview: StreamingT2V extends a video diffusion model (VDM) by the conditional attention module (CAM) as short-

term memory, and the appearance preservation module (APM) as long-term memory. CAM conditions the VDM on the previous chunk

using a frame encoder E

cond

. The attentional mechanism of CAM leads to smooth transitions between chunks and videos with high motion

amount at the same time. APM extracts from an anchor frame high-level image features and injects it to the text cross-attentions of the

VDM. APM helps to preserve object/scene features across the autogressive video generation.

4.1. Conditional Attention Module

To train a conditional network for our Streaming T2V stage,

we leverage the pre-trained power of a text-to-video model

(e.g. Modelscope [39]), as a prior for long video gener-

ation in an autoregressive manner. In the further writing

we will refer this pre-trained text-to-(short)video model as

Video-LDM. To autoregressively condition Video-LDM by

some short-term information from the previous chunk (see

Fig. 2, mid), we propose the Conditional Attention Mod-

ule (CAM), which consists of a feature extractor, and a fea-

ture injector into Video-LDM UNet, inspired by ControlNet

[46]. The feature extractor utilizes a frame-wise image en-

coder E

cond

, followed by the same encoder layers that the

Video-LDM UNet uses up to its middle layer (and initial-

ized with the UNet’s weights). For the feature injection,

we let each long-range skip connection in the UNet at-

tend to corresponding features generated by CAM via cross-

attention.

Let x denote the output of E

cond

after zero-convolution.

We use addition, to fuse x with the output of the ﬁrst

temporal transformer block of CAM. For the injection of

CAM’s features into the Video-LDM Unet, we consider

the UNet’s skip-connection features x

∈ R

b×F ×h×w×c

(see Fig.3) with batch size b. We apply spatio-temporal

group norm, and a linear projection P

on x

. Let

′

∈ R

(b·w·h)×F ×c

be the resulting tensor after reshap-

ing. We condition x

′

on the corresponding CAM fea-

ture x

CAM

∈ R

(b·w·h)×F

cond

×c

(see Fig.3), where F

cond

is the number of conditioning frames, via temporal multi-

head attention (T-MHA) [36], i.e. independently for each

spatial position (and batch). Using learnable linear maps

, P

, for queries, keys, and values, we apply T-

MHA using keys and values from x

CAM

and queries from

剩余21页未读，继续阅读

雨过朦胧影

粉丝: 142
资源: 44

会员权益专享

AIGC驱动的StreamingT2V：2分钟长视频生成，兼容SVD与animatediff，推动开源生态

一个AI开源的文生视频StreamingT2V

SVD++_SVD_SVD++代码_推荐系统svd_源码.zip

matlab生成用Monte-Carlo方法产生的传染病SI模型的感染者和易感者的数据快照矩阵并进行SVD处理可视化的代码

U,S,V = svd(sigma)

cv2.DECOMP_SVD

Lee-Carter模型+奇异值分解实现代码matlab

使用双边Jacobi方法求解SVD的Matlab代码示例

svd算法matlab代码

SVD和最小二乘法的关系

lee-carter模型SVD代码python

快速SVD+matlab

svd算法代码matlab

单边Jacobi求解SVD和双边Jacobi求解SVD方法的Matlab区别

tfidf+svd算法

单边Jacobi求解SVD和双边Jacobi求解SVD方法的Matlab代码

cv::SVD::V_T

基于单边和双边Jacobi的svd Matlab算法

L1SVD的DOA估计代码

基于双边jacobi的svd算法Matlab代码和数值模拟

L1SVD进行角度估计matlab代码

会员权益专享

最新资源