跨模态视觉音频互动生成框架CMCGAN：挖掘互补信息提升视频性能

179 浏览量更新于2024-08-28 收藏 2.61MB PDF 举报

本文献标题《CMCGAN: A Uniform Framework for Cross-Modal Visual-Audio Mutual Generation》探讨的是跨模态视觉与音频的互动生成方法。在视频中，视觉和音频是两种相互依存的模态，它们包含着共享的通用信息以及互补的内容。有效的模态融合能够显著提升相关视频任务的表现。然而，现实情况中，由于环境干扰或传感器故障，可能只有一种模态存在，而另一种被放弃或丢失。因此，通过利用两者之间共享的普遍信息和特定模态的先验知识，从现有模态中恢复缺失的模态，对于视觉任务来说具有重要的潜在价值。 CMCGAN（Cross-Modal Cycle Generative Adversarial Network）是一种创新的框架，它设计了一种统一的方法来处理这种跨模态的互动生成问题。该模型的核心在于引入了循环生成对抗网络（CycleGAN）的概念，旨在通过学习两个模态之间的映射关系，使得在一种模态数据的帮助下，可以生成缺失的另一种模态的样本。这涉及到两个主要步骤：首先，生成器网络尝试将视觉信息转化为音频，同时鉴别器网络评估生成的音频是否真实；反之，另一个生成器则负责将音频转换回视觉，鉴别器则检查重构的视觉信息是否忠实于原始信息。该研究的重要贡献包括： 1. **统一框架**：CMCGAN提供了一个通用的架构，适应不同场景下的视觉和音频模态转换，无需针对每个任务单独设计模型。 2. **循环一致性**：通过引入循环一致性损失，确保了生成的模态转换既准确又能保持原始信息的内在结构，即使在没有完全匹配的训练样本时也能进行稳健的模态转换。 3. **模态融合**：通过结合生成的互补模态信息，可以在缺乏某种原始输入的情况下增强视觉任务的性能，如视频分类、对象检测等。 4. **应用潜力**：CMCGAN不仅在理论上有创新，而且具有广泛的实际应用前景，如多媒体内容生成、增强现实、视听同步等领域。这篇研究展示了如何利用深度学习技术来解决跨模态数据的互换和补全问题，为未来的跨模态学习和数据增强提供了新的思路和技术支持。同时，它也强调了在实际场景中，特别是面对数据不完整的情况时，利用多模态信息融合的重要性。

CMCGAN: A Uniform Framework for Cross-Modal Visual-Audio Mutual

Generation

Wangli Hao

1,4

, Zhaoxiang Zhang

1,2,3,4, ∗

, He Guan

1,4

Research Center for Brain-inspired Intelligence, CASIA

National Laboratory of Pattern Recognition, CASIA

CAS Center for Excellence in Brain Science and Intelligence Technology, CAS

University of Chinese Academy of Sciences

{haowangli2015,zhaoxiang.zhang,guanhe2015}@ia.ac.cn

Abstract

Visual and audio modalities are two symbiotic modalities un-

derlying videos, which contain both common and comple-

mentary information. If they can be mined and fused sufﬁ-

ciently, performances of related video tasks can be signiﬁ-

cantly enhanced. However, due to the environmental inter-

ference or sensor fault, sometimes, only one modality exists

while the other is abandoned or missing. By recovering the

missing modality from the existing one based on the com-

mon information shared between them and the prior informa-

tion of the speciﬁc modality, great bonus will be gained for

various vision tasks. In this paper, we propose a Cross-Modal

Cycle Generative Adversarial Network (CMCGAN) to han-

dle cross-modal visual-audio mutual generation. Speciﬁcal-

ly, CMCGAN is composed of four kinds of subnetworks:

audio-to-visual, visual-to-audio, audio-to-audio and visual-

to-visual subnetworks respectively, which are organized in

a cycle architecture. CMCGAN has several remarkable ad-

vantages. Firstly, CMCGAN uniﬁes visual-audio mutual gen-

eration into a common framework by a joint corresponding

adversarial loss. Secondly, through introducing a latent vec-

tor with Gaussian distribution, CMCGAN can handle dimen-

sion and structure asymmetry over visual and audio modali-

ties effectively. Thirdly, CMCGAN can be trained end-to-end

to achieve better convenience. Beneﬁting from CMCGAN,

we develop a dynamic multimodal classiﬁcation network to

handle the modality missing problem. Abundant experiments

have been conducted and validate that CMCGAN obtains

the state-of-the-art cross-modal visual-audio generation re-

sults. Furthermore, it is shown that the generated modality

achieves comparable effects with those of original modality,

which demonstrates the effectiveness and advantages of our

proposed method.

Video mainly contains two symbiotic modalities, the visu-

al and the audio ones. Information embedded in these two

modalities owns both common and complementary informa-

tion respectively. Common information can make the trans-

lation over visual and audio modalities be possible. Mean-

while, complementary information can be adopted as the pri-

or of one modality to facilitate the associative tasks. Thus,

sufﬁcient utilization of these common and complementary

information will further boost the performances of related

∗

Corresponding author. (zhaoxiang.zhang@ia.ac.cn)

 2018, Association for the Advancement of Artiﬁcial

video tasks. However, due to the environmental disturbance

and sensor fault, one of the modality may be missing or dam-

aged, which will bring signiﬁcant inconveniences such as

silent ﬁlm and screen blurred. If we can restore the missing

modality from the remaining modality based on the cross-

modal prior, great bonus will be gained for various multime-

dia tasks and many traditional single-modal databases can

be reused in conjunction to gain better performance.

Generative Adversarial Networks (GANs) have gained

extraordinary popularity because of their ability in generat-

ing high-quality realistic samples, which is superior to oth-

er generative models. Compared to numerous work focus-

ing on static information translation, such as image-to-image

(Isola et al. 2016; Zhu et al. 2017) and text-to-image (Reed

et al. 2016), few of methods concern dynamic visual-audio

modality conversion and generation. Chen et al. ﬁrstly de-

sign Conditional GANs for cross-modal visual-audio gen-

eration. Drawbacks of their work are that the mutual gen-

eration process relies on different models and it cannot be

trained end-to-end.

Inspired by (Isola et al. 2016; Zhu et al. 2017), we propose

Cross-Modal Cycle Generative Adversarial Network (CM-

CGAN) to achieve cross-modal visual-audio mutual gener-

ation. Compared to CycleGAN, CMCGAN introduces a la-

tent vector to handle dimension and structure asymmetry a-

mong different modalities. Moreover, another two genera-

tion paths are coupled with CycleGAN to facilitate cross-

modal visual-audio translation and generation. Finally, a

joint corresponding adversarial loss is designed to unify the

visual-audio mutual generation in a common framework. In

addition, CMCGAN can be trained end-to-end to obtain bet-

ter convenience.

Beneﬁting from CMCGAN, a dynamic multimodal clas-

siﬁcation network is developed for double modalities. Once

only single modal as input, we will supplement the absent

one in the aid of CMCGAN and then perform the subsequent

classiﬁcation task. In summary, we make the following con-

tributions:

• We propose a Cross-Modal Cycle Generative Adversari-

al Network(CMCGAN) to simultaneously handle cross-

modal visual-audio mutual generation in the same model.

• We develop a joint adversarial loss to unify visual-audio

mutual generation, which makes it possible not only to

下载后可阅读完整内容，剩余7页未读，立即下载

weixin_38517122

粉丝: 7
资源: 907

跨模态视觉音频互动生成框架CMCGAN：挖掘互补信息提升视频性能

广东省惠州市惠东县2017_2018学年七年级英语下册Unit4_5复习新版人教新目标版

专题资料（2021-2022年）保安部英语.doc

计算机专业英语练习题

conflicting declaration 'std::uniform_int_distribution<> distrib' std::uniform_int_distribution<> distrib(0, ScanNums.size() - 1);

std::uniform_random_bit_generator怎么用

def generate_individual(): return random.uniform(-10, 10)

cpp random

Food(int width, int height) { std::random_device rd; std::mt19937 gen(rd()); std::uniform_int_distribution<> disx(1, width - 2); std::uniform_int_distribution<> disy(1, height - 2);这是什么意思

const waterUniforms = water.material.uniforms;的uniforms属性中有哪些参数

如何使用python生成具有品名、市场、分类、时间、价格属性的农产品的大量模拟数据,并且每一个品名有对应的分类

最新资源