Real-Time Neural Style Transfer for Videos
Haozhi Huang
†‡∗
Hao Wang
‡
Wenhan Luo
‡
Lin Ma
‡
Wenhao Jiang
‡
Xiaolong Zhu
‡
Zhifeng Li
‡
Wei Liu
‡∗
†
Tsinghua University
‡
Tencent AI Lab
∗
Correspondence: huanghz08@gmail.com wliu@ee.columbia.edu
Abstract
Recent research endeavors have shown the potential of
using feed-forward convolutional neural networks to ac-
complish fast style transfer for images. In this work, we
take one step further to explore the possibility of exploiting
a feed-forward network to perform style transfer for videos
and simultaneously maintain temporal consistency among
stylized video frames. Our feed-forward network is trained
by enforcing the outputs of consecutive frames to be both
well stylized and temporally consistent. More specifically,
a hybrid loss is proposed to capitalize on the content in-
formation of input frames, the style information of a given
style image, and the temporal information of consecutive
frames. To calculate the temporal loss during the train-
ing stage, a novel two-frame synergic training mechanism
is proposed. Compared with directly applying an existing
image style transfer method to videos, our proposed method
employs the trained network to yield temporally consistent
stylized videos which are much more visually pleasant. In
contrast to the prior video style transfer method which relies
on time-consuming optimization on the fly, our method runs
in real time while generating competitive visual results.
1. Introduction
Recently, great progress has been achieved by apply-
ing deep convolutional neural networks (CNNs) to image
transformation tasks, where a feed-forward CNN receives
an input image, possibly equipped with some auxiliary in-
formation, and transforms it into a desired output image.
This kind of tasks includes style transfer [12, 27], seman-
tic segmentation [19], super-resolution [12, 7], coloriza-
tion [11, 31], etc.
A natural way to extend image processing techniques to
videos is to perform a certain image transformation frame
by frame. However, this scheme inevitably brings temporal
inconsistencies and thus causes severe flicker artifacts. The
second row in Fig. 1 shows an example of directly applying
the feed-forward network based image style transfer method
Style Image
Figure 1: Video style transfer without and with temporal
consistency. The first row displays two consecutive input
frames and a given style image. The second row shows
the stylized results generated by the method of Johnson
et al. [12]. The zoom-in regions in the middle show that
the stylized patterns are of different appearances between
the consecutive frames, which creates flicker artifacts. The
third row shows the stylized results of our method, where
the stylized patterns maintain the same appearance.
of Johnson et al . [12] to videos. It can be observed that the
zoom-in content marked by white rectangles is stylized in-
to different appearances between two consecutive frames,
therefore creating flicker artifacts. The reason is that slight
variations between adjacent video frames may be amplified
by the frame-based feed-forward network and thus result in
obviously different stylized frames. In the literature, one
solution to retain temporal coherence after video transfor-
mation is to explicitly consider temporal consistency during
the frame generation or optimization process [18, 1, 14, 22].
While effective, they are case-specific methods and thus
cannot be easily generalized to other problems. Among
them, the method of Ruder et al. [22] is specifically de-
signed for video style transfer. However, it relies on time-
consuming optimization on the fly, and takes about three
minutes to process a single frame even with pre-computed
optical flows. Another solution to maintaining temporal
consistency is to apply post-processing [15, 2]. A draw-
783
下载后可阅读完整内容，剩余8页未读，立即下载
chenbtravel
- 粉丝: 25
- 资源: 296
我的内容管理
展开
最新资源
- 最优条件下三次B样条小波边缘检测算子研究
- 深入解析：wav文件格式结构
- JIRA系统配置指南：代理与SSL设置
- 入门必备：电阻电容识别全解析
- U盘制作启动盘：详细教程解决无光驱装系统难题
- Eclipse快捷键大全：提升开发效率的必备秘籍
- C++ Primer Plus中文版：深入学习C++编程必备
- Eclipse常用快捷键汇总与操作指南
- JavaScript作用域解析与面向对象基础
- 软通动力Java笔试题解析
- 自定义标签配置与使用指南
- Android Intent深度解析：组件通信与广播机制
- 增强MyEclipse代码提示功能设置教程
- x86下VMware环境中Openwrt编译与LuCI集成指南
- S3C2440A嵌入式终端电源管理系统设计探讨
- Intel DTCP-IP技术在数字家庭中的内容保护
资源上传下载、课程学习等过程中有任何疑问或建议，欢迎提出宝贵意见哦~我们会及时处理！
点击此处反馈
