Deep Visual Analogy-Making
Scott Reed Yi Zhang Yuting Zhang Honglak Lee
University of Michigan, Ann Arbor, MI 48109, USA
{reedscot,yeezhang,yutingzh,honglak}@umich.edu
Abstract
In addition to identifying the content within a single image, relating images and
generating related images are critical tasks for image understanding. Recently,
deep convolutional networks have yielded breakthroughs in predicting image la-
bels, annotations and captions, but have only just begun to be used for generat-
ing high-quality images. In this paper we develop a novel deep network trained
end-to-end to perform visual analogy making, which is the task of transforming a
query image according to an example pair of related images. Solving this problem
requires both accurately recognizing a visual relationship and generating a trans-
formed query image accordingly. Inspired by recent advances in language mod-
eling, we propose to solve visual analogies by learning to map images to a neural
embedding in which analogical reasoning is simple, such as by vector subtraction
and addition. In experiments, our model effectively models visual analogies on
several datasets: 2D shapes, animated video game sprites, and 3D car models.
1 Introduction
Humans are good at considering “what-if?” questions about objects in their environment. What if
this chair were rotated 30 degrees clockwise? What if I dyed my hair blue? We can easily imagine
roughly how objects would look according to various hypothetical questions. However, current
generative models of images struggle to perform this kind of task without encoding significant prior
knowledge about the environment and restricting the allowed transformations.
Infer Relationship
Transform query
Figure 1: Visual analogy making concept. We learn
an encoder function f mapping images into a space
in which analogies can be performed, and a decoder
g mapping back to the image space.
Often, these visual hypothetical questions
can be effectively answered by analogi-
cal reasoning.
1
Having observed many
similar objects rotating, one could learn
to mentally rotate new objects. Having
observed objects with different colors (or
textures), one could learn to mentally re-
color (or re-texture) new objects.
Solving the analogy problem requires the
ability to identify relationships among im-
ages and transform query images accord-
ingly. In this paper, we propose to solve the problem by directly training on visual analogy comple-
tion; that is, to generate the transformed image output. Note that we do not make any claim about
how humans solve the problem, but we show that in many cases thinking by analogy is enough to
solve it, without exhaustively encoding first principles into a complex model.
We denote a valid analogy as a 4-tuple A : B :: C : D, often spoken as “A is to B as C is to D”. Given
such an analogy, there are several questions one might ask:
• A ? B :: C ? D - What is the common relationship?
• A : B ? C : D - Are A and B related in the same way that C and D are related?
• A : B :: C : ? - What is the result of applying the transformation A : B to C?
1
See [2] for a deeper philosophical discussion of analogical reasoning.
1
下载后可阅读完整内容，剩余8页未读，立即下载
kaichu2
- 粉丝: 850
- 资源: 71
我的内容管理
展开
最新资源
- 李兴华Java基础教程：从入门到精通
- U盘与硬盘启动安装教程：从菜鸟到专家
- C++面试宝典：动态内存管理与继承解析
- C++ STL源码深度解析：专家级剖析与关键技术
- C/C++调用DOS命令实战指南
- 神经网络补偿的多传感器航迹融合技术
- GIS中的大地坐标系与椭球体解析
- 海思Hi3515 H.264编解码处理器用户手册
- Oracle基础练习题与解答
- 谷歌地球3D建筑筛选新流程详解
- CFO与CIO携手：数据管理与企业增值的战略
- Eclipse IDE基础教程：从入门到精通
- Shell脚本专家宝典：全面学习与资源指南
- Tomcat安装指南：附带JDK配置步骤
- NA3003A电子水准仪数据格式解析与转换研究
- 自动化专业英语词汇精华：必备术语集锦
资源上传下载、课程学习等过程中有任何疑问或建议，欢迎提出宝贵意见哦~我们会及时处理！
点击此处反馈
