Information Fusion 73 (2021) 22–71
30
X. Jiang et al.
Nonparametric model-based methods are developed to handle both
rigid and nonrigid transformations, thus showing more flexibility. Such
methods are representative by defining deformation functions in a high-
dimensional form, such as triangulated 2D mesh [318] or a kernel rep-
resentation in reproducing kernel Hilbert space with Tikhonov regular-
ization [319–321]. These methods are typically optimized through tai-
lored robust optimizers, such as Huber estimator [322], L2E [320], sup-
port vector regression [323], or EM solution in a Bayesian model [319,
321,324–327].
Another active topic is the investigation of relaxed methods for
mismatch removal. These relaxation rules are typically based on the as-
sumption of locality or piecewise consistency [328], such as grid-based
motion statistics (GMS) [329], locality preserving matching (LPM)
[330] and its varieties [331,332], feature matching using spatial clus-
tering with heavy outliers [333], and coherence-based decision bound-
aries [334]. Other strategies include using a filtering theory [335,336]
or Markov random field formulation [337]. These methods are efficient
and can handle both rigid and nonrigid image matching. Thus, they
are more acceptable for real-time required applications because they
achieve solutions quickly by using less-strict geometric constraints.
However, these relaxed methods are commonly sensitive to parameter
setting and greatly rely on a dense match set, which should well
maintain the local coherency of correct matches.
In recent years, a learning technique has been widely studied and
equipped to eliminate the outliers and/or estimate model parame-
ters through training a deep regressor [338–340] or classifier [341–
343]. In general, parameter regression and inlier/outlier classification
are trained jointly for performance enhancement [341]. Deep regres-
sor is inspired by the classical RANSAC and aims to estimate the
transformation model, such as fundamental matrix [344] or epipo-
lar geometry [339]. For example, a differentiable RANSAC, namely,
DSAC [338], is trained in an end-to-end manner by using reinforcement
learning, while other efforts are made to improve the sampling strat-
egy [339,340]. As for classifier learning, Yi et al. [341] first introduced
a pipeline called LFGC to find good feature correspondences by training
a network from a set of putative match sets together with their image
intrinsics. Ma et al. [342] proposed LMR, a general two-class classifier
learning framework for mismatch removal, by using a few training
image pairs and handcrafted geometrical representations for training
and testing. Zhang et al. [345] focused more on geometrical recovery
in their order-aware networks. Apart from learning with multilayer
perceptron (MLP), another method conducted this task with graph
convolutional networks (GCN) [346].
Learning from point data is not as easy as that on raw images by
using deep convolutional networks due to the unordered structure and
dispersed nature of the sparse points. Even so, this approach is still
worthy of attention because many recent studies have shown great
potential in using the GCN and MLP network structure together with
tailored normalization terms to learn to address sparse point-based
tasks [2,333].
2.3. Summary
Existing matching methods for multimodal images could be sys-
tematically classified into area-based and feature-based pipelines. The
area-based method would be largely affected by the choice of measure
metric. Two possible strategies are commonly applied for MMIM: one
is the use of a modality-independent measure metric such as MI and
its varieties, and the other is the reduction of different modalities into
a common domain. However, area-based methods are limited by their
large computational burden and their requirement for image pairs to
have large overlaps and undergo slight geometrical deformations. Deep
metric learning or deep transformation estimation in this framework
dramatically alleviates the predicament in matching criteria design and
iterative optimization. However, this learning strategy is still restricted
by the high image resolution and large or complex deformations, partic-
ularly with the requirement of sufficient training data. A feature-based
pipeline can efficiently address the problem in geometrical deforma-
tion. Direct feature matching, such as graph matching and point set
registration, is more suitable for image pairs containing less texture
(even binary images), or heavy modality or semantic variances. In these
cases, the image patch-based descriptor would be invalid. However,
the graph structure among potential true point correspondences may
be steadily preserved, which requires an overall corresponding matrix
to be optimized to find an optimal solution. But these direct feature-
based matching methods are limited by high computational burden
and outlier sensitivity. As for indirect feature matching, extracting a
high number and ratio of interest points then constructing distinctive
descriptions and corresponding them accurately are difficult because
of the significant nonlinear intensity variance between two modalities.
Proposing an advanced paradigm with better registration performance
in terms of both accuracy and efficiency remains an open problem for
researchers.
3. Modality-independent review
As aforementioned, matching for multimodal images can be seen as
one specific case. Apart from these challenges presented in the general
image matching task [2], the primary one in the multimodal case also
includes eliminating the domain or modality gap between two input
images. However, the natures of images would vary considerably across
different imaging sensors or data types in different research areas.
Thus, in the following, we will review these typical and latest image
matching methods that are designed for specific multimodal scenarios
under different research areas, including medical, remote sensing, and
computer vision. Simultaneously, the image natures of each modality
will be analyzed first in the corresponding parts to reinforce the under-
standing of the challenges and need for image registration of different
modalities.
3.1. Medical
The biggest family of MMIR may lie in the medical community.
With the rapid development of visualization of computer imaging tech-
niques, medical imaging has developed from statistic to dynamic, plane
to solid, and morphological to functional imaging, which has played
a significant role in modern medical diagnosis. Commonly used med-
ical imaging techniques include X-ray, ultrasonic imaging (UI or US),
computer tomography (CT), single-photon emission computed tomog-
raphy (SPECT), magnetic resonance imaging (MRI), positron emission
tomography (PET), and functional magnetic resonance imaging (fMRI),
generating medical images of different modalities. From the perspective
of medical applications, these modalities can be briefly classified into
anatomical images such as CT and MRI, and functional images such
as PET, fMRI, and SPECT [347]. Anatomical images have high spatial
resolution that can clearly display geometrical information, such as the
anatomical structures of viscera and bones, but without any functional
information. In contrast, functional images can well display the func-
tional transformation during the metabolic procedure, but the images
are usually not clear enough to reveal the structure information, re-
sulting in difficulties in anatomical structure and boundary localization.
Consequently, the complementary information from the images of these
two types needs to be combined, which first requires the image pairs
to be spatially aligned. The registration target also includes the MRI
with different weights, such as T1, T2, and proton density (PD), or
the retinal images of different angiographies such as digital subtraction
angiography (DSA), fundus photography and fluoroscopy angiography
(FA).
In the field of medical research, MMIM is a hot topic and has
been giving rise to an increasing number and diversity of registration
techniques. Several main strategies have been proposed to solve this
剩余49页未读，继续阅读
Aries_59
- 粉丝: 2
- 资源: 2
我的内容管理
展开
最新资源
- Vue实现iOS原生Picker组件：详细解析与实现思路
- Arduino蓝牙小车：参数调试与功能控制
- 百度Java面试精华：200页精选资源涵盖核心知识点
- Swift使用CoreData填坑指南：CoreData在Swift 3.0的变化
- 微距离无线充电器创新设计及其实验探索
- MTK Android平台开发全攻略：44步详解流程
- RecyclerView全面解析：替代ListView的新选择
- Android开发：自动适配中英文键盘解决方案
- Android调用WebService接口教程
- Android开发：BitmapUtil图片处理全解析与实例
- Android多线程断点续传实现详解
- PCA算法在人脸识别会议签到系统中的应用
- EventBus 3.0：Android事件总线详解与实战应用
- Android FileUtil：全面解析文件操作实用技巧与实例
- RecyclerView添加头部和尾部实战教程
- Android实现微博滑动固定顶部栏实战与优化
资源上传下载、课程学习等过程中有任何疑问或建议，欢迎提出宝贵意见哦~我们会及时处理！
点击此处反馈
