can also be used to increase the robustness of this iterative
refinement.
2.2. Global groupwise reassembly
Pairwise matching of fragments may result in wrong
matching (usually due to either local minima, similar
geometry/texture information of nonadjacent fragments,
or erosion-caused of geometry/texture dissimilarity of
adjacent fragments). An examination of multiple pieces
globally can better eliminate such error or ambiguity. In
[11], the author proposed a global method to solve jigsaw
puzzles and their algorithm can solve puzzle with a large
number of pieces. But their algorithm requires the preli-
minary knowledge on the shapes of the puzzle and thus
is not suitable for general image reassembly. In [12], all
matching pairs are examined and the matching with the
highest score is selected, the corresponding pair is merged
into a new piece. This process is repeated until only one
piece is left. Since the incorrect matching pair can easily
be picked during this merging process, a backtracking
strategy is incorporated in the process. But since the incor-
rect matching occurs very often, this merging process
becomes computationally expensive and thus not practical
in real applications. In [4], this best-first search strategy
with backtracking is still used. They merge three pieces
at a time based on the assumptions that generic breaks
in puzzles only produce ‘‘T’’ and ‘‘Y’’ junctions. Similarly,
in [13], the matching with triple junctions is granted with
much higher score, thus is more likely to be selected in the
global search process. Then in [6], the local reconstruction
is resolved within the process of global search, which
demonstrates better global results.
2.3. Reassembly of 3D geometries
The reassembly of 3D objects is another closely related
topic. One approach is to project 3D objects to 2D and
match their 2D planar boundaries using 2D methods
[14]. The boundaries are usually detected through the dif-
ference of surface texture or geometry on the intact surface
and fracture surface or manually identified. These methods
work well for thin shells whose break surface geometry
can usually be ignored and treated as curves. Cooper
et al. [15] assembled 3D pot fragments by matching the
break-curves and sherd normals. Willis and Cooper [16]
improved this approach in pots assembly by applying
Bayesian analysis through a semi-automatic matching.
Their experiments demonstrated to be very effective in
pottery assembly. However, this approach is for geometric
objects that are axially symmetric and geometrically sim-
ple/smooth, such as pots and vases. For more general 3D
solid models, in [17,18], fragments are segmented into
multiple fracture surfaces and matched via depth maps;
in [19], a 4-step framework was proposed, also relying on
fracture region matching. Both of these methods require
effective fracture region segmentation on each 3D frag-
ment, which is usually highly nontrivial especially when
the fragment is small. In [20,21], templates are used to
guide initial reassembly of thin-shell fragments, then the
initial compositions are refined through break-curve
analysis. These algorithms were used to restore/repair
fragmented human skulls [22,23], which have subtle and
complex geometric details, but available template models
with similar global structure.
3. Problem formulation and experimental settings
Given a set of fragments F ¼fF
i
g; i ¼ 1; ...; n; F
i
R
2
,
which are from a torn image I, we want to solve the trans-
formation for each fragment F
i
, i.e., a 3 3 matrix T
i
,so
that these transformations compose all the fragments back
into the original image I
0
¼
S
T
i
ðF
i
Þ.
In our experiments, each image is torn into multiple
fragmented pieces. We scan all the fragments. For each
fragment F
i
we obtain a digital image scan I
i
which has a
colorful region appeared on a white background. We can
segment such a region F
i
out from the white background
by either using existing segmentation algorithms or by
simply removing the exterior regions composed of con-
nected white pixels. We use these extracted fragments as
the input to the reassembly algorithm.
4. Pairwise matching between fragments
This section elaborates our algorithm of computing the
pairwise matching between two image fragments. First,
the boundary of each fragment F
i
is extracted from the
scan image I
i
; the boundary is a 2D curve loop. Then, the
pairwise matching between two image fragments can
reduce to a partial curve matching problem. Our pairwise
matching algorithm utilizes both the color and geometry
information. Unlike existing algorithms that directly use
dynamic programming [5] or geometric hashing [8] for
curve matching, we first cluster each boundary curve loop
into several segments using both color and geometry, then
match the clusters with the similar color and geometry.
Intuitively, if two boundary curves are matched, then they
should share a common sub-curve with the similar color.
Therefore, by simply searching through all the matches
between the similar segments we can evaluate all the pos-
sible matches between two curves.
The four-step pairwise matching pipeline is formulated
as follows: first, cluster the boundary of each fragment into
curve segments; second, align fragment pairs by comput-
ing transformations between their curve segments; third,
refine the matchings by an accordingly modified ICP algo-
rithm; finally, evaluate and sort the matching scores of all
computed pairwise transformations. The detail algorithms
are illustrated in the following Sections 4.1–4.4.
Notations for pairwise matching. Given two boundary
curves C
i
and C
j
, we aim to calculate a transformation
T
ij
ðC
i
Þ that matches C
i
to C
j
. A desirable transformation
T
ij
should align a portion of C
i
with that of C
j
, indicating
the two fragments are adjacent and share one or multiple
common curve segments. A curve is partitioned into multi-
ple curve segments, or also called clusters here. A cluster or
a curve segment in C
i
is denoted as S
ik
. The length of S
ik
is
denoted as lðS
ik
Þ. The average color of the S
ik
is denoted as
cðS
ik
Þ.
486 K. Zhang, X. Li / Graphical Models 76 (2014) 484–495