Tightness-aware Evaluation Protocol for Scene Text Detection
Yuliang Liu, Lianwen Jin
∗
, Zecheng Xie, Canjie Luo, Shuaitao Zhang, Lele Xie
College of Electronic Information Engineering, South China University of Technology
liu.yuliang@mail.scut.edu.cn; lianwen.jin@gmail.com
Abstract
Evaluation protocols play key role in the developmental
progress of text detection methods. There are strict require-
ments to ensure that the evaluation methods are fair, ob-
jective and reasonable. However, existing metrics exhibit
some obvious drawbacks: 1) They are not goal-oriented; 2)
they cannot recognize the tightness of detection methods;
3) existing one-to-many and many-to-one solutions involve
inherent loopholes and deficiencies. Therefore, this pa-
per proposes a novel evaluation protocol called Tightness-
aware Intersect-over-Union (TIoU) metric that could quan-
tify completeness of ground truth, compactness of detection,
and tightness of matching degree. Specifically, instead of
merely using the IoU value, two common detection behav-
iors are properly considered; meanwhile, directly using the
score of TIoU to recognize the tightness. In addition, we
further propose a straightforward method to address the an-
notation granularity issue, which can fairly evaluate word
and text-line detections simultaneously. By adopting the
detection results from published methods and general ob-
ject detection frameworks, comprehensive experiments on
ICDAR 2013 and ICDAR 2015 datasets are conducted to
compare recent metrics and the proposed TIoU metric. The
comparison demonstrated some promising new prospects,
e.g., determining the methods and frameworks for which the
detection is tighter and more beneficial to recognize. Our
method is extremely simple; however, the novelty is none
other than the proposed metric can utilize simplest but rea-
sonable improvements to lead to many interesting and in-
sightful prospects and solving most the issues of the pre-
vious metrics. The code is publicly available at https:
//github.com/Yuliang-Liu/TIoU-metric.
1. Introduction
Recent metrics for evaluating text detection have been
adopted from the object detection Pascal VOC metric [4].
However, unlike object detection, text detection tasks re-
quire the bounding box to be tighter because the primary
goal of detection is to recognize the text. Simply adopting
(a) Cutting.
(b) Pure.
(c) Outlier-GTs.
(d) Cutting & Outlier-GTs.
Figure 1. Unreasonable cases obtained using recent evaluation
metrics. (a), (b), (c), and (d) all have the same IoU of 0.66 against
the GT. Red: GT. Blue: detection.
the same IoU metric for text detection leads to the following
issues:
• As shown in Fig. 1 (a), detection over a fixed IoU
threshold with the ground truth (GT) may not com-
pletely recall the text (some characters are missed);
however, previous metrics consider that the GT has
been entirely recalled.
• As shown in Figs. 1 (b), (c), and (d), detection over
a fixed IoU threshold with the GT may still contain
background noise; however, previous metrics consider
such detection to have 100% precision.
• As shown in Fig. 1, previous metrics consider detec-
tions (a), (b), (c), and (d) to be equivalent perfect de-
tections because they all have the same IoU value that
is higher than a threshold. However, considering that
the primary goal of detection is to recognize the text,
these detections are not equivalent: 1) In (a), there is
no way to recognize the characters outside the detec-
tion bounding box; 2) in (c), it is very difficult for a
recognizer to distinguish which is the target GT; 3) the
issues pertaining to both (a) and (c) can simultaneously
occur for (d); 4) as for (b), it is easy for a normal text
recognizer to recognize the content correctly.
• Previous metrics severely rely on an IoU threshold.
4321
arXiv:1904.00813v1 [cs.CV] 27 Mar 2019