EMNIST: an extension of MNIST to handwritten
letters
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andr
´
e van Schaik
The MARCS Institute for Brain, Behaviour and Development
Western Sydney University
Penrith, Australia 2751
Email: g.cohen@westernsydney.edu.au
Abstract—The MNIST dataset has become a standard bench-
mark for learning, classification and computer vision systems.
Contributing to its widespread adoption are the understandable
and intuitive nature of the task, its relatively small size and
storage requirements and the accessibility and ease-of-use of
the database itself. The MNIST database was derived from a
larger dataset known as the NIST Special Database 19 which
contains digits, uppercase and lowercase handwritten letters. This
paper introduces a variant of the full NIST dataset, which we
have called Extended MNIST (EMNIST), which follows the same
conversion paradigm used to create the MNIST dataset. The
result is a set of datasets that constitute a more challenging
classification tasks involving letters and digits, and that shares
the same image structure and parameters as the original MNIST
task, allowing for direct compatibility with all existing classifiers
and systems. Benchmark results are presented along with a
validation of the conversion process through the comparison of
the classification results on converted NIST digits and the MNIST
digits.
I. INTRODUCTION
The importance of good benchmarks and standardized prob-
lems cannot be understated, especially in competitive and fast-
paced fields such as machine learning and computer vision.
Such tasks provide a quick, quantitative and fair means of
analyzing and comparing different learning approaches and
techniques. This allows researchers to quickly gain insight into
the performance and peculiarities of methods and algorithms,
especially when the task is an intuitive and conceptually simple
one.
As single dataset may only cover a specific task, the
existence of a varied suite of benchmark tasks is important in
allowing a more holistic approach to assessing and characteriz-
ing the performance of an algorithm or system. In the machine
learning community, there are several standardized datasets
that are widely used and have become highly competitive.
These include the MNIST dataset [1], the CIFAR-10 and
CIFAR-100 [2] datasets, the STL-10 dataset [3], and Street
View House Numbers (SVHN) dataset [4].
Comprising a 10-class handwritten digit classification task
and first introduced in 1998, the MNIST dataset remains the
most widely known and used dataset in the computer vision
and neural networks community. However, a good dataset
needs to represent a sufficiently challenging problem to make
it both useful and to ensure its longevity [5]. This is perhaps
where MNIST has suffered in the face of the increasingly high
accuracies achieved using deep learning and convolutional
neural networks. Multiple research groups have published
accuracies above 99.7% [6]–[10], a classification accuracy at
which the dataset labeling can be called into question. Thus,
it has become more of a means to test and validate a classifi-
cation system than a meaningful or challenging benchmark.
The accessibility of the MNIST dataset has almost certainly
contributed to its widespread use. The entire dataset is rel-
atively small (by comparison to more recent benchmarking
datasets), free to access and use, and is encoded and stored
in an entirely straightforward manner. The encoding does
not make use of complex storage structures, compression, or
proprietary data formats. For this reason, it is remarkably easy
to access and include the dataset from any platform or through
any programming language.
The MNIST database is a subset of a much larger dataset
known as the NIST Special Database 19 [11]. This dataset
contains both handwritten numerals and letters and represents
a much larger and more extensive classification task, along
with the possibility of adding more complex tasks such as
writer identification, transcription tasks and case detection.
The NIST dataset, by contrast to MNIST, has remained
difficult to access and use. Driven by the higher cost and
availability of storage when it was collected, the NIST dataset
was originally stored in a remarkably efficient and compact
manner. Although source code to access the data is provided,
it remains challenging to use on modern computing platforms.
For this reason, the NIST recently released a second edition
of the NIST dataset [12]. The second edition of the dataset
is easier to access, but the structure of the dataset, and the
images contained within, differ from that of MNIST and are
not directly compatible.
The NIST dataset has been used occasionally in neural
network systems. Many classifiers make use of only the digit
classes [13], [14], whilst others tackle the letter classes as
well [15]–[18]. Each paper tackles the task of formulating the
classification tasks in a slightly different manner, varying such
fundamental aspects as the number of classes to include, the
training and testing splits, and the preprocessing of the images.
In order to bolster the use of this dataset, there is a clear
need to create a suite of well-defined datasets that thoroughly
specify the nature of the classification task and the structure of
the dataset, thereby allowing for easy and direct comparisons
arXiv:1702.05373v1 [cs.CV] 17 Feb 2017
评论0