SQLiteKV: An Efficient LSM-tree-based
SQLite-like Database Engine for Mobile Devices
Yuanjing Shi
Department of Computing
The Hong Kong Polytechnic University
Email: 13104584d@connect.polyu.hk
Zhaoyan Shen
Department of Computing
The Hong Kong Polytechnic University
Email: cszyshen@comp.polyu.edu.hk
Zili Shao
Department of Computing
The Hong Kong Polytechnic University
Email: cszlshao@comp.polyu.edu.hk
Abstract—SQLite has been deployed in millions of mobile
devices from web to smartphone applications on various mobile
operating systems. However, SQLite is not efficient with low
transactions per second. In this paper, we for the first time
propose a new SQLite-like database engine, called SQLiteKV,
which adopts the LSM-tree-based data structure but retains the
SQLite operation interfaces. With its SQLite interface, SQLiteKV
can be utilized by existing applications without any modification,
while providing high performance with its LSM-tree-based data
structure.
In SQLiteKV, we develop a light-weight SQLite to key-value
compiler to solve the semantic mismatch, so SQL statements can
be efficiently translated into KV operations. We also design a
novel coordination caching mechanism so query results can be
effectively cached inside SQLiteKV by alleviating the discrepancy
of data management between front-end SQLite statements and
back-end key-value data organization. We have implemented
and deployed SQLiteKV on a Google Nexus 6P smartphone.
Experiments results show that SQLiteKV outperforms SQLite
up to 6 times.
I. INTRODUCTION
SQLite is a server-less, transactional SQL database en-
gine which has been widely deployed in mobile devices.
Popular mobile applications such as messenger, email and
social network services rely on SQLite for data management.
However, due to the inefficient date organization and the poor
coordination between its database engine and the underlying
storage system [1, 2], SQLite suffers from poor transactional
performance.
Many efforts have been done to optimize the performance
of SQLite. These optimization approaches mainly fall into
two categories: (1) Investigating I/O characteristics of different
workloads of SQLite and mitigating its journaling over journal
problem [1, 2]; (2) Utilizing emerging non-volatile memory
technology, such as phase change memory, to eliminate small,
random updates to storage devices [3, 4]. Though various
mechanisms have been proposed, they all culminate with
limited performance improvement. In this work, we for the
first time propose to leverage the LSM-tree-based key-value
data structure to improve SQLite performance.
Key-value database engine, which offers higher efficiency,
scalability, availability, and usually works with simple NoSQL
schema, is becoming more and more popular. To utilize the
advantages of key-value database under SQL environments,
Apache Phoenix [5] provides a SQL-like interface and trans-
lates SQL queries into a series of key-value scans in a
NoSQL database HBase. Phoenix demonstrates outstanding
performance in data cluster environment. However, it cannot
be directly adopted by mobile devices as it is designed for
scalable and distributed computing environments with large
datasets.
There exist key-value databases on mobile device, such as
SnappyDB [6]. However, they are not widely used in mobile
devices for two major reasons. Firstly, nowadays, most mobile
applications are built with SQL statements. Lacking of the
SQL interface causes semantic mismatch between applications
and key-value databases. Thus, mobile applications need to
be redesigned to adopt key-value database, which incurs too
much overhead. Secondly, current key-value databases require
large memory footprints to maintain in-memory meta-data.
Indexes for on disk data blocks need to be maintained in mem-
ory to serve key-value queries. Such meta-data management
approach improves query performance with notable memory
occupation. In most of cloud computing environments, this
is not a critical issue. However, mobile devices are usually
constrained with limited memory space [7].
To make mobile applications benefit from the efficient
key-value database engine with SQL interfaces, in this pa-
per, we propose a novel SQLite-like database engine, called
SQLiteKV, which adopts the LSM-tree-based key-value data
structure but retains the SQLite interfaces. SQLiteKV consists
of two parts: (1) Front-end layer: A SQLite-to-KV compiler
and a novel coordination caching mechanism. (2) Back-end
layer: An LSM-tree-based key-value database engine with an
effective meta-data management scheme.
In the front-end, the compiler receives SQL queries and
translates them into the corresponding key-value operations.
A caching mechanism is designed to alleviate the discrepancy
of data organization between SQLite and key-value database.
As for the back-end, we deploy an LSM-tree-based key-
value database engine which transforms random writes to
sequential writes by aggregating multiple updates in memory
and dumping them to storage in a “batch” manner. To mitigate
the memory requirement by our key-value engine, we propose
to store exclusively the metadata for top levels of the LSM
tree in memory and leave others on disk.
We have implemented and deployed SQLiteKV on a Google
Nexus Android platform based on a key-value database-