NanoLog
Preprocessor
C++
User
Sources
User C++
Object Files
Metadata
NanoLog
Combiner
Compile
Generated
Library Code
Link
Compile
User Sources with
Injected Code
Compact
Log
Decompressor
Human
Readable Log
Compile-Time
Runtime
Post
Execution
User Application
NanoLog
Compaction
Thread
User Thread Staging Buffer
User Thread Staging Buffer
User Thread Staging Buffer
NanoLog
Library
Figure 2: Overview of the NanoLog system. At compile time, the user sources are passed through the NanoLog preprocessor,
which injects optimized logging code into the application and generates a metadata file for each source file. The modified
user code is then compiled to produce C++ object files. The metadata files are aggregated by the NanoLog combiner to build
a portion of the NanoLog Library. The NanoLog library is then compiled and linked with the user object files to create an
application executable and a decompressor application. At runtime, the user application threads interact with the NanoLog
staging buffers and background compaction thread to produce a compact log. At post execution, the compact log is passed into
the decompressor to generate a final, human-readable log file.
3 Overview
NanoLog’s low latency comes from performing work
at compile-time to extract static components from log
messages and deferring formating to an off-line process.
As a result, the NanoLog system decomposes into three
components as shown in Figure 2:
Preprocessor/Combiner: extracts and catalogs static
components from log messages at compile-time, re-
places original logging statements with optimized
code, generates a unique compaction function for
each log message, and generates a function to out-
put the dictionary of static information.
Runtime Library: provides the infrastructure to buffer
log messages from multiple logging threads and
outputs the log in a compact, binary format using
the generated compaction and dictionary functions.
Decompressor: recombines the compact, binary log file
with the static information in the dictionary to either
inflate the logs to a human-readable format or run
analytics over the log contents.
Users of NanoLog interact with the system in the fol-
lowing fashion. First, they embed NANO
LOG() func-
tion calls in their C++ applications where they’d like
log messages. The function has a signature similar to
printf [17, 33] and supports all the features of printf
with the exception of the “%n” specifier, which requires
dynamic computation. Next, users integrate into their
GNUmakefiles [40] a macro provided by NanoLog that
serves as a drop-in replacement for a compiler invoca-
tion, such as g++. This macro will invoke the NanoLog
preprocessor and combiner on the user’s behalf and gen-
erate two executables: the user application linked against
the NanoLog library, and a decompressor executable to
inflate/run analytics over the compact log files. As the
application runs, a compacted log is generated. Finally,
the NanoLog decompressor can be invoked to read the
compacted log and produce a human-readable log.
4 Detailed Design
We implemented the NanoLog system for C++ appli-
cations and this section describes the design in detail.
4.1 Preprocessor
The NanoLog preprocessor interposes in the compila-
tion process of the user application (Figure 2). It pro-
cesses the user source files and generates a metadata
file and a modified source file for each user source file.
The modified source files are then compiled into object
files. Before the final link step for the application, the
NanoLog combiner reads all the metadata files and gen-
erates an additional C++ source file that is compiled into
the NanoLog Runtime Library. This library is then linked
into the modified user application.
In order to improve the performance of logging, the
NanoLog preprocessor analyzes the NANO LOG() state-
ments in the source code and replaces them with faster
code. The replacement code provides three benefits.
First, it reduces I/O bandwidth by logging only infor-
mation that cannot be known until runtime. Second,
NanoLog logs information in a compacted form. Third,
the replacement code executes much more quickly than
the original code. For example, it need not combine the
dynamic data with the format string, or convert binary
values to strings; data is logged in a binary format. The
preprocessor also extracts type and order information
from the format string (e.g., a "%d %f" format string
indicates that the log function should encode an integer
followed by a float). This allows the preprocessor to gen-
erate more efficient code that accepts and processes ex-
actly the arguments provided to the log message. Type
safety is ensured by leveraging the GNU format at-
tribute compiler extension [10].
The NanoLog preprocessor generates two functions
for each NANO
LOG() statement. The first func-
tion, record(), is invoked in lieu of the original
NANO LOG() statement. It records the dynamic in-
formation associated with the log message into an in-
USENIX Association 2018 USENIX Annual Technical Conference 337
详细设计
预处理
1.生成元数据
2.修改源文件
3.编译源文件
4.读取元数据,
生成额外的源文
件,它将被编译
为运行时库
5.这个库将连接
到用户应用中
不能使用动态
字符串
提取类型
和order