没有合适的资源?快使用搜索试试~ 我知道了~
首页纳米级日志系统 NanoLog:速度提升百倍的高效 logging 实现
纳米级日志系统 NanoLog:速度提升百倍的高效 logging 实现
需积分: 10 4 下载量 6 浏览量
更新于2024-07-09
收藏 617KB PDF 举报
NanoLog是一个专为现代IT环境设计的革命性日志系统,它在2018年USENIX Annual Technical Conference(USENIX ATC'18)上被提出。该论文由Stephen Yang、SeoJin Park和John Ousterhout三位来自斯坦福大学的研究者共同撰写。 NanoLog的主要创新在于其惊人的性能提升,与传统日志系统如Log4j2、spdlog、Boostlog以及Windows的Event Tracing相比,它的处理速度提升了大约一个至两个数量级,达到每秒8000万条简单消息的惊人吞吐量。 在微基准测试中,NanoLog展示了极低的延迟,单个日志调用的开销平均仅为8纳秒。这意味着在日常应用中,即使是频繁的日志记录操作,也几乎感觉不到明显的影响。这得益于NanoLog对工作流程的优化,它将大部分日志处理任务移出运行时的热点路径,实现了轻量级和高效的并发处理。 这种设计理念使得NanoLog在保持类似printf接口易用性的前提下,实现了高性能和低延迟的完美平衡。它的设计理念对于实时性要求极高的系统,如云计算、大数据处理或高并发服务,具有显著的优势。此外,由于论文被公开访问,用户可以直接通过USENIX官网获取并尝试使用这个技术。 NanoLog不仅是一项技术突破,也是日志管理领域的革新,它重新定义了我们对实时、高效日志记录的理解,并可能对未来的软件开发和性能优化带来深远影响。开发者们可以关注这一研究,以期在提高系统性能的同时,保持良好的可维护性和代码简洁性。
资源详情
资源推荐
NanoLog
Preprocessor
C++
User
Sources
User C++
Object Files
Metadata
NanoLog
Combiner
Compile
Generated
Library Code
Link
Compile
User Sources with
Injected Code
Compact
Log
Decompressor
Human
Readable Log
Compile-Time
Runtime
Post
Execution
User Application
NanoLog
Compaction
Thread
User Thread Staging Buffer
User Thread Staging Buffer
User Thread Staging Buffer
NanoLog
Library
Figure 2: Overview of the NanoLog system. At compile time, the user sources are passed through the NanoLog preprocessor,
which injects optimized logging code into the application and generates a metadata file for each source file. The modified
user code is then compiled to produce C++ object files. The metadata files are aggregated by the NanoLog combiner to build
a portion of the NanoLog Library. The NanoLog library is then compiled and linked with the user object files to create an
application executable and a decompressor application. At runtime, the user application threads interact with the NanoLog
staging buffers and background compaction thread to produce a compact log. At post execution, the compact log is passed into
the decompressor to generate a final, human-readable log file.
3 Overview
NanoLog’s low latency comes from performing work
at compile-time to extract static components from log
messages and deferring formating to an off-line process.
As a result, the NanoLog system decomposes into three
components as shown in Figure 2:
Preprocessor/Combiner: extracts and catalogs static
components from log messages at compile-time, re-
places original logging statements with optimized
code, generates a unique compaction function for
each log message, and generates a function to out-
put the dictionary of static information.
Runtime Library: provides the infrastructure to buffer
log messages from multiple logging threads and
outputs the log in a compact, binary format using
the generated compaction and dictionary functions.
Decompressor: recombines the compact, binary log file
with the static information in the dictionary to either
inflate the logs to a human-readable format or run
analytics over the log contents.
Users of NanoLog interact with the system in the fol-
lowing fashion. First, they embed NANO
LOG() func-
tion calls in their C++ applications where they’d like
log messages. The function has a signature similar to
printf [17, 33] and supports all the features of printf
with the exception of the “%n” specifier, which requires
dynamic computation. Next, users integrate into their
GNUmakefiles [40] a macro provided by NanoLog that
serves as a drop-in replacement for a compiler invoca-
tion, such as g++. This macro will invoke the NanoLog
preprocessor and combiner on the user’s behalf and gen-
erate two executables: the user application linked against
the NanoLog library, and a decompressor executable to
inflate/run analytics over the compact log files. As the
application runs, a compacted log is generated. Finally,
the NanoLog decompressor can be invoked to read the
compacted log and produce a human-readable log.
4 Detailed Design
We implemented the NanoLog system for C++ appli-
cations and this section describes the design in detail.
4.1 Preprocessor
The NanoLog preprocessor interposes in the compila-
tion process of the user application (Figure 2). It pro-
cesses the user source files and generates a metadata
file and a modified source file for each user source file.
The modified source files are then compiled into object
files. Before the final link step for the application, the
NanoLog combiner reads all the metadata files and gen-
erates an additional C++ source file that is compiled into
the NanoLog Runtime Library. This library is then linked
into the modified user application.
In order to improve the performance of logging, the
NanoLog preprocessor analyzes the NANO LOG() state-
ments in the source code and replaces them with faster
code. The replacement code provides three benefits.
First, it reduces I/O bandwidth by logging only infor-
mation that cannot be known until runtime. Second,
NanoLog logs information in a compacted form. Third,
the replacement code executes much more quickly than
the original code. For example, it need not combine the
dynamic data with the format string, or convert binary
values to strings; data is logged in a binary format. The
preprocessor also extracts type and order information
from the format string (e.g., a "%d %f" format string
indicates that the log function should encode an integer
followed by a float). This allows the preprocessor to gen-
erate more efficient code that accepts and processes ex-
actly the arguments provided to the log message. Type
safety is ensured by leveraging the GNU format at-
tribute compiler extension [10].
The NanoLog preprocessor generates two functions
for each NANO
LOG() statement. The first func-
tion, record(), is invoked in lieu of the original
NANO LOG() statement. It records the dynamic in-
formation associated with the log message into an in-
USENIX Association 2018 USENIX Annual Technical Conference 337
详细设计
预处理
1.生成元数据
2.修改源文件
3.编译源文件
4.读取元数据,
生成额外的源文
件,它将被编译
为运行时库
5.这个库将连接
到用户应用中
不能使用动态
字符串
提取类型
和order
剩余15页未读,继续阅读
rtoax
- 粉丝: 2736
- 资源: 216
上传资源 快速赚钱
- 我的内容管理 展开
- 我的资源 快来上传第一个资源
- 我的收益 登录查看自己的收益
- 我的积分 登录查看自己的积分
- 我的C币 登录后查看C币余额
- 我的收藏
- 我的下载
- 下载帮助
最新资源
- JDK 17 Linux版本压缩包解压与安装指南
- C++/Qt飞行模拟器教员控制台系统源码发布
- TensorFlow深度学习实践:CNN在MNIST数据集上的应用
- 鸿蒙驱动HCIA资料整理-培训教材与开发者指南
- 凯撒Java版SaaS OA协同办公软件v2.0特性解析
- AutoCAD二次开发中文指南下载 - C#编程深入解析
- C语言冒泡排序算法实现详解
- Pointofix截屏:轻松实现高效截图体验
- Matlab实现SVM数据分类与预测教程
- 基于JSP+SQL的网站流量统计管理系统设计与实现
- C语言实现删除字符中重复项的方法与技巧
- e-sqlcipher.dll动态链接库的作用与应用
- 浙江工业大学自考网站开发与继续教育官网模板设计
- STM32 103C8T6 OLED 显示程序实现指南
- 高效压缩技术:删除重复字符压缩包
- JSP+SQL智能交通管理系统:违章处理与交通效率提升
资源上传下载、课程学习等过程中有任何疑问或建议,欢迎提出宝贵意见哦~我们会及时处理!
点击此处反馈
安全验证
文档复制为VIP权益,开通VIP直接复制
信息提交成功