Measuring Function Duration with Ftrace
Tim Bird
Sony Corporation of America
tim.bird@am.sony.com
Abstract
FTrace is a relatively new kernel tool for tracing func-
tion execution in the Linux kernel. Recently, FTrace
added the ability to trace function exit in addition to
function entry. This allows for measurement of func-
tion duration, which adds an incredibly powerful tool
for finding time-consuming areas of kernel execution.
In this paper, the current state of the art for measuring
function duration with FTrace is described. This in-
cludes recent work to add a new capability to filter the
trace data by function duration, and tools for analyzing
kernel function call graphs and visualizing kernel boot
time execution.
Introduction
Analyzing a running operating system kernel can be a
difficult task. In the 2.6.27 version of the kernel, a
powerful tracing mechanism called Ftrace was added to
mainline Linux. Ftrace provides some very nice facil-
ities for instrumenting the kernel, recording trace data,
and outputting the data to user space.
The Ftrace system provides a generic tracing framework
in the kernel, upon which several different kinds of trac-
ers can be implemented. Different kinds of tracers uti-
lize different methods of instrumenting the kernel code
and different data collection algorithms.
Ftrace supports the ability do basic function tracing,
which consists of recording information at the time of
entry to every function executed in the kernel. Addition-
ally, on some architectures, Ftrace supports the ability to
perform function graph tracing, which involves tracking
not just function entry but also function exit, and the
ability to measure function duration. This is is useful to
find performance problems and latency problems in the
kernel.
This paper presents work by the author to add function
graph tracing to the ARM architecture. This includes
a description of the mechanisms used and some of the
issues involved on the ARM architecture.
Also, this paper describes the author’s efforts to add du-
ration filtering to the function graph tracer. Even on
a relatively slow processor, the kernel executes many
thousands of functions per second. Without filtering,
the length of time that data can be captured in the trace
log without loss is very limited. By adding duration fil-
tering, it is possible to greatly extend the duration of a
trace, to capture more events of interest and to help iso-
late problem areas.
1 Overview of Ftrace Operation
1.1 Instrumentation
Ftrace operates by adding tracepoints to the Linux ker-
nel. The insertion into the Linux kernel of locations
where tracing information is recorded is referred to as
instrumentation. Instrumentation comes in two main
forms—explicitly declared tracepoints, and implicit tra-
cepoints. Explicit tracepoints consist of developer-
defined declarations which specify the location of the
tracepoint, and additional information about what data
should be collected at a particular trace site. Implicit
tracepoints are placed into the code automatically by the
compiler, either due to compiler flags or by developer
redefinition of commonly used macros.
Function tracing and function graph tracing utilize im-
plicit instrumentation. The kernel consists of many
thousands of C functions, and it would be extremely im-
practical to maintain explicit tracepoint definitions for
all of them. To instrument functions implicitly, when
the kernel is configured to support function tracing the
kernel build system adds -pg to the flags used with
the compiler. This causes the compiler to add code to
1