![](https://csdnimg.cn/release/download_crawler_static/88732265/bg13.jpg)
2.2 Measuring Performance In Production
and repeatedly randomizing the placement of code, stack, and heap objects at runtime. Sadly,
these ideas didn’t go much further, and right now, this project is almost abandoned.
Personal Experience:
Remember that even running a task manager tool,
like Linux top, can affect measurements since some CPU core will be activated
and assigned to it. This might affect the frequency of the core that is running
the actual benchmark.
Having consistent measurements requires running all iterations of the benchmark with the
same conditions. However, it is not possible to replicate the exact same environment and
eliminate bias completely: there could be different temperature conditions, power delivery
spikes, neighbor processes running, etc. Chasing all potential sources of noise and variation in
the system can be a never-ending story. Sometimes it cannot be achieved, for example, when
benchmarking large distributed cloud service.
So, eliminating non-determinism in a system is helpful for well-defined, stable performance
tests, e.g., microbenchmarks. For instance, when you implement some code change and want
to know the relative speedup ratio by benchmarking two different versions of the same program.
This is a scenario where you can control most of the variables in the benchmark, including
its input, environment configuration, etc. In this situation, eliminating non-determinism
in a system helps to get a more consistent and accurate comparison. After finishing with
local testing, remember to make sure projected performance improvements were mirrored in
real-world measurements. Readers can find some examples of features that can bring noise
into performance measurements and how to disable them in Appendix A. Also, there are tools
that can set up the environment to ensure benchmarking results with a low variance; one of
them is temci
24
.
It is not recommended to eliminate system non-deterministic behavior when estimating
real-world performance improvements. Engineers should try to replicate the target system
configuration, which they are optimizing for. Introducing any artificial tuning to the system
under test will diverge results from what users of your service will see in practice. Also, any
performance analysis work, including profiling (see section 5.4), should be done on a system
that is configured similar to what will be used in a real deployment.
Finally, it’s important to keep in mind that even if a particular HW or SW feature has non-
deterministic behavior, that doesn’t mean it is considered harmful. It could give an inconsistent
result, but it is designed to improve the overall performance of the system. Disabling such a
feature might reduce the noise in microbenchmarks but make the whole suite run longer. This
might be especially important for CI/CD performance testing when there are time limits for
how long it should take to run the whole benchmark suite.
2.2 Measuring Performance In Production
When an application runs on shared infrastructure (typical in a public cloud), there usually
will be other workloads from other customers running on the same servers. With technologies
like virtualization and containers becoming more popular, public cloud providers try to fully
utilize the capacity of their servers. Unfortunately, it creates additional obstacles for measuring
performance in such an environment. Sharing resources with neighbor processes can influence
performance measurements in unpredictable ways.
Analyzing production workloads by recreating them in a lab can be tricky. Sometimes it’s not
24
Temci - https://github.com/parttimenerd/temci.
19