Autotest官方文档：完整指南

需积分: 10 11 浏览量更新于2024-07-20 收藏 2.54MB PDF 举报

"Autotest英文官方文档，版本0.16.3-4-g14d34，由Autotest Team于2016年6月28日发布，包含Autotest的详细信息，包括本地客户端、远程服务器、前端、系统管理、调度器和开发者指南等内容。文档还列举了clientPackage中的多个模块，如autotest_local、base_sysinfo、base_utils等，用于自动化测试的各种功能和工具。" Autotest是一个自动化测试框架，该框架提供了全面的文档来指导用户进行测试活动。以下是文档中涵盖的一些主要知识点： 1. **General Information**: 这一部分可能包括Autotest框架的基本介绍，它的设计理念、目标以及在什么环境下运行。它可能会讲解如何安装、配置和启动Autotest系统。 2. **Local (Former Client)**: 这部分详细描述了Autotest的本地客户端部分，可能是关于如何在本地机器上设置和执行测试用例，以及客户端如何与远程服务器通信的细节。 3. **Remote (Former Server)**: 这部分可能涵盖了Autotest的远程服务器组件，它可能涉及到服务器的配置、测试结果的存储和管理，以及如何通过服务器进行远程测试执行。 4. **Frontend**: 前端部分可能涉及用户界面，如何通过图形界面或命令行界面与Autotest交互，包括创建、编辑和管理测试套件。 5. **System Administration**: 这部分可能包含系统管理员需要知道的信息，如用户权限管理、日志记录、性能监控和故障排查。 6. **Scheduler**: 调度器部分将详细说明如何计划和执行自动化测试，包括基于时间、事件或其他条件的测试调度。 7. **Developer**: 开发者指南将提供有关如何扩展和自定义Autotest的信息，包括编写新的测试脚本、模块开发和API使用。在clientPackage中，文档列举了一系列模块，每个模块都有特定的功能： - **autotest_localModule**: 本地测试相关的模块，可能包含了与本地环境交互的函数。 - **base_sysinfoModule**: 提供系统信息查询的模块。 - **base_utilsModule**: 基础通用工具模块，可能包含各种通用的辅助函数。 - **bkr_proxyModule**: 可能是一个代理模块，用于在客户端和服务器之间转发请求。 - **bkr_xmlModule**: XML处理模块，可能用于解析或生成测试结果的XML格式。 - **client_logging_configModule**: 客户端日志配置模块，负责设置和管理日志记录。 - **cmdparserModule**: 命令行参数解析模块，帮助处理命令行输入。 - **commonModule**: 共享通用功能的模块。 - **configModule**: 配置管理模块，用于读取和操作配置文件。 - **cpusetModule**: 可能涉及CPU集管理，用于分配资源。 - **fsdev_disksModule**: 文件系统设备和磁盘管理模块。 - **fsdev_mgrModule**: 文件系统设备管理模块。 - **fsinfoModule**: 文件系统信息获取模块。 - **harnessModule**: 测试套件的执行框架，可能包含测试运行时的控制逻辑。 - **harness_autoservModule, harness_beakerModule, harness_simpleModule, harness_standaloneModule**: 各种类型的测试 Harness，针对不同场景或服务进行测试。 - **jobModule**: 任务管理模块，处理测试作业的生命周期。 - **kernelModule**: 内核相关模块，可能用于获取内核信息或执行内核测试。 - **kernel_configModule, kernel_versionsModule, kernelexpandModule**: 内核配置、版本管理和扩展模块。 - **kvm_controlModule**: KVM（Kernel-based Virtual Machine）控制模块，用于虚拟化测试。 - **local_hostModule**: 本地主机相关的模块，可能涉及主机信息的获取和管理。 - **lv_utilsModule**: LVM（Logical Volume Manager）工具模块。 - **optparserModule**: 选项解析模块，处理命令行选项和参数。这些模块提供了Autotest系统的核心功能，让开发者和测试工程师能够构建和执行复杂的自动化测试流程。通过理解并熟练使用这些模块，用户可以有效地利用Autotest进行软件质量保证工作。

autotest Documentation, Release 0.16.3-4-g14d34

release/machine combination. Performance data is also analysed, in order to produce historical performance graphs

for a selection of benchmarks.

Results Publication: results are made available automatically on the TKO web site. However, this is currently a ‘polled’

model; no automatic action is taken in the face of either test failures or if performance regressions are detected, it relies

on developers to monitor the site. These failures should be actively pushed back to the community via an appropriate

publication mechanism (such as email, with links back to more detailed data).

Observed problems: When a problem (functional or performance) is observed by a developer monitoring the analysed

and published results, this is manually communicated back to the development community (normally via email). This

often results in additional patches to test, which can be manually injected into the job queues via a simple script, but

currently only by an IBM engineer. These then automatically ﬂow through with the regular releases, right through to

publication on the matrix and performance graphs allowing comparison with those releases.

TKO in Action

The regular compile and boot testing frequently shakes out bugs as the patch that carried them enters the -mm tree. By

testing multiple architectures, physical conﬁgurations, and kernel conﬁgurations we often catch untested combinations

and are able to report them to the patch author. Most often these are compile failures, or boot failures, but several

performance regressions have also been identiﬁed.

As a direct example, recently the performance of highly parallel workloads dropped off signiﬁcantly on some types

of systems, speciﬁcally with the -mm tree. This was clearly indicated by a drop off in the kernbench performance

ﬁgures. In the graph above we can see the sudden increase in elapsed time to a new plateau with 2.6.14-rc2-mm1.

12 Chapter 1. Autotest Documentation

autotest Documentation, Release 0.16.3-4-g14d34

Note the vertical error bars for each data point – doing multiple test runs inside the same job allows us to calculate

error margins, and clearly display them.

Once the problem was identiﬁed some further analysis narrowed the bug to a small number of scheduler patches

which were then also tested; these appear as the blue line (‘other’ releases) in the graph. Once the regression was

identiﬁed the patch owner was then contacted, several iterations of updated ﬁxes were then produced and tested before

a corrected patch was applied. This can be seen in the ﬁgures for 2.6.16-rc1-mm4.

The key thing to note here is that the regression never made it to the mainline kernel let alone into a released distro

kernel; user exposure was prevented. Early testing ensured that the developer was still available and retained context

on the change.

Summary

The current system is providing regular and useful testing feedback on new releases and providing ongoing trend

analysis against historical releases. It is providing the results of this testing in a public framework available to all

developers with a reasonable turn round time from release. It is also helping developers by testing on rarer hardware

combinations to which they have no access and cannot test.

However, the system is not without its problems. The underlying tests are run on a in-house testing framework (ABAT)

which is currently not in the public domain; this prevents easy transport of these tests to other testers. As a result there

is only one contributor to the result set at this time, IBM. Whilst the whole stack needs to be made open, we explain

in the next section why we have chosen to start ﬁrst with the client test harness.

The tests themselves are very limited, covering a subset of the kernel. They are run on a small number of machines,

each with a few, ﬁxed conﬁgurations. There are more tests which should be run but lack of developer input and lack

of hardware resources on which to test prevent signiﬁcant expansion.

The results analysis also does not communicate data back as effectively as it could to the community – problems

(especially performance regressions) are not as clearly isolated as they could be, and notiﬁcation is not as prompt and

clear as it could be. More data ‘folding’ needs to be done as we analyse across a multi-dimensional space of kernel

version, kernel conﬁguration, machine type, toolchain, and tests.

Client Harnesses

As we have seen, any system which will provide the required level of testing needs to form a highly distributed

system, and be able to run across a large test system base. This will necessitate a highly ﬂexible client test harness; a

key component of such a system. We have used our experiences with the IBM autobench client, and the TKO analysis

system to deﬁne requirements for such a client. This section will discuss client harnesses in general and lead on to a

discussion of the Autotest project’s new test harness.

We chose to attack the problem of the client harness ﬁrst as it seems to be the most pressing issue. With this solved,

we can share not only results, but the tests themselves more easily, and empower a wide range of individuals and

corporations to run tests easily, and share the results. By deﬁning a consistent results format, we can enable automated

collation and analysis of huge amounts of data.

Requirements / Design Goals

A viable client harness must be operable stand-alone or under an external scheduler infrastructure. Corporations

already have signiﬁcant resources invested in bespoke testing harnesses which they are not going to be willing to

waste; the client needs to be able to plug into those, and timeshare resources with them. On the other hand, some

testers and developers will have a single machine and want something simple they can install and use. This bimodal

ﬂexibility is particularly relevant where we want to be able to pass a failing test back to a patch author, and have them

reproduce the problem.

1.1. General Information 13

autotest Documentation, Release 0.16.3-4-g14d34

The client harness must be modular, with a clean internal infrastructure with simple, well deﬁned APIs. It is critical

that there is clear separation between tests, and between tests and the core, such that adding a new test cannot break

existing tests.

The client must be simple to use for newcomers, and yet provide a powerful syntax for complex testing if necessary.

Tests across multiple machines, rebooting, loops, and parallelism all need to be supported.

We want distributed scalable maintainership, the core being maintained by a core team and the tests by the contributors.

It must be able to reuse the effort that has gone into developing existing tests, by providing a simple way to encapsulate

them. Whilst open tests are obviously superior, we also need to allow the running of proprietary tests which cannot be

contributed to the central repository.

There must be a low knowledge barrier to entry for development, in order to encourage a wide variety of new develop-

ers to start contributing. In particular, we desire it to be easy to write new tests and proﬁlers, abstracting the complexity

into the core as much as possible.

We require a high level of maintainability. We want a consistent language throughout, one which is powerful and yet

easy to understand when returning to the code later, not only by the author, but also by other developers.

The client must be robust, and produce consistent results. Error handling is critical – tests that do not produce reliable

results are useless. Developers will never add sufﬁcient error checking into scripts, we must have a system which fails

on any error unless you take afﬁrmative action. Where possible it should isolate hardware or harness failures from

failures of the code under test; if something goes wrong in initialisation or during a test we need to know and reject

that test result.

Finally, we want a consistent results architecture – it is no use to run thousands of tests if we cannot understand or

parse the results. On such a scale such analysis must be fully automatable. Any results structure needs to be consistent

across tests and across machines, even if the tests are being run by a wide diversity of testers.

What Tests are Needed?

As we mentioned previously, the current published automated testing is very limited in its scope. We need very broad

testing coverage if we are going to catch a high proportion of problems before they reach the user population, and need

those tests to be freely sharable to maximise test coverage.

Most of the current testing is performed in order to verify that the machine and OS stack is ﬁt for a particular workload.

The real workload is often difﬁcult to set up, may require proprietary software, and is overly complex and does not give

sufﬁciently consistent reproducible results, so use is made of a simpliﬁed simulation of that workload encapsulated

within a test. This has the advantage of allowing these simulated workloads to be shared. We need tests in all of the

areas below:

Build tests simply check that the kernel will build. Given the massive diversity of different architectures to build for,

different conﬁguration options to build for, and different toolchains to build with, this is an extensive problem. We

need to check for warnings, as well as errors.

Static veriﬁcation tests run static analysis across the code with tools like sparse, lint, and the Stanford checker, in the

hope of ﬁnding bugs in the code without having to actually execute it.

Inbuilt debugging options (e.g. CONFIG_DEBUG_PAGEALLOC, CONFIG_DEBUG_SLAB) and fault insertion

routines (e.g. fail every 100th memory allocation, fake a disk error occasionally) offer the opportunity to allow the

kernel to test itself. These need to be a separated set of test runs from the normal functional and performance tests,

though they may reuse the same tests.

Functional or unit tests are designed to exercise one speciﬁc piece of functionality. They are used to test that piece in

isolation to ensure it meets some speciﬁcation for its expected operation. Examples of this kind of test include LTP

and Crashme.

Performance tests verify the relative performance of a particular workload on a speciﬁc system. They are used to

produce comparisons between tests to either identify performance changes, or conﬁrm none is present. Examples

14 Chapter 1. Autotest Documentation

autotest Documentation, Release 0.16.3-4-g14d34

of these include: CPU performance with Kernbench and AIM7/reaim; disk performance with bonnie, tbench and

iobench; and network performance with netperf.

Stress tests are used to identify system behaviour when pushed to the very limits of its capabilities. For example a

kernel compile executed completely in parallel creates a compile process for each ﬁle. Examples of this kind of test

include kernbench (conﬁgured appropriately), and deliberately running under heavy memory pressure such as running

with a small physical memory.

Proﬁling and debugging is another key area. If we can identify a performance regression, or some types of functional

regression, it is important for us to be able to gather data about what the system was doing at the time in order to

diagnose it. Proﬁlers range from statistical tools like readproﬁle and lockmeter to monitoring tools like vmstat and sar.

Debug tools might range from dumping out small pieces of information to full blown crashdumps.

Existing Client Harnesses

There are a number of pre-existing test harnesses in use by testers in the community. Each has its features and

problems, we touch on a few of them below.

IBM autobench is a fairly fully featured client harness, it is completely written in a combination of shell and perl. It

has support for tests containing kernel builds and system boots. However, error handling is very complex and must be

explicitly added in all cases, but does encapsulate the success or failure state of the test. The use of multiple different

languages may have been very efﬁcient for the original author, but greatly increases the maintenance overheads. Whilst

it does support running multiple tests in parallel, loops within the job control ﬁle are not supported nor is any complex

‘programming’.

OSDL STP The Open Systems Development Lab (OSDL) has the Scalable Test Platform (STP). This is a fully in-

tegrated testing environment with both a server harness and client wrapper. The client wrapper here is very simple

consisting of a number of shell support functions. Support for reboot is minimal and kernel installation is not part of

the client. There is no inbuilt handling of the meaning of results. Error checking is down to the test writer; as this is

shell it needs to be explicit else no checking is performed. It can operate in isolation and results are emailable, reboot

is currently being added.

LTP (http://ltp.sourceforge.net/) The Linux Test Project is a functional / regression test suite. It contains approximately

2900 small regression tests which are applied to the system running LTP. There is no support for building kernels or

booting them, performance testing or proﬁling. Whilst it contains a lot of useful tests, it is not a general heavy weight

testing client.

A number of other testing environments currently exist, most appear to suffer from the same basic issues, they evolved

from the simplest possible interface (a script) into a test suite; they were not designed to meet the level of requirements

we have identiﬁed and speciﬁed.

All of those we have reviewed seem to have a number of key failings. Firstly, most lack most lack bottom up error

handling. Where support exists it must be handled explicitly, testers never will think of everything. Secondly, most

lack consistent machine parsable results. There is often no consistent way to tell if a test passes, let alone get any

details from it. Lastly, due to their evolved nature they are not easy to understand nor to maintain. Fortunately it

should be reasonably easy to wrap tests such as LTP, or to port tests from STP and autobench.

Autotest - a Powerful Open Client

The Autotest open client is an attempt to address the issues we have identiﬁed. The aim is to produce a client which

is open source, implicitly handles errors, produces consistent results, is easily installable, simple to maintain and runs

either standalone or within any server harness.

Autotest is an all new client harness implementation. It is completely written in Python; chosen for a number of

reasons, it has a simple, clean and consistent syntax, it is object oriented from inception, and it has very powerful error

and exception handling. Whist no language is perfect, it meets the key design goals well, and it is open source and

widely supported.

1.1. General Information 15

autotest Documentation, Release 0.16.3-4-g14d34

As we have already indicated, there are a number of existing client harnesses; some are even open-source and therefore

a possible basis for a new client. Starting from scratch is a bold step, but we believe that the beneﬁts from a designed

approach outweigh the effort required initially to get to a workable position. Moreover, much of the existing collection

of tests can easily be imported or wrapped.

Another key goal is the portability of the tests and the results; we want to be able to run tests anywhere and to contribute

those test results back. The use of a common programming language, one with a strict syntax and semantics should

make the harness and its contained tests very portable. Good design of the harness and results speciﬁcations should

help to maintain portable results.

The autotest Test Harness

Autotest utilises an executable control ﬁle to represent and drives the users job. This control ﬁle is an executable

fragment of Python and may contain any valid Python constructs, allowing the simple representation of loops and

conditionals. Surrounding this control ﬁle is the Autotest harness, which is a set of support functions and classes to

simplify execution of tests and allow control over the job.

The key component is the job object which represents the executing job, provides access to the test environment, and

provides the framework to the job. It is responsible for the creation of the results directory, for ensuring the job output

is recorded, and for any interactions with any server harness. Below is a trivial example of a control ﬁle:

job.runtest('test1', 'kernbench', 2, 5)

One key beneﬁt of the use of a real programming language is the ability to use the full range of its control structures

in the example below we use an iterator:

for i in range(0, 5):

job.runtest('test%d' % i, 'kernbench',

2, 5)

Obviously as we are interested in testing Linux, support for building, installing and booting kernels is key. When

using this feature, we need a little added complexity to cope with the interruption to control ﬂow caused by the system

reboot. This is handled using a phase stepper which maintains ﬂow across execution interruptions, below is an example

of such a job, combining booting with iteration:

def step_init():

step_test(1)

def step_test(iteration):

if (iteration < 5):

job.next_step([step_test,

iteration + 1])

print "boot: %d" % iteration

kernel = job.distro_kernel()

kernel.boot()

Tests are represented by the test object; each test added to Autotest will be a subclass of this. This allows all tests to

share behaviour, such as creating a consistent location and layout for the results, and recording the result of the test in

a computer readable form. Below is the class deﬁnition for the kernbench benchmark. As we can see it is a subclass

of test, and as such beneﬁts from its management of the results directory hierarchy.

import test

from autotest_utils import

class kernbench(test):

16 Chapter 1. Autotest Documentation

剩余410页未读，继续阅读

孔令飞

粉丝: 638
资源: 149

Autotest官方文档：完整指南

virt-test英文官方文档

基于Java的lmc-autotest全链路自动化测试工具设计源码

autotest

Autotest

AutoTest

WebDriver AutoTest

selenium autotest

AutoTest Java

autotest02

AutoTest_SVAV

最新资源