projects/linux·blob master · refs

454 lines · 15.1 KiB

1======================
2Linux Kernel Selftests
3======================
4
5The kernel contains a set of "self tests" under the tools/testing/selftests/
6directory. These are intended to be small tests to exercise individual code
7paths in the kernel. Tests are intended to be run after building, installing
8and booting a kernel.
9
10Kselftest from mainline can be run on older stable kernels. Running tests
11from mainline offers the best coverage. Several test rings run mainline
12kselftest suite on stable releases. The reason is that when a new test
13gets added to test existing code to regression test a bug, we should be
14able to run that test on an older kernel. Hence, it is important to keep
15code that can still test an older kernel and make sure it skips the test
16gracefully on newer releases.
17
18On some systems, hot-plug tests could hang forever waiting for cpu and
19memory to be ready to be offlined. A special hot-plug target is created
20to run the full range of hot-plug tests. In default mode, hot-plug tests run
21in safe mode with a limited scope. In limited mode, cpu-hotplug test is
22run on a single cpu as opposed to all hotplug capable cpus, and memory
23hotplug test is run on 2% of hotplug capable memory instead of 10%.
24
25kselftest runs as a userspace process. Tests that can be written/run in
26userspace may wish to use the `Test Harness`_. Tests that need to be
27run in kernel space may wish to use a `Test Module`_.
28
29Documentation on the tests
30==========================
31
32For documentation on the kselftests themselves, see:
33
34.. toctree::
35
36 testing-devices
37
38Running the selftests (hotplug tests are run in limited mode)
39=============================================================
40
41To build the tests::
42
43 $ make headers
44 $ make -C tools/testing/selftests
45
46To run the tests::
47
48 $ make -C tools/testing/selftests run_tests
49
50To build and run the tests with a single command, use::
51
52 $ make kselftest
53
54Note that some tests will require root privileges.
55
56Kselftest supports saving output files in a separate directory and then
57running tests. To locate output files in a separate directory two syntaxes
58are supported. In both cases the working directory must be the root of the
59kernel src. This is applicable to "Running a subset of selftests" section
60below.
61
62To build, save output files in a separate directory with O= ::
63
64 $ make O=/tmp/kselftest kselftest
65
66To build, save output files in a separate directory with KBUILD_OUTPUT ::
67
68 $ export KBUILD_OUTPUT=/tmp/kselftest; make kselftest
69
70The O= assignment takes precedence over the KBUILD_OUTPUT environment
71variable.
72
73The above commands by default run the tests and print full pass/fail report.
74Kselftest supports "summary" option to make it easier to understand the test
75results. Please find the detailed individual test results for each test in
76/tmp/testname file(s) when summary option is specified. This is applicable
77to "Running a subset of selftests" section below.
78
79To run kselftest with summary option enabled ::
80
81 $ make summary=1 kselftest
82
83Running a subset of selftests
84=============================
85
86You can use the "TARGETS" variable on the make command line to specify
87single test to run, or a list of tests to run.
88
89To run only tests targeted for a single subsystem::
90
91 $ make -C tools/testing/selftests TARGETS=ptrace run_tests
92
93You can specify multiple tests to build and run::
94
95 $ make TARGETS="size timers" kselftest
96
97To build, save output files in a separate directory with O= ::
98
99 $ make O=/tmp/kselftest TARGETS="size timers" kselftest
100
101To build, save output files in a separate directory with KBUILD_OUTPUT ::
102
103 $ export KBUILD_OUTPUT=/tmp/kselftest; make TARGETS="size timers" kselftest
104
105Additionally you can use the "SKIP_TARGETS" variable on the make command
106line to specify one or more targets to exclude from the TARGETS list.
107
108To run all tests but a single subsystem::
109
110 $ make -C tools/testing/selftests SKIP_TARGETS=ptrace run_tests
111
112You can specify multiple tests to skip::
113
114 $ make SKIP_TARGETS="size timers" kselftest
115
116You can also specify a restricted list of tests to run together with a
117dedicated skiplist::
118
119 $ make TARGETS="breakpoints size timers" SKIP_TARGETS=size kselftest
120
121See the top-level tools/testing/selftests/Makefile for the list of all
122possible targets.
123
124Requiring all targets to build successfully
125===========================================
126
127By default, the build succeeds as long as at least one target builds
128without error. Set ``FORCE_TARGETS=1`` to instead require every target to
129build successfully; make will abort as soon as any target fails::
130
131 $ make -C tools/testing/selftests FORCE_TARGETS=1
132
133This applies to both the ``all`` and ``install`` targets and is useful in
134CI environments where a silent partial build would be misleading.
135
136Running the full range hotplug selftests
137========================================
138
139To build the hotplug tests::
140
141 $ make -C tools/testing/selftests hotplug
142
143To run the hotplug tests::
144
145 $ make -C tools/testing/selftests run_hotplug
146
147Note that some tests will require root privileges.
148
149
150Install selftests
151=================
152
153You can use the "install" target of "make" (which calls the `kselftest_install.sh`
154tool) to install selftests in the default location (`tools/testing/selftests/kselftest_install`),
155or in a user specified location via the `INSTALL_PATH` "make" variable.
156
157To install selftests in default location::
158
159 $ make -C tools/testing/selftests install
160
161To install selftests in a user specified location::
162
163 $ make -C tools/testing/selftests install INSTALL_PATH=/some/other/path
164
165Running installed selftests
166===========================
167
168Found in the install directory, as well as in the Kselftest tarball,
169is a script named `run_kselftest.sh` to run the tests.
170
171You can simply do the following to run the installed Kselftests. Please
172note some tests will require root privileges::
173
174 $ cd kselftest_install
175 $ ./run_kselftest.sh
176
177To see the list of available tests, the `-l` option can be used::
178
179 $ ./run_kselftest.sh -l
180
181The `-c` option can be used to run all the tests from a test collection, or
182the `-t` option for specific single tests. Either can be used multiple times::
183
184 $ ./run_kselftest.sh -c size -c seccomp -t timers:posix_timers -t timer:nanosleep
185
186For other features see the script usage output, seen with the `-h` option.
187
188Timeout for selftests
189=====================
190
191Selftests are designed to be quick and so a default timeout is used of 45
192seconds for each test. Tests can override the default timeout by adding
193a settings file in their directory and set a timeout variable there to the
194configured a desired upper timeout for the test. Only a few tests override
195the timeout with a value higher than 45 seconds, selftests strives to keep
196it that way. Timeouts in selftests are not considered fatal because the
197system under which a test runs may change and this can also modify the
198expected time it takes to run a test. If you have control over the systems
199which will run the tests you can configure a test runner on those systems to
200use a greater or lower timeout on the command line as with the `-o` or
201the `--override-timeout` argument. For example to use 165 seconds instead
202one would use::
203
204 $ ./run_kselftest.sh --override-timeout 165
205
206You can look at the TAP output to see if you ran into the timeout. Test
207runners which know a test must run under a specific time can then optionally
208treat these timeouts then as fatal.
209
210Packaging selftests
211===================
212
213In some cases packaging is desired, such as when tests need to run on a
214different system. To package selftests, run::
215
216 $ make -C tools/testing/selftests gen_tar
217
218This generates a tarball in the `INSTALL_PATH/kselftest-packages` directory. By
219default, `.gz` format is used. The tar compression format can be overridden by
220specifying a `FORMAT` make variable. Any value recognized by `tar's auto-compress`_
221option is supported, such as::
222
223 $ make -C tools/testing/selftests gen_tar FORMAT=.xz
224
225`make gen_tar` invokes `make install` so you can use it to package a subset of
226tests by using variables specified in `Running a subset of selftests`_
227section::
228
229 $ make -C tools/testing/selftests gen_tar TARGETS="size" FORMAT=.xz
230
231.. _tar's auto-compress: https://www.gnu.org/software/tar/manual/html_node/gzip.html#auto_002dcompress
232
233Contributing new tests
234======================
235
236In general, the rules for selftests are
237
238 * Do as much as you can if you're not root;
239
240 * Don't take too long;
241
242 * Don't break the build on any architecture, and
243
244 * Don't cause the top-level "make run_tests" to fail if your feature is
245 unconfigured.
246
247 * The output of tests must conform to the TAP standard to ensure high
248 testing quality and to capture failures/errors with specific details.
249 The kselftest.h and kselftest_harness.h headers provide wrappers for
250 outputting test results. These wrappers should be used for pass,
251 fail, exit, and skip messages. CI systems can easily parse TAP output
252 messages to detect test results.
253
254Contributing new tests (details)
255================================
256
257 * In your Makefile, use facilities from lib.mk by including it instead of
258 reinventing the wheel. Specify flags and binaries generation flags on
259 need basis before including lib.mk. ::
260
261 CFLAGS = $(KHDR_INCLUDES)
262 TEST_GEN_PROGS := close_range_test
263 include ../lib.mk
264
265 * Use TEST_GEN_XXX if such binaries or files are generated during
266 compiling.
267
268 TEST_PROGS, TEST_GEN_PROGS mean it is the executable tested by
269 default.
270
271 TEST_GEN_MODS_DIR should be used by tests that require modules to be built
272 before the test starts. The variable will contain the name of the directory
273 containing the modules.
274
275 TEST_CUSTOM_PROGS should be used by tests that require custom build
276 rules and prevent common build rule use.
277
278 TEST_PROGS are for test shell scripts. Please ensure shell script has
279 its exec bit set. Otherwise, lib.mk run_tests will generate a warning.
280
281 TEST_CUSTOM_PROGS and TEST_PROGS will be run by common run_tests.
282
283 TEST_PROGS_EXTENDED, TEST_GEN_PROGS_EXTENDED mean it is the
284 executable which is not tested by default.
285
286 TEST_FILES, TEST_GEN_FILES mean it is the file which is used by
287 test.
288
289 TEST_INCLUDES is similar to TEST_FILES, it lists files which should be
290 included when exporting or installing the tests, with the following
291 differences:
292
293 * symlinks to files in other directories are preserved
294 * the part of paths below tools/testing/selftests/ is preserved when
295 copying the files to the output directory
296
297 TEST_INCLUDES is meant to list dependencies located in other directories of
298 the selftests hierarchy.
299
300 * First use the headers inside the kernel source and/or git repo, and then the
301 system headers. Headers for the kernel release as opposed to headers
302 installed by the distro on the system should be the primary focus to be able
303 to find regressions. Use KHDR_INCLUDES in Makefile to include headers from
304 the kernel source.
305
306 * If a test needs specific kernel config options enabled, add a config file in
307 the test directory to enable them.
308
309 e.g: tools/testing/selftests/android/config
310
311 * Create a .gitignore file inside test directory and add all generated objects
312 in it.
313
314 * Add new test name in TARGETS in selftests/Makefile::
315
316 TARGETS += android
317
318 * All changes should pass::
319
320 kselftest-{all,install,clean,gen_tar}
321 kselftest-{all,install,clean,gen_tar} O=abo_path
322 kselftest-{all,install,clean,gen_tar} O=rel_path
323 make -C tools/testing/selftests {all,install,clean,gen_tar}
324 make -C tools/testing/selftests {all,install,clean,gen_tar} O=abs_path
325 make -C tools/testing/selftests {all,install,clean,gen_tar} O=rel_path
326
327Test Module
328===========
329
330Kselftest tests the kernel from userspace. Sometimes things need
331testing from within the kernel, one method of doing this is to create a
332test module. We can tie the module into the kselftest framework by
333using a shell script test runner. ``kselftest/module.sh`` is designed
334to facilitate this process. There is also a header file provided to
335assist writing kernel modules that are for use with kselftest:
336
337- ``tools/testing/selftests/kselftest_module.h``
338- ``tools/testing/selftests/kselftest/module.sh``
339
340Note that test modules should taint the kernel with TAINT_TEST. This will
341happen automatically for modules which are in the ``tools/testing/``
342directory, or for modules which use the ``kselftest_module.h`` header above.
343Otherwise, you'll need to add ``MODULE_INFO(test, "Y")`` to your module
344source. selftests which do not load modules typically should not taint the
345kernel, but in cases where a non-test module is loaded, TEST_TAINT can be
346applied from userspace by writing to ``/proc/sys/kernel/tainted``.
347
348How to use
349----------
350
351Here we show the typical steps to create a test module and tie it into
352kselftest. We use kselftests for lib/ as an example.
353
3541. Create the test module
355
3562. Create the test script that will run (load/unload) the module
357 e.g. ``tools/testing/selftests/lib/bitmap.sh``
358
3593. Add line to config file e.g. ``tools/testing/selftests/lib/config``
360
3614. Add test script to makefile e.g. ``tools/testing/selftests/lib/Makefile``
362
3635. Verify it works:
364
365.. code-block:: sh
366
367 # Assumes you have booted a fresh build of this kernel tree
368 cd /path/to/linux/tree
369 make kselftest-merge
370 make modules
371 sudo make modules_install
372 make TARGETS=lib kselftest
373
374Example Module
375--------------
376
377A bare bones test module might look like this:
378
379.. code-block:: c
380
381 // SPDX-License-Identifier: GPL-2.0+
382
383 #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
384
385 #include "../tools/testing/selftests/kselftest_module.h"
386
387 KSTM_MODULE_GLOBALS();
388
389 /*
390 * Kernel module for testing the foobinator
391 */
392
393 static int __init test_function()
394 {
395 ...
396 }
397
398 static void __init selftest(void)
399 {
400 KSTM_CHECK_ZERO(do_test_case("", 0));
401 }
402
403 KSTM_MODULE_LOADERS(test_foo);
404 MODULE_AUTHOR("John Developer <[email protected]>");
405 MODULE_LICENSE("GPL");
406 MODULE_INFO(test, "Y");
407
408Example test script
409-------------------
410
411.. code-block:: sh
412
413 #!/bin/bash
414 # SPDX-License-Identifier: GPL-2.0+
415 $(dirname $0)/../kselftest/module.sh "foo" test_foo
416
417
418Test Harness
419============
420
421The kselftest_harness.h file contains useful helpers to build tests. The
422test harness is for userspace testing, for kernel space testing see `Test
423Module`_ above.
424
425The tests from tools/testing/selftests/seccomp/seccomp_bpf.c can be used as
426example.
427
428Example
429-------
430
431.. kernel-doc:: tools/testing/selftests/kselftest_harness.h
432 :doc: example
433
434
435Helpers
436-------
437
438.. kernel-doc:: tools/testing/selftests/kselftest_harness.h
439 :functions: TH_LOG TEST TEST_SIGNAL FIXTURE FIXTURE_DATA FIXTURE_SETUP
440 FIXTURE_TEARDOWN TEST_F TEST_HARNESS_MAIN FIXTURE_VARIANT
441 FIXTURE_VARIANT_ADD
442
443Operators
444---------
445
446.. kernel-doc:: tools/testing/selftests/kselftest_harness.h
447 :doc: operators
448
449.. kernel-doc:: tools/testing/selftests/kselftest_harness.h
450 :functions: ASSERT_EQ ASSERT_NE ASSERT_LT ASSERT_LE ASSERT_GT ASSERT_GE
451 ASSERT_NULL ASSERT_TRUE ASSERT_NULL ASSERT_TRUE ASSERT_FALSE
452 ASSERT_STREQ ASSERT_STRNE EXPECT_EQ EXPECT_NE EXPECT_LT
453 EXPECT_LE EXPECT_GT EXPECT_GE EXPECT_NULL EXPECT_TRUE
454 EXPECT_FALSE EXPECT_STREQ EXPECT_STRNE