linux - tools/testing/selftests/tc-testing/README

210 lines · 7.4 KiB

1	tdc - Linux Traffic Control (tc) unit testing suite
2
3	Author: Lucas Bates - [email protected]
4
5	tdc is a Python script to load tc unit tests from a separate JSON file and
6	execute them inside a network namespace dedicated to the task.
7
8
9	REQUIREMENTS
10	------------
11
12	* Minimum Python version of 3.8.
13
14	* The kernel must have network namespace support if using nsPlugin
15
16	* The kernel must have veth support available, as a veth pair is created
17	prior to running the tests when using nsPlugin.
18
19	* The kernel must have the appropriate infrastructure enabled to run all tdc
20	unit tests. See the config file in this directory for minimum required
21	features. As new tests will be added, config options list will be updated.
22
23	* All tc-related features being tested must be built in or available as
24	modules. To check what is required in current setup run:
25	./tdc.py -c
26
27	Note:
28	In the current release, tdc run will abort due to a failure in setup or
29	teardown commands - which includes not being able to run a test simply
30	because the kernel did not support a specific feature. (This will be
31	handled in a future version - the current workaround is to run the tests
32	on specific test categories that your kernel supports)
33
34
35	BEFORE YOU RUN
36	--------------
37
38	The path to the tc executable that will be most commonly tested can be defined
39	in the tdc_config.py file. Find the 'TC' entry in the NAMES dictionary and
40	define the path.
41
42	If you need to test a different tc executable on the fly, you can do so by
43	using the -p option when running tdc:
44	./tdc.py -p /path/to/tc
45
46
47	RUNNING TDC
48	-----------
49
50	To use tdc, root privileges are required. This is because the
51	commands being tested must be run as root. The code that enforces
52	execution by root uid has been moved into a plugin (see PLUGIN
53	ARCHITECTURE, below).
54
55	Tests that use a network device should have nsPlugin.py listed as a
56	requirement for that test. nsPlugin executes all commands within a
57	network namespace and creates a veth pair which may be used in those test
58	cases. To disable execution within the namespace, pass the -N option
59	to tdc when starting a test run; the veth pair will still be created
60	by the plugin.
61
62	Running tdc without any arguments will run all tests. Refer to the section
63	on command line arguments for more information, or run:
64	./tdc.py -h
65
66	tdc will list the test names as they are being run, and print a summary in
67	TAP (Test Anything Protocol) format when they are done. If tests fail,
68	output captured from the failing test will be printed immediately following
69	the failed test in the TAP output.
70
71
72	OVERVIEW OF TDC EXECUTION
73	-------------------------
74
75	One run of tests is considered a "test suite" (this will be refined in the
76	future). A test suite has one or more test cases in it.
77
78	A test case has four stages:
79
80	- setup
81	- execute
82	- verify
83	- teardown
84
85	The setup and teardown stages can run zero or more commands. The setup
86	stage does some setup if the test needs it. The teardown stage undoes
87	the setup and returns the system to a "neutral" state so any other test
88	can be run next. These two stages require any commands run to return
89	success, but do not otherwise verify the results.
90
91	The execute and verify stages each run one command. The execute stage
92	tests the return code against one or more acceptable values. The
93	verify stage checks the return code for success, and also compares
94	the stdout with a regular expression.
95
96	Each of the commands in any stage will run in a shell instance.
97
98	Each test is an atomic unit. A test that for whatever reason spans multiple test
99	definitions is a bug.
100
101	A test that runs inside a namespace (requires "nsPlugin") will run in parallel
102	with other tests.
103
104	Tests that use netdevsim or don't run inside a namespace run serially with regards
105	to each other.
106
107
108	USER-DEFINED CONSTANTS
109	----------------------
110
111	The tdc_config.py file contains multiple values that can be altered to suit
112	your needs. Any value in the NAMES dictionary can be altered without affecting
113	the tests to be run. These values are used in the tc commands that will be
114	executed as part of the test. More will be added as test cases require.
115
116	Example:
117	$TC qdisc add dev $DEV1 ingress
118
119	The NAMES values are used to substitute into the commands in the test cases.
120
121
122	COMMAND LINE ARGUMENTS
123	----------------------
124
125	Run tdc.py -h to see the full list of available arguments.
126
127	PLUGIN ARCHITECTURE
128	-------------------
129
130	There is now a plugin architecture, and some of the functionality that
131	was in the tdc.py script has been moved into the plugins.
132
133	The plugins are in the directory plugin-lib. The are executed from
134	directory plugins. Put symbolic links from plugins to plugin-lib,
135	and name them according to the order you want them to run. This is not
136	necessary if a test case being run requires a specific plugin to work.
137
138	Example:
139
140	bjb@bee:~/work/tc-testing$ ls -l plugins
141	total 4
142	lrwxrwxrwx 1 bjb bjb 27 Oct 4 16:12 10-rootPlugin.py -> ../plugin-lib/rootPlugin.py
143	lrwxrwxrwx 1 bjb bjb 25 Oct 12 17:55 20-nsPlugin.py -> ../plugin-lib/nsPlugin.py
144	-rwxr-xr-x 1 bjb bjb 0 Sep 29 15:56 __init__.py
145
146	The plugins are a subclass of TdcPlugin, defined in TdcPlugin.py and
147	must be called "SubPlugin" so tdc can find them. They are
148	distinguished from each other in the python program by their module
149	name.
150
151	This base class supplies "hooks" to run extra functions. These hooks are as follows:
152
153	pre- and post-suite
154	pre- and post-case
155	pre- and post-execute stage
156	adjust-command (runs in all stages and receives the stage name)
157
158	The pre-suite hook receives the number of tests and an array of test ids.
159	This allows you to dump out the list of skipped tests in the event of a
160	failure during setup or teardown stage.
161
162	The pre-case hook receives the ordinal number and test id of the current test.
163
164	The adjust-command hook receives the stage id (see list below) and the
165	full command to be executed. This allows for last-minute adjustment
166	of the command.
167
168	The stages are identified by the following strings:
169
170	- pre (pre-suite)
171	- setup
172	- command
173	- verify
174	- teardown
175	- post (post-suite)
176
177
178	To write a plugin, you need to inherit from TdcPlugin in
179	TdcPlugin.py. To use the plugin, you have to put the
180	implementation file in plugin-lib, and add a symbolic link to it from
181	plugins. It will be detected at run time and invoked at the
182	appropriate times. There are a few examples in the plugin-lib
183	directory:
184
185	- rootPlugin.py:
186	implements the enforcement of running as root
187	- nsPlugin.py:
188	sets up a network namespace and runs all commands in that namespace,
189	while also setting up dummy devices to be used in testing.
190	- valgrindPlugin.py
191	runs each command in the execute stage under valgrind,
192	and checks for leaks.
193	This plugin will output an extra test for each test in the test file,
194	one is the existing output as to whether the test passed or failed,
195	and the other is a test whether the command leaked memory or not.
196	(This one is a preliminary version, it may not work quite right yet,
197	but the overall template is there and it should only need tweaks.)
198
199
200	ACKNOWLEDGEMENTS
201	----------------
202
203	Thanks to:
204
205	Jamal Hadi Salim, for providing valuable test cases
206	Keara Leibovitz, who wrote the CLI test driver that I used as a base for the
207	first version of the tc testing suite. This work was presented at
208	Netdev 1.2 Tokyo in October 2016.
209	Samir Hussain, for providing help while I dove into Python for the first time
210	and being a second eye for this code.