linux - tools/objtool/Documentation/klp-write-tests.txt

266 lines · 10.5 KiB

1	.. SPDX-License-Identifier: GPL-2.0
2
3	=====================================
4	Writing a test for objtool's klp code
5	=====================================
6
7	Instructions for adding a test to tools/objtool/tests/. Read
8	klp-test-design.txt first if you need to know how the harness works; this
9	document is the procedure and the rules.
10
11	Two kinds of request bring you here:
12
13	* "write a test for commit <sha>" -- a fix went in without one.
14	* "port the test at <location>, written against another harness" -- a case
15	exists elsewhere and should live in tree.
16
17	Both follow the same procedure.
18
19
20	The one rule that matters
21	=========================
22
23	A test is not finished until you have watched it fail.
24
25	Break the thing it guards -- revert the fix, or sabotage the exact line -- and
26	confirm the test fails. Then restore and confirm it passes. A test that has
27	never failed is not known to test anything, and this suite has produced
28	several that passed against deliberately broken code:
29
30	* an alternatives fixture whose empty entry pointed at its own end label rather
31	than the neighbour's replacement, so the bug it guarded made no difference;
32	* a sympos fixture where symbol-table order and address order agreed, so
33	counting and reading the linked image gave the same answer;
34	* a string fixture using a named ``char[]``, which never reached the
35	contents-hashing path because that keys on ``SHF_STRINGS``;
36	* a static array the compiler proved constant, folded to zero, and emitted no
37	relocation for -- so the two builds were byte-identical.
38
39	Every one looked correct. Say in the commit message how you verified, and if
40	you could not isolate the behaviour to a single line, say that too rather
41	than implying otherwise.
42
43
44	Procedure
45	=========
46
47	1. Read the fix. What input reaches the broken line? What is observable
48	in the output object when it misbehaves -- a missing section, a relocation
49	naming the wrong symbol, an unchanged checksum, a rejected build? If
50	nothing is observable, stop and say so; see `When to give up`_.
51
52	2. Decide where it lives. ``generic/`` unless the fixture needs
53	architecture-specific assembly or the behaviour is architecture-specific,
54	in which case ``x86/`` (or a new directory named for the architecture).
55
56	3. Write the fixture in the same directory's ``fixtures/``. Reuse an
57	existing one if it already produces the shape; add a ``-D`` knob rather
58	than copying a fixture to change one line.
59
60	4. Write the test. Assert the premise before the result -- see
61	`State the premise`_.
62
63	5. Verify by breaking the code. Then restore.
64
65	6. Run the whole suite under both compilers::
66
67	make -C tools/objtool tests
68	CC=clang make -C tools/objtool tests
69
70	7. Commit the test and its fixture together, alone. One test per commit.
71
72
73	Writing the fixture
74	===================
75
76	Fixtures are freestanding C. No kernel headers -- write out the kernel
77	structure by hand if you need one, as the existing special-section fixtures do.
78
79	Every fixture needs a ``.modinfo`` name, because klp diff reads the object's
80	module name from it::
81
82	static const char __modinfo[]
83	__attribute__((section(".modinfo"), used, aligned(1))) = "\0name=vmlinux";
84
85	Use ``MODNAME`` if the test needs to vary it. Note that most fixtures hardcode
86	``vmlinux``: passing ``-DMODNAME`` to one that does silently does nothing and
87	the test quietly becomes a vmlinux test.
88
89	The patched build is selected with ``-DPATCHED``. For a fixture with several
90	variants, gate each on both, so the original is always the baseline::
91
92	#if defined(PATCHED) && defined(WHICH_CALL)
93	r = callee_b(x);
94	#else
95	r = callee_a(x);
96	#endif
97
98	and select one per build: ``build_pair foo.c -DWHICH_CALL``. Without the
99	``defined(PATCHED)`` the flag applies to both builds and nothing differs.
100
101	Traps that have bitten before
102	-----------------------------
103
104	* The compiler optimises your fixture away. A static never written is
105	proved constant, its reads folded, and no relocation emitted. Add a writer
106	the compiler cannot see through.
107	* String literals versus named arrays. The contents-hashing path keys on
108	``SHF_STRINGS``, which the compiler sets on the mergeable section a literal
109	lands in, not on a ``char[]`` given a section of its own.
110	* Per-function sections hide movement. With the default
111	``-ffunction-sections`` every function sits at offset 0 of its own section,
112	so nothing ever moves. A test about position needs
113	``build_pair foo.c -fno-function-sections``.
114	* Special sections need boundaries. Either an entsize on the section or an
115	``ANNOTATE_DATA_SPECIAL`` annotation, or klp diff reports "missing special
116	section entsize or annotations". Their targets need real (global) symbols,
117	or it reports "failed to convert reloc sym".
118	* Prefer letting objtool generate what objtool generates.
119	``.static_call_sites``, ``.mcount_loc``, ``.ibt_endbr_seal`` and ORC come
120	from its check pass. Call ``run_objtool_check --mcount`` and let it build
121	them; a hand-written copy tests your reading of the format, not the format.
122
123	Write one by hand only when the test needs a shape objtool will not produce,
124	and say so in the fixture. generic/fixtures/static_call.c does: objtool
125	emits the site but not the ``ANNOTATE_DATA_SPECIAL`` that describes its
126	boundaries -- those come from the kernel's macros -- so a fixture which has
127	to vary whether the annotation is there writes both itself.
128
129
130	Writing the test
131	================
132
133	Start from the shortest existing test, generic/test-basic.sh.
134
135	State the premise
136	-----------------
137
138	A test which asserts only on the output passes when the compiler never emitted
139	the construct in the first place, and reads as coverage it does not have. Say
140	what the input must contain::
141
142	assert_input_section __jump_table # the fixture must produce it -> fail
143	require_input_section .kcfi_traps # this compiler may not -> skip
144
145	Prefer ``assert_``. Reach for ``require_`` only where absence genuinely
146	depends on compiler version or flags, and follow it with something
147	unconditional so the test can never be entirely vacuous.
148
149	Where a test would otherwise duplicate a sibling, assert what makes it
150	different. ``test-jump-label-module-static-key`` checks that the key really is
151	reached through its section symbol -- without that it is a second copy of
152	``test-jump-label-module-key``.
153
154	Assert both directions
155	----------------------
156
157	Check that the right thing happened and that the wrong thing did not. A klp
158	diff which clones everything is as wrong as one which clones nothing::
159
160	assert_patched changed
161	assert_not_patched untouched
162
163	Skips
164	-----
165
166	* ``gcc_only``/``clang_only`` -- a settled fact about the compiler. Declared,
167	so it reads as expected forever.
168	* ``probe_skip`` -- this toolchain did not produce the construct. Include what
169	to do about it if there is anything::
170
171	probe_skip "no matching clang/lld pair for a ThinLTO link; set THIN_CC and THIN_LD to one"
172
173	* A bare ``skip`` is counted as a failure. Never use it.
174
175	Standing in for a kernel configuration
176	--------------------------------------
177
178	``FIXTURE_CFLAGS`` is what a fixture is built with. Since objtool reads no
179	``.config``, changing these flags is how a test covers a configuration without
180	building a kernel. Two ways:
181
182	* trailing arguments to ``build_pair``/``build_one``, which win, and cover
183	anything expressible as a negation::
184
185	build_pair foo.c -fno-function-sections
186
187	* otherwise assign ``FIXTURE_CFLAGS`` before building.
188
189	Either way **say in a comment which kernel configuration the change stands in
190	for**. A flag with no stated motive is indistinguishable from a mistake.
191
192
193	What the test's comment must say
194	================================
195
196	The comment at the top is the test's justification. It should let a reader
197	decide, without archaeology, whether a skip or a failure matters. Include:
198
199	* what breaks in the running kernel if the behaviour regresses -- not the
200	mechanism, the consequence;
201	* why it is not caught otherwise, which is usually "nothing fails at build
202	time";
203	* the fix commit it guards, if there is one;
204	* anything load-bearing about the fixture that is not obvious, especially
205	anything you got wrong first.
206
207	That last point is the one people skip. If the fixture has to be built without
208	per-function sections, or the static must not be named ``__warned``, or the key
209	must be file-local -- write it down, or the next person will simplify it away.
210
211
212	Porting a test from another harness
213	===================================
214
215	Read the original's case, not its code. The other harness probably builds a
216	real kernel module; here you write freestanding C. A transliteration will
217	usually test something else.
218
219	* Work out which objtool behaviour the case exercises, then produce that shape
220	the cheapest way here.
221	* Verify by breaking the code, exactly as for a new test -- a port is not
222	correct because the original was.
223	* If the original names a fix commit, cite it.
224	* Credit the source in the commit message with the trailers the original
225	carried, followed by your own.
226
227	Sometimes the port shows the case is already covered, and sometimes it shows
228	the case cannot be reproduced here. Both are results; report them rather than
229	committing something that passes vacuously.
230
231
232	When to give up
233	===============
234
235	Some behaviour cannot be reached from a compiled fixture. Say so, with what
236	you tried, instead of committing a test that passes either way. Examples that
237	were genuinely abandoned:
238
239	* a memory leak -- needs valgrind, not an assertion on ELF;
240	* ``mkstemp`` with long paths, and other I/O edge cases;
241	* a NULL dereference reachable only through a debug path;
242	* changes made redundant by a fallback: removing the code changes no output
243	because something else already handles the case;
244	* a fix whose code has since been rewritten, so there is nothing left to
245	revert.
246
247	Also stop when the behaviour depends on something outside the fixture's
248	control -- an ELF library's handling of empty sections, or a compiler version's
249	naming of anonymous data. A test which passes for you and skips for everyone
250	else is worse than none.
251
252
253	Checklist
254	=========
255
256	Before committing:
257
258	* the test fails with the code broken, and passes with it fixed
259	* the whole suite passes under both gcc and clang
260	* the premise is asserted, not assumed
261	* both directions are asserted where that applies
262	* no bare ``skip``
263	* the fixture is in the same directory as the test
264	* the comment names the consequence, the fix commit, and anything load-bearing
265	* the commit contains one test and its fixtures, and nothing else
266	* the commit message says how you verified it