| 1 | |
| 2 |
|
| 3 | ===================================================
|
| 4 | The Kernel Test Anything Protocol (KTAP), version 1
|
| 5 | ===================================================
|
| 6 |
|
| 7 | TAP, or the Test Anything Protocol is a format for specifying test results used
|
| 8 | by a number of projects. Its website and specification are found at this `link
|
| 9 | <https://testanything.org/>`_. The Linux Kernel largely uses TAP output for test
|
| 10 | results. However, Kernel testing frameworks have special needs for test results
|
| 11 | which don't align with the original TAP specification. Thus, a "Kernel TAP"
|
| 12 | (KTAP) format is specified to extend and alter TAP to support these use-cases.
|
| 13 | This specification describes the generally accepted format of KTAP as it is
|
| 14 | currently used in the kernel.
|
| 15 |
|
| 16 | KTAP test results describe a series of tests (which may be nested: i.e., test
|
| 17 | can have subtests), each of which can contain both diagnostic data -- e.g., log
|
| 18 | lines -- and a final result. The test structure and results are
|
| 19 | machine-readable, whereas the diagnostic data is unstructured and is there to
|
| 20 | aid human debugging.
|
| 21 |
|
| 22 | KTAP output is built from four different types of lines:
|
| 23 |
|
| 24 | - Version lines
|
| 25 | - Plan lines
|
| 26 | - Test case result lines
|
| 27 | - Diagnostic lines
|
| 28 |
|
| 29 | In general, valid KTAP output should also form valid TAP output, but some
|
| 30 | information, in particular nested test results, may be lost. Also note that
|
| 31 | there is a stagnant draft specification for TAP14, KTAP diverges from this in
|
| 32 | a couple of places (notably the "Subtest" header), which are described where
|
| 33 | relevant later in this document.
|
| 34 |
|
| 35 | Version lines
|
| 36 | -------------
|
| 37 |
|
| 38 | All KTAP-formatted results begin with a "version line" which specifies which
|
| 39 | version of the (K)TAP standard the result is compliant with.
|
| 40 |
|
| 41 | For example:
|
| 42 |
|
| 43 | - "KTAP version 1"
|
| 44 | - "TAP version 13"
|
| 45 | - "TAP version 14"
|
| 46 |
|
| 47 | Note that, in KTAP, subtests also begin with a version line, which denotes the
|
| 48 | start of the nested test results. This differs from TAP14, which uses a
|
| 49 | separate "Subtest" line.
|
| 50 |
|
| 51 | While, going forward, "KTAP version 1" should be used by compliant tests, it
|
| 52 | is expected that most parsers and other tooling will accept the other versions
|
| 53 | listed here for compatibility with existing tests and frameworks.
|
| 54 |
|
| 55 | Plan lines
|
| 56 | ----------
|
| 57 |
|
| 58 | A test plan provides the number of tests (or subtests) in the KTAP output.
|
| 59 |
|
| 60 | Plan lines must follow the format of "1..N" where N is the number of tests or subtests.
|
| 61 | Plan lines follow version lines to indicate the number of nested tests.
|
| 62 |
|
| 63 | While there are cases where the number of tests is not known in advance -- in
|
| 64 | which case the test plan may be omitted -- it is strongly recommended one is
|
| 65 | present where possible.
|
| 66 |
|
| 67 | Test case result lines
|
| 68 | ----------------------
|
| 69 |
|
| 70 | Test case result lines indicate the final status of a test.
|
| 71 | They are required and must have the format:
|
| 72 |
|
| 73 | .. code-block:: none
|
| 74 |
|
| 75 | <result> <number> [<description>][ # [<directive>] [<diagnostic data>]]
|
| 76 |
|
| 77 | The result can be either "ok", which indicates the test case passed,
|
| 78 | or "not ok", which indicates that the test case failed.
|
| 79 |
|
| 80 | <number> represents the number of the test being performed. The first test must
|
| 81 | have the number 1 and the number then must increase by 1 for each additional
|
| 82 | subtest within the same test at the same nesting level.
|
| 83 |
|
| 84 | The description is a description of the test, generally the name of
|
| 85 | the test, and can be any string of characters other than # or a
|
| 86 | newline. The description is optional, but recommended.
|
| 87 |
|
| 88 | The directive and any diagnostic data is optional. If either are present, they
|
| 89 | must follow a hash sign, "#".
|
| 90 |
|
| 91 | A directive is a keyword that indicates a different outcome for a test other
|
| 92 | than passed and failed. The directive is optional, and consists of a single
|
| 93 | keyword preceding the diagnostic data. In the event that a parser encounters
|
| 94 | a directive it doesn't support, it should fall back to the "ok" / "not ok"
|
| 95 | result.
|
| 96 |
|
| 97 | Currently accepted directives are:
|
| 98 |
|
| 99 | - "SKIP", which indicates a test was skipped (note the result of the test case
|
| 100 | result line can be either "ok" or "not ok" if the SKIP directive is used)
|
| 101 | - "TODO", which indicates that a test is not expected to pass at the moment,
|
| 102 | e.g. because the feature it is testing is known to be broken. While this
|
| 103 | directive is inherited from TAP, its use in the kernel is discouraged.
|
| 104 | - "XFAIL", which indicates that a test is expected to fail. This is similar
|
| 105 | to "TODO", above, and is used by some kselftest tests.
|
| 106 | - “TIMEOUT”, which indicates a test has timed out (note the result of the test
|
| 107 | case result line should be “not ok” if the TIMEOUT directive is used)
|
| 108 | - “ERROR”, which indicates that the execution of a test has failed due to a
|
| 109 | specific error that is included in the diagnostic data. (note the result of
|
| 110 | the test case result line should be “not ok” if the ERROR directive is used)
|
| 111 |
|
| 112 | The diagnostic data is a plain-text field which contains any additional details
|
| 113 | about why this result was produced. This is typically an error message for ERROR
|
| 114 | or failed tests, or a description of missing dependencies for a SKIP result.
|
| 115 |
|
| 116 | The diagnostic data field is optional, and results which have neither a
|
| 117 | directive nor any diagnostic data do not need to include the "#" field
|
| 118 | separator.
|
| 119 |
|
| 120 | Example result lines include::
|
| 121 |
|
| 122 | ok 1 test_case_name
|
| 123 |
|
| 124 | The test "test_case_name" passed.
|
| 125 |
|
| 126 | ::
|
| 127 |
|
| 128 | not ok 1 test_case_name
|
| 129 |
|
| 130 | The test "test_case_name" failed.
|
| 131 |
|
| 132 | ::
|
| 133 |
|
| 134 | ok 1 test # SKIP necessary dependency unavailable
|
| 135 |
|
| 136 | The test "test" was SKIPPED with the diagnostic message "necessary dependency
|
| 137 | unavailable".
|
| 138 |
|
| 139 | ::
|
| 140 |
|
| 141 | not ok 1 test # TIMEOUT 30 seconds
|
| 142 |
|
| 143 | The test "test" timed out, with diagnostic data "30 seconds".
|
| 144 |
|
| 145 | ::
|
| 146 |
|
| 147 | ok 5 check return code # rcode=0
|
| 148 |
|
| 149 | The test "check return code" passed, with additional diagnostic data “rcode=0”
|
| 150 |
|
| 151 |
|
| 152 | Diagnostic lines
|
| 153 | ----------------
|
| 154 |
|
| 155 | If tests wish to output any further information, they should do so using
|
| 156 | "diagnostic lines". Diagnostic lines are optional, freeform text, and are
|
| 157 | often used to describe what is being tested and any intermediate results in
|
| 158 | more detail than the final result and diagnostic data line provides.
|
| 159 |
|
| 160 | Diagnostic lines are formatted as "# <diagnostic_description>", where the
|
| 161 | description can be any string. Diagnostic lines can be anywhere in the test
|
| 162 | output. As a rule, diagnostic lines regarding a test are directly before the
|
| 163 | test result line for that test.
|
| 164 |
|
| 165 | Note that most tools will treat unknown lines (see below) as diagnostic lines,
|
| 166 | even if they do not start with a "#": this is to capture any other useful
|
| 167 | kernel output which may help debug the test. It is nevertheless recommended
|
| 168 | that tests always prefix any diagnostic output they have with a "#" character.
|
| 169 |
|
| 170 | Unknown lines
|
| 171 | -------------
|
| 172 |
|
| 173 | There may be lines within KTAP output that do not follow the format of one of
|
| 174 | the four formats for lines described above. This is allowed, however, they will
|
| 175 | not influence the status of the tests.
|
| 176 |
|
| 177 | This is an important difference from TAP. Kernel tests may print messages
|
| 178 | to the system console or a log file. Both of these destinations may contain
|
| 179 | messages either from unrelated kernel or userspace activity, or kernel
|
| 180 | messages from non-test code that is invoked by the test. The kernel code
|
| 181 | invoked by the test likely is not aware that a test is in progress and
|
| 182 | thus can not print the message as a diagnostic message.
|
| 183 |
|
| 184 | Nested tests
|
| 185 | ------------
|
| 186 |
|
| 187 | In KTAP, tests can be nested. This is done by having a test include within its
|
| 188 | output an entire set of KTAP-formatted results. This can be used to categorize
|
| 189 | and group related tests, or to split out different results from the same test.
|
| 190 |
|
| 191 | The "parent" test's result should consist of all of its subtests' results,
|
| 192 | starting with another KTAP version line and test plan, and end with the overall
|
| 193 | result. If one of the subtests fail, for example, the parent test should also
|
| 194 | fail.
|
| 195 |
|
| 196 | Additionally, all lines in a subtest should be indented. One level of
|
| 197 | indentation is two spaces: " ". The indentation should begin at the version
|
| 198 | line and should end before the parent test's result line.
|
| 199 |
|
| 200 | "Unknown lines" are not considered to be lines in a subtest and thus are
|
| 201 | allowed to be either indented or not indented.
|
| 202 |
|
| 203 | An example of a test with two nested subtests:
|
| 204 |
|
| 205 | ::
|
| 206 |
|
| 207 | KTAP version 1
|
| 208 | 1..1
|
| 209 | KTAP version 1
|
| 210 | 1..2
|
| 211 | ok 1 test_1
|
| 212 | not ok 2 test_2
|
| 213 | # example failed
|
| 214 | not ok 1 example
|
| 215 |
|
| 216 | An example format with multiple levels of nested testing:
|
| 217 |
|
| 218 | ::
|
| 219 |
|
| 220 | KTAP version 1
|
| 221 | 1..2
|
| 222 | KTAP version 1
|
| 223 | 1..2
|
| 224 | KTAP version 1
|
| 225 | 1..2
|
| 226 | not ok 1 test_1
|
| 227 | ok 2 test_2
|
| 228 | not ok 1 test_3
|
| 229 | ok 2 test_4 # SKIP
|
| 230 | not ok 1 example_test_1
|
| 231 | ok 2 example_test_2
|
| 232 |
|
| 233 |
|
| 234 | Major differences between TAP and KTAP
|
| 235 | --------------------------------------
|
| 236 |
|
| 237 | ================================================== ========= ===============
|
| 238 | Feature TAP KTAP
|
| 239 | ================================================== ========= ===============
|
| 240 | yaml and json in diagnosic message ok not recommended
|
| 241 | TODO directive ok not recognized
|
| 242 | allows an arbitrary number of tests to be nested no yes
|
| 243 | "Unknown lines" are in category of "Anything else" yes no
|
| 244 | "Unknown lines" are incorrect allowed
|
| 245 | ================================================== ========= ===============
|
| 246 |
|
| 247 | The TAP14 specification does permit nested tests, but instead of using another
|
| 248 | nested version line, uses a line of the form
|
| 249 | "Subtest: <name>" where <name> is the name of the parent test.
|
| 250 |
|
| 251 | Example KTAP output
|
| 252 | --------------------
|
| 253 | ::
|
| 254 |
|
| 255 | KTAP version 1
|
| 256 | 1..1
|
| 257 | KTAP version 1
|
| 258 | 1..3
|
| 259 | KTAP version 1
|
| 260 | 1..1
|
| 261 | # test_1: initializing test_1
|
| 262 | ok 1 test_1
|
| 263 | ok 1 example_test_1
|
| 264 | KTAP version 1
|
| 265 | 1..2
|
| 266 | ok 1 test_1 # SKIP test_1 skipped
|
| 267 | ok 2 test_2
|
| 268 | ok 2 example_test_2
|
| 269 | KTAP version 1
|
| 270 | 1..3
|
| 271 | ok 1 test_1
|
| 272 | # test_2: FAIL
|
| 273 | not ok 2 test_2
|
| 274 | ok 3 test_3 # SKIP test_3 skipped
|
| 275 | not ok 3 example_test_3
|
| 276 | not ok 1 main_test
|
| 277 |
|
| 278 | This output defines the following hierarchy:
|
| 279 |
|
| 280 | A single test called "main_test", which fails, and has three subtests:
|
| 281 |
|
| 282 | - "example_test_1", which passes, and has one subtest:
|
| 283 |
|
| 284 | - "test_1", which passes, and outputs the diagnostic message "test_1: initializing test_1"
|
| 285 |
|
| 286 | - "example_test_2", which passes, and has two subtests:
|
| 287 |
|
| 288 | - "test_1", which is skipped, with the explanation "test_1 skipped"
|
| 289 | - "test_2", which passes
|
| 290 |
|
| 291 | - "example_test_3", which fails, and has three subtests
|
| 292 |
|
| 293 | - "test_1", which passes
|
| 294 | - "test_2", which outputs the diagnostic line "test_2: FAIL", and fails.
|
| 295 | - "test_3", which is skipped with the explanation "test_3 skipped"
|
| 296 |
|
| 297 | Note that the individual subtests with the same names do not conflict, as they
|
| 298 | are found in different parent tests. This output also exhibits some sensible
|
| 299 | rules for "bubbling up" test results: a test fails if any of its subtests fail.
|
| 300 | Skipped tests do not affect the result of the parent test (though it often
|
| 301 | makes sense for a test to be marked skipped if _all_ of its subtests have been
|
| 302 | skipped).
|
| 303 |
|
| 304 | See also:
|
| 305 | ---------
|
| 306 |
|
| 307 | - The TAP specification:
|
| 308 | https://testanything.org/tap-version-13-specification.html
|
| 309 | - The (stagnant) TAP version 14 specification:
|
| 310 | https://github.com/TestAnything/Specification/blob/tap-14-specification/specification.md
|
| 311 | - The kselftest documentation:
|
| 312 | Documentation/dev-tools/kselftest.rst
|
| 313 | - The KUnit documentation:
|
| 314 | Documentation/dev-tools/kunit/index.rst
|