| 1 | Dynamic debug
|
| 2 | +++++++++++++
|
| 3 |
|
| 4 |
|
| 5 | Introduction
|
| 6 | ============
|
| 7 |
|
| 8 | Dynamic debug allows you to dynamically enable/disable kernel
|
| 9 | debug-print code to obtain additional kernel information.
|
| 10 |
|
| 11 | If ``/proc/dynamic_debug/control`` exists, your kernel has dynamic
|
| 12 | debug. You'll need root access (sudo su) to use this.
|
| 13 |
|
| 14 | Dynamic debug provides:
|
| 15 |
|
| 16 | * a Catalog of all *prdbgs* in your kernel.
|
| 17 | ``cat /proc/dynamic_debug/control`` to see them.
|
| 18 |
|
| 19 | * a Simple query/command language to alter *prdbgs* by selecting on
|
| 20 | any combination of 0 or 1 of:
|
| 21 |
|
| 22 | - source filename
|
| 23 | - function name
|
| 24 | - line number (including ranges of line numbers)
|
| 25 | - module name
|
| 26 | - format string
|
| 27 | - class name (as known/declared by each module)
|
| 28 |
|
| 29 | NOTE: To actually get the debug-print output on the console, you may
|
| 30 | need to adjust the kernel ``loglevel=``, or use ``ignore_loglevel``.
|
| 31 | Read about these kernel parameters in
|
| 32 | Documentation/admin-guide/kernel-parameters.rst.
|
| 33 |
|
| 34 | Viewing Dynamic Debug Behaviour
|
| 35 | ===============================
|
| 36 |
|
| 37 | You can view the currently configured behaviour in the *prdbg* catalog::
|
| 38 |
|
| 39 | :#> head -n7 /proc/dynamic_debug/control
|
| 40 | # filename:lineno [module]function flags format
|
| 41 | init/main.c:1179 [main]initcall_blacklist =_ "blacklisting initcall %s\n"
|
| 42 | init/main.c:1218 [main]initcall_blacklisted =_ "initcall %s blacklisted\n"
|
| 43 | init/main.c:1424 [main]run_init_process =_ " with arguments:\n"
|
| 44 | init/main.c:1426 [main]run_init_process =_ " %s\n"
|
| 45 | init/main.c:1427 [main]run_init_process =_ " with environment:\n"
|
| 46 | init/main.c:1429 [main]run_init_process =_ " %s\n"
|
| 47 |
|
| 48 | The 3rd space-delimited column shows the current flags, preceded by
|
| 49 | a ``=`` for easy use with grep/cut. ``=p`` shows enabled callsites.
|
| 50 |
|
| 51 | Controlling dynamic debug Behaviour
|
| 52 | ===================================
|
| 53 |
|
| 54 | The behaviour of *prdbg* sites are controlled by writing
|
| 55 | query/commands to the control file. Example::
|
| 56 |
|
| 57 | # grease the interface
|
| 58 | :#> alias ddcmd='echo $* > /proc/dynamic_debug/control'
|
| 59 |
|
| 60 | :#> ddcmd '-p; module main func run* +p'
|
| 61 | :#> grep =p /proc/dynamic_debug/control
|
| 62 | init/main.c:1424 [main]run_init_process =p " with arguments:\n"
|
| 63 | init/main.c:1426 [main]run_init_process =p " %s\n"
|
| 64 | init/main.c:1427 [main]run_init_process =p " with environment:\n"
|
| 65 | init/main.c:1429 [main]run_init_process =p " %s\n"
|
| 66 |
|
| 67 | Error messages go to console/syslog::
|
| 68 |
|
| 69 | :#> ddcmd mode foo +p
|
| 70 | dyndbg: unknown keyword "mode"
|
| 71 | dyndbg: query parse failed
|
| 72 | bash: echo: write error: Invalid argument
|
| 73 |
|
| 74 | If debugfs is also enabled and mounted, ``dynamic_debug/control`` is
|
| 75 | also under the mount-dir, typically ``/sys/kernel/debug/``.
|
| 76 |
|
| 77 | Command Language Reference
|
| 78 | ==========================
|
| 79 |
|
| 80 | At the basic lexical level, a command is a sequence of words separated
|
| 81 | by spaces or tabs. So these are all equivalent::
|
| 82 |
|
| 83 | :#> ddcmd file svcsock.c line 1603 +p
|
| 84 | :#> ddcmd "file svcsock.c line 1603 +p"
|
| 85 | :#> ddcmd ' file svcsock.c line 1603 +p '
|
| 86 |
|
| 87 | Command submissions are bounded by a write() system call.
|
| 88 | Multiple commands can be written together, separated by ``;`` or ``\n``::
|
| 89 |
|
| 90 | :#> ddcmd "func pnpacpi_get_resources +p; func pnp_assign_mem +p"
|
| 91 | :#> ddcmd <<"EOC"
|
| 92 | func pnpacpi_get_resources +p
|
| 93 | func pnp_assign_mem +p
|
| 94 | EOC
|
| 95 | :#> cat query-batch-file > /proc/dynamic_debug/control
|
| 96 |
|
| 97 | You can also use wildcards in each query term. The match rule supports
|
| 98 | ``*`` (matches zero or more characters) and ``?`` (matches exactly one
|
| 99 | character). For example, you can match all usb drivers::
|
| 100 |
|
| 101 | :#> ddcmd file "drivers/usb/*" +p # "" to suppress shell expansion
|
| 102 |
|
| 103 | Syntactically, a command is pairs of keyword values, followed by a
|
| 104 | flags change or setting::
|
| 105 |
|
| 106 | command ::= match-spec* flags-spec
|
| 107 |
|
| 108 | The match-spec's select *prdbgs* from the catalog, upon which to apply
|
| 109 | the flags-spec, all constraints are ANDed together. An absent keyword
|
| 110 | is the same as keyword "*".
|
| 111 |
|
| 112 | Note that since the match-spec can be empty, the flags are checked 1st,
|
| 113 | then the pairs of keyword and value. Flag errs will hide keyword errs::
|
| 114 |
|
| 115 | bash-5.2# ddcmd mod bar +foo
|
| 116 | dyndbg: read 13 bytes from userspace
|
| 117 | dyndbg: query 0: "mod bar +foo" mod:*
|
| 118 | dyndbg: unknown flag 'o'
|
| 119 | dyndbg: flags parse failed
|
| 120 | dyndbg: processed 1 queries, with 0 matches, 1 errs
|
| 121 |
|
| 122 | So a match-spec is a keyword, which selects the attribute of the
|
| 123 | callsite to be compared, and a value to compare against. Possible
|
| 124 | keywords are::
|
| 125 |
|
| 126 | match-spec ::= 'func' string |
|
| 127 | 'file' string |
|
| 128 | 'module' string |
|
| 129 | 'format' string |
|
| 130 | 'class' string |
|
| 131 | 'line' line-range
|
| 132 |
|
| 133 | line-range ::= lineno |
|
| 134 | '-'lineno |
|
| 135 | lineno'-' |
|
| 136 | lineno'-'lineno
|
| 137 |
|
| 138 | lineno ::= unsigned-int
|
| 139 |
|
| 140 | .. note::
|
| 141 |
|
| 142 | ``line-range`` cannot contain space, e.g.
|
| 143 | "1-30" is valid range but "1 - 30" is not.
|
| 144 |
|
| 145 |
|
| 146 | The meanings of each keyword are:
|
| 147 |
|
| 148 | func
|
| 149 | The given string is compared against the function name
|
| 150 | of each callsite. Example::
|
| 151 |
|
| 152 | func svc_tcp_accept
|
| 153 | func *recv* # in rfcomm, bluetooth, ping, tcp
|
| 154 |
|
| 155 | file
|
| 156 | The given string is compared against either the src-root relative
|
| 157 | pathname, or the basename of the source file of each callsite.
|
| 158 | Examples::
|
| 159 |
|
| 160 | file svcsock.c
|
| 161 | file kernel/freezer.c # ie column 1 of control file
|
| 162 | file drivers/usb/* # all callsites under it
|
| 163 | file inode.c:start_* # parse :tail as a func (above)
|
| 164 | file inode.c:1-100 # parse :tail as a line-range (above)
|
| 165 |
|
| 166 | module
|
| 167 | The given string is compared against the module name
|
| 168 | of each callsite. The module name is the string as
|
| 169 | seen in ``lsmod``, i.e. without the directory or the ``.ko``
|
| 170 | suffix and with ``-`` changed to ``_``. Examples::
|
| 171 |
|
| 172 | module sunrpc
|
| 173 | module nfsd
|
| 174 | module drm* # both drm, drm_kms_helper
|
| 175 |
|
| 176 | format
|
| 177 | The given string is searched for in the dynamic debug format
|
| 178 | string. Note that the string does not need to match the
|
| 179 | entire format, only some part. Whitespace and other
|
| 180 | special characters can be escaped using C octal character
|
| 181 | escape ``\ooo`` notation, e.g. the space character is ``\040``.
|
| 182 | Alternatively, the string can be enclosed in double quote
|
| 183 | characters (``"``) or single quote characters (``'``).
|
| 184 | Examples::
|
| 185 |
|
| 186 | format svcrdma: // many of the NFS/RDMA server pr_debugs
|
| 187 | format readahead // some pr_debugs in the readahead cache
|
| 188 | format nfsd:\040SETATTR // one way to match a format with whitespace
|
| 189 | format "nfsd: SETATTR" // a neater way to match a format with whitespace
|
| 190 | format 'nfsd: SETATTR' // yet another way to match a format with whitespace
|
| 191 |
|
| 192 | class
|
| 193 | The given class_name is validated against each module, which may
|
| 194 | have declared a list of known class_names. If the class_name is
|
| 195 | found for a module, callsite & class matching and adjustment
|
| 196 | proceeds. Examples::
|
| 197 |
|
| 198 | class DRM_UT_KMS # a DRM.debug category
|
| 199 | class JUNK # silent non-match
|
| 200 | // class TLD_* # NOTICE: no wildcard in class names
|
| 201 |
|
| 202 | line
|
| 203 | The given line number or range of line numbers is compared
|
| 204 | against the line number of each ``pr_debug()`` callsite. A single
|
| 205 | line number matches the callsite line number exactly. A
|
| 206 | range of line numbers matches any callsite between the first
|
| 207 | and last line number inclusive. An empty first number means
|
| 208 | the first line in the file, an empty last line number means the
|
| 209 | last line number in the file. Examples::
|
| 210 |
|
| 211 | line 1603 // exactly line 1603
|
| 212 | line 1600-1605 // the six lines from line 1600 to line 1605
|
| 213 | line -1605 // the 1605 lines from line 1 to line 1605
|
| 214 | line 1600- // all lines from line 1600 to the end of the file
|
| 215 |
|
| 216 | The flags specification comprises a change operation followed
|
| 217 | by one or more flag characters. The change operation is one
|
| 218 | of the characters::
|
| 219 |
|
| 220 | - remove the given flags
|
| 221 | + add the given flags
|
| 222 | = set the flags to the given flags
|
| 223 |
|
| 224 | The flags are::
|
| 225 |
|
| 226 | p enables the pr_debug() callsite.
|
| 227 | _ enables no flags.
|
| 228 |
|
| 229 | Decorator flags add to the message-prefix, in order:
|
| 230 | t Include thread ID, or <intr>
|
| 231 | m Include module name
|
| 232 | f Include the function name
|
| 233 | s Include the source file name
|
| 234 | l Include line number
|
| 235 | d Include call trace
|
| 236 |
|
| 237 | For ``print_hex_dump_debug()`` and ``print_hex_dump_bytes()``, only
|
| 238 | the ``p`` flag has meaning, other flags are ignored.
|
| 239 |
|
| 240 | Note the regexp ``^[-+=][fslmptd_]+$`` matches a flags specification.
|
| 241 | To clear all flags at once, use ``=_`` or ``-fslmptd``.
|
| 242 |
|
| 243 |
|
| 244 | Debug messages during Boot Process
|
| 245 | ==================================
|
| 246 |
|
| 247 | To activate debug messages for core code and built-in modules during
|
| 248 | the boot process, even before userspace and debugfs exists, use
|
| 249 | ``dyndbg="QUERY"`` or ``module.dyndbg="QUERY"``. QUERY follows
|
| 250 | the syntax described above, but must not exceed 1023 characters. Your
|
| 251 | bootloader may impose lower limits.
|
| 252 |
|
| 253 | These ``dyndbg`` params are processed just after the ddebug tables are
|
| 254 | processed, as part of the early_initcall. Thus you can enable debug
|
| 255 | messages in all code run after this early_initcall via this boot
|
| 256 | parameter.
|
| 257 |
|
| 258 | On an x86 system for example ACPI enablement is a subsys_initcall and::
|
| 259 |
|
| 260 | dyndbg="file ec.c +p"
|
| 261 |
|
| 262 | will show early Embedded Controller transactions during ACPI setup if
|
| 263 | your machine (typically a laptop) has an Embedded Controller.
|
| 264 | PCI (or other devices) initialization also is a hot candidate for using
|
| 265 | this boot parameter for debugging purposes.
|
| 266 |
|
| 267 | If ``foo`` module is not built-in, ``foo.dyndbg`` will still be processed at
|
| 268 | boot time, without effect, but will be reprocessed when module is
|
| 269 | loaded later. Bare ``dyndbg=`` is only processed at boot.
|
| 270 |
|
| 271 |
|
| 272 | Debug Messages at Module Initialization Time
|
| 273 | ============================================
|
| 274 |
|
| 275 | When ``modprobe foo`` is called, modprobe scans ``/proc/cmdline`` for
|
| 276 | ``foo.params``, strips ``foo.``, and passes them to the kernel along with
|
| 277 | params given in modprobe args or ``/etc/modprobe.d/*.conf`` files,
|
| 278 | in the following order:
|
| 279 |
|
| 280 | 1. parameters given via ``/etc/modprobe.d/*.conf``::
|
| 281 |
|
| 282 | options foo dyndbg=+pt
|
| 283 | options foo dyndbg # defaults to +p
|
| 284 |
|
| 285 | 2. ``foo.dyndbg`` as given in boot args, ``foo.`` is stripped and passed::
|
| 286 |
|
| 287 | foo.dyndbg=" func bar +p; func buz +mp"
|
| 288 |
|
| 289 | 3. args to modprobe::
|
| 290 |
|
| 291 | modprobe foo dyndbg==pmf # override previous settings
|
| 292 |
|
| 293 | These ``dyndbg`` queries are applied in order, with last having final say.
|
| 294 | This allows boot args to override or modify those from ``/etc/modprobe.d``
|
| 295 | (sensible, since 1 is system wide, 2 is kernel or boot specific), and
|
| 296 | modprobe args to override both.
|
| 297 |
|
| 298 | In the ``foo.dyndbg="QUERY"`` form, the query must exclude ``module foo``.
|
| 299 | ``foo`` is extracted from the param-name, and applied to each query in
|
| 300 | ``QUERY``, and only 1 match-spec of each type is allowed.
|
| 301 |
|
| 302 | The ``dyndbg`` option is a "fake" module parameter, which means:
|
| 303 |
|
| 304 | - modules do not need to define it explicitly
|
| 305 | - every module gets it tacitly, whether they use pr_debug or not
|
| 306 | - it doesn't appear in ``/sys/module/$module/parameters/``
|
| 307 | To see it, grep the control file, or inspect ``/proc/cmdline.``
|
| 308 |
|
| 309 | For ``CONFIG_DYNAMIC_DEBUG`` kernels, any settings given at boot-time (or
|
| 310 | enabled by ``-DDEBUG`` flag during compilation) can be disabled later via
|
| 311 | the debugfs interface if the debug messages are no longer needed::
|
| 312 |
|
| 313 | echo "module module_name -p" > /proc/dynamic_debug/control
|
| 314 |
|
| 315 | Examples
|
| 316 | ========
|
| 317 |
|
| 318 | ::
|
| 319 |
|
| 320 | // enable the message at line 1603 of file svcsock.c
|
| 321 | :#> ddcmd 'file svcsock.c line 1603 +p'
|
| 322 |
|
| 323 | // enable all the messages in file svcsock.c
|
| 324 | :#> ddcmd 'file svcsock.c +p'
|
| 325 |
|
| 326 | // enable all the messages in the NFS server module
|
| 327 | :#> ddcmd 'module nfsd +p'
|
| 328 |
|
| 329 | // enable all 12 messages in the function svc_process()
|
| 330 | :#> ddcmd 'func svc_process +p'
|
| 331 |
|
| 332 | // disable all 12 messages in the function svc_process()
|
| 333 | :#> ddcmd 'func svc_process -p'
|
| 334 |
|
| 335 | // enable messages for NFS calls READ, READLINK, READDIR and READDIR+.
|
| 336 | :#> ddcmd 'format "nfsd: READ" +p'
|
| 337 |
|
| 338 | // enable messages in files of which the paths include string "usb"
|
| 339 | :#> ddcmd 'file *usb* +p'
|
| 340 |
|
| 341 | // enable all messages
|
| 342 | :#> ddcmd '+p'
|
| 343 |
|
| 344 | // add module, function to all enabled messages
|
| 345 | :#> ddcmd '+mf'
|
| 346 |
|
| 347 | // boot-args example, with newlines and comments for readability
|
| 348 | Kernel command line: ...
|
| 349 | // see what's going on in dyndbg=value processing
|
| 350 | dynamic_debug.verbose=3
|
| 351 | // enable pr_debugs in the btrfs module (can be builtin or loadable)
|
| 352 | btrfs.dyndbg="+p"
|
| 353 | // enable pr_debugs in all files under init/
|
| 354 | // and the function parse_one, #cmt is stripped
|
| 355 | dyndbg="file init/* +p #cmt ; func parse_one +p"
|
| 356 | // enable pr_debugs in 2 functions in a module loaded later
|
| 357 | pc87360.dyndbg="func pc87360_init_device +p; func pc87360_find +p"
|
| 358 |
|
| 359 | Kernel Configuration
|
| 360 | ====================
|
| 361 |
|
| 362 | Dynamic Debug is enabled via kernel config items::
|
| 363 |
|
| 364 | CONFIG_DYNAMIC_DEBUG=y # build catalog, enables CORE
|
| 365 | CONFIG_DYNAMIC_DEBUG_CORE=y # enable mechanics only, skip catalog
|
| 366 |
|
| 367 | If you do not want to enable dynamic debug globally (i.e. in some embedded
|
| 368 | system), you may set ``CONFIG_DYNAMIC_DEBUG_CORE`` as basic support of dynamic
|
| 369 | debug and add ``ccflags := -DDYNAMIC_DEBUG_MODULE`` into the Makefile of any
|
| 370 | modules which you'd like to dynamically debug later.
|
| 371 |
|
| 372 |
|
| 373 | Kernel *prdbg* API
|
| 374 | ==================
|
| 375 |
|
| 376 | The following functions are cataloged and controllable when dynamic
|
| 377 | debug is enabled::
|
| 378 |
|
| 379 | pr_debug()
|
| 380 | dev_dbg()
|
| 381 | print_hex_dump_debug()
|
| 382 | print_hex_dump_bytes()
|
| 383 |
|
| 384 | Otherwise, they are off by default; ``ccflags += -DDEBUG`` or
|
| 385 | ``#define DEBUG`` in a source file will enable them appropriately.
|
| 386 |
|
| 387 | If ``CONFIG_DYNAMIC_DEBUG`` is not set, ``print_hex_dump_debug()`` is
|
| 388 | just a shortcut for ``print_hex_dump(KERN_DEBUG)``.
|
| 389 |
|
| 390 | For ``print_hex_dump_debug()``/``print_hex_dump_bytes()``, format string is
|
| 391 | its ``prefix_str`` argument, if it is constant string; or ``hexdump``
|
| 392 | in case ``prefix_str`` is built dynamically.
|