| 1 | =================================================
|
| 2 | Using kgdb, kdb and the kernel debugger internals
|
| 3 | =================================================
|
| 4 |
|
| 5 | :Author: Jason Wessel
|
| 6 |
|
| 7 | Introduction
|
| 8 | ============
|
| 9 |
|
| 10 | The kernel has two different debugger front ends (kdb and kgdb) which
|
| 11 | interface to the debug core. It is possible to use either of the
|
| 12 | debugger front ends and dynamically transition between them if you
|
| 13 | configure the kernel properly at compile and runtime.
|
| 14 |
|
| 15 | Kdb is simplistic shell-style interface which you can use on a system
|
| 16 | console with a keyboard or serial console. You can use it to inspect
|
| 17 | memory, registers, process lists, dmesg, and even set breakpoints to
|
| 18 | stop in a certain location. Kdb is not a source level debugger, although
|
| 19 | you can set breakpoints and execute some basic kernel run control. Kdb
|
| 20 | is mainly aimed at doing some analysis to aid in development or
|
| 21 | diagnosing kernel problems. You can access some symbols by name in
|
| 22 | kernel built-ins or in kernel modules if the code was built with
|
| 23 | ``CONFIG_KALLSYMS``.
|
| 24 |
|
| 25 | Kgdb is intended to be used as a source level debugger for the Linux
|
| 26 | kernel. It is used along with gdb to debug a Linux kernel. The
|
| 27 | expectation is that gdb can be used to "break in" to the kernel to
|
| 28 | inspect memory, variables and look through call stack information
|
| 29 | similar to the way an application developer would use gdb to debug an
|
| 30 | application. It is possible to place breakpoints in kernel code and
|
| 31 | perform some limited execution stepping.
|
| 32 |
|
| 33 | Two machines are required for using kgdb. One of these machines is a
|
| 34 | development machine and the other is the target machine. The kernel to
|
| 35 | be debugged runs on the target machine. The development machine runs an
|
| 36 | instance of gdb against the vmlinux file which contains the symbols (not
|
| 37 | a boot image such as bzImage, zImage, uImage...). In gdb the developer
|
| 38 | specifies the connection parameters and connects to kgdb. The type of
|
| 39 | connection a developer makes with gdb depends on the availability of
|
| 40 | kgdb I/O modules compiled as built-ins or loadable kernel modules in the
|
| 41 | test machine's kernel.
|
| 42 |
|
| 43 | Compiling a kernel
|
| 44 | ==================
|
| 45 |
|
| 46 | - In order to enable compilation of kdb, you must first enable kgdb.
|
| 47 |
|
| 48 | - The kgdb test compile options are described in the kgdb test suite
|
| 49 | chapter.
|
| 50 |
|
| 51 | Kernel config options for kgdb
|
| 52 | ------------------------------
|
| 53 |
|
| 54 | To enable ``CONFIG_KGDB`` you should look under
|
| 55 | :menuselection:`Kernel hacking --> Kernel debugging` and select
|
| 56 | :menuselection:`KGDB: kernel debugger`.
|
| 57 |
|
| 58 | While it is not a hard requirement that you have symbols in your vmlinux
|
| 59 | file, gdb tends not to be very useful without the symbolic data, so you
|
| 60 | will want to turn on ``CONFIG_DEBUG_INFO`` which is called
|
| 61 | :menuselection:`Compile the kernel with debug info` in the config menu.
|
| 62 |
|
| 63 | It is advised, but not required, that you turn on the
|
| 64 | ``CONFIG_FRAME_POINTER`` kernel option which is called :menuselection:`Compile
|
| 65 | the kernel with frame pointers` in the config menu. This option inserts code
|
| 66 | into the compiled executable which saves the frame information in registers
|
| 67 | or on the stack at different points which allows a debugger such as gdb to
|
| 68 | more accurately construct stack back traces while debugging the kernel.
|
| 69 |
|
| 70 | If the architecture that you are using supports the kernel option
|
| 71 | ``CONFIG_STRICT_KERNEL_RWX``, you should consider turning it off. This
|
| 72 | option will prevent the use of software breakpoints because it marks
|
| 73 | certain regions of the kernel's memory space as read-only. If kgdb
|
| 74 | supports it for the architecture you are using, you can use hardware
|
| 75 | breakpoints if you desire to run with the ``CONFIG_STRICT_KERNEL_RWX``
|
| 76 | option turned on, else you need to turn off this option.
|
| 77 |
|
| 78 | Next you should choose one or more I/O drivers to interconnect the debugging
|
| 79 | host and debugged target. Early boot debugging requires a KGDB I/O
|
| 80 | driver that supports early debugging and the driver must be built into
|
| 81 | the kernel directly. Kgdb I/O driver configuration takes place via
|
| 82 | kernel or module parameters which you can learn more about in the
|
| 83 | section that describes the parameter kgdboc.
|
| 84 |
|
| 85 | Here is an example set of ``.config`` symbols to enable or disable for kgdb::
|
| 86 |
|
| 87 | # CONFIG_STRICT_KERNEL_RWX is not set
|
| 88 | CONFIG_FRAME_POINTER=y
|
| 89 | CONFIG_KGDB=y
|
| 90 | CONFIG_KGDB_SERIAL_CONSOLE=y
|
| 91 |
|
| 92 | Kernel config options for kdb
|
| 93 | -----------------------------
|
| 94 |
|
| 95 | Kdb is quite a bit more complex than the simple gdbstub sitting on top
|
| 96 | of the kernel's debug core. Kdb must implement a shell, and also adds
|
| 97 | some helper functions in other parts of the kernel, responsible for
|
| 98 | printing out interesting data such as what you would see if you ran
|
| 99 | ``lsmod``, or ``ps``. In order to build kdb into the kernel you follow the
|
| 100 | same steps as you would for kgdb.
|
| 101 |
|
| 102 | The main config option for kdb is ``CONFIG_KGDB_KDB`` which is called
|
| 103 | :menuselection:`KGDB_KDB: include kdb frontend for kgdb` in the config menu.
|
| 104 | In theory you would have already also selected an I/O driver such as the
|
| 105 | ``CONFIG_KGDB_SERIAL_CONSOLE`` interface if you plan on using kdb on a
|
| 106 | serial port, when you were configuring kgdb.
|
| 107 |
|
| 108 | If you want to use a PS/2-style keyboard with kdb, you would select
|
| 109 | ``CONFIG_KDB_KEYBOARD`` which is called :menuselection:`KGDB_KDB: keyboard as
|
| 110 | input device` in the config menu. The ``CONFIG_KDB_KEYBOARD`` option is not
|
| 111 | used for anything in the gdb interface to kgdb. The ``CONFIG_KDB_KEYBOARD``
|
| 112 | option only works with kdb.
|
| 113 |
|
| 114 | Here is an example set of ``.config`` symbols to enable/disable kdb::
|
| 115 |
|
| 116 | # CONFIG_STRICT_KERNEL_RWX is not set
|
| 117 | CONFIG_FRAME_POINTER=y
|
| 118 | CONFIG_KGDB=y
|
| 119 | CONFIG_KGDB_SERIAL_CONSOLE=y
|
| 120 | CONFIG_KGDB_KDB=y
|
| 121 | CONFIG_KDB_KEYBOARD=y
|
| 122 |
|
| 123 | Kernel Debugger Boot Arguments
|
| 124 | ==============================
|
| 125 |
|
| 126 | This section describes the various runtime kernel parameters that affect
|
| 127 | the configuration of the kernel debugger. The following chapter covers
|
| 128 | using kdb and kgdb as well as providing some examples of the
|
| 129 | configuration parameters.
|
| 130 |
|
| 131 | Kernel parameter: kgdboc
|
| 132 | ------------------------
|
| 133 |
|
| 134 | The kgdboc driver was originally an abbreviation meant to stand for
|
| 135 | "kgdb over console". Today it is the primary mechanism to configure how
|
| 136 | to communicate from gdb to kgdb as well as the devices you want to use
|
| 137 | to interact with the kdb shell.
|
| 138 |
|
| 139 | For kgdb/gdb, kgdboc is designed to work with a single serial port. It
|
| 140 | is intended to cover the circumstance where you want to use a serial
|
| 141 | console as your primary console as well as using it to perform kernel
|
| 142 | debugging. It is also possible to use kgdb on a serial port which is not
|
| 143 | designated as a system console. Kgdboc may be configured as a kernel
|
| 144 | built-in or a kernel loadable module. You can only make use of
|
| 145 | ``kgdbwait`` and early debugging if you build kgdboc into the kernel as
|
| 146 | a built-in.
|
| 147 |
|
| 148 | Optionally you can elect to activate kms (Kernel Mode Setting)
|
| 149 | integration. When you use kms with kgdboc and you have a video driver
|
| 150 | that has atomic mode setting hooks, it is possible to enter the debugger
|
| 151 | on the graphics console. When the kernel execution is resumed, the
|
| 152 | previous graphics mode will be restored. This integration can serve as a
|
| 153 | useful tool to aid in diagnosing crashes or doing analysis of memory
|
| 154 | with kdb while allowing the full graphics console applications to run.
|
| 155 |
|
| 156 | kgdboc arguments
|
| 157 | ~~~~~~~~~~~~~~~~
|
| 158 |
|
| 159 | Usage::
|
| 160 |
|
| 161 | kgdboc=[kms][[,]kbd][[,]serial_device][,baud]
|
| 162 |
|
| 163 | The order listed above must be observed if you use any of the optional
|
| 164 | configurations together.
|
| 165 |
|
| 166 | Abbreviations:
|
| 167 |
|
| 168 | - kms = Kernel Mode Setting
|
| 169 |
|
| 170 | - kbd = Keyboard
|
| 171 |
|
| 172 | You can configure kgdboc to use the keyboard, and/or a serial device
|
| 173 | depending on if you are using kdb and/or kgdb, in one of the following
|
| 174 | scenarios. The order listed above must be observed if you use any of the
|
| 175 | optional configurations together. Using kms + only gdb is generally not
|
| 176 | a useful combination.
|
| 177 |
|
| 178 | Using loadable module or built-in
|
| 179 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 180 |
|
| 181 | 1. As a kernel built-in:
|
| 182 |
|
| 183 | Use the kernel boot argument::
|
| 184 |
|
| 185 | kgdboc=<tty-device>,[baud]
|
| 186 |
|
| 187 | 2. As a kernel loadable module:
|
| 188 |
|
| 189 | Use the command::
|
| 190 |
|
| 191 | modprobe kgdboc kgdboc=<tty-device>,[baud]
|
| 192 |
|
| 193 | Here are two examples of how you might format the kgdboc string. The
|
| 194 | first is for an x86 target using the first serial port. The second
|
| 195 | example is for the ARM Versatile AB using the second serial port.
|
| 196 |
|
| 197 | 1. ``kgdboc=ttyS0,115200``
|
| 198 |
|
| 199 | 2. ``kgdboc=ttyAMA1,115200``
|
| 200 |
|
| 201 | Configure kgdboc at runtime with sysfs
|
| 202 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
| 203 |
|
| 204 | At run time you can enable or disable kgdboc by writing parameters
|
| 205 | into sysfs. Here are two examples:
|
| 206 |
|
| 207 | 1. Enable kgdboc on ttyS0::
|
| 208 |
|
| 209 | echo ttyS0 > /sys/module/kgdboc/parameters/kgdboc
|
| 210 |
|
| 211 | 2. Disable kgdboc::
|
| 212 |
|
| 213 | echo "" > /sys/module/kgdboc/parameters/kgdboc
|
| 214 |
|
| 215 | .. note::
|
| 216 |
|
| 217 | You do not need to specify the baud if you are configuring the
|
| 218 | console on tty which is already configured or open.
|
| 219 |
|
| 220 | More examples
|
| 221 | ^^^^^^^^^^^^^
|
| 222 |
|
| 223 | You can configure kgdboc to use the keyboard, and/or a serial device
|
| 224 | depending on if you are using kdb and/or kgdb, in one of the following
|
| 225 | scenarios.
|
| 226 |
|
| 227 | 1. kdb and kgdb over only a serial port::
|
| 228 |
|
| 229 | kgdboc=<serial_device>[,baud]
|
| 230 |
|
| 231 | Example::
|
| 232 |
|
| 233 | kgdboc=ttyS0,115200
|
| 234 |
|
| 235 | 2. kdb and kgdb with keyboard and a serial port::
|
| 236 |
|
| 237 | kgdboc=kbd,<serial_device>[,baud]
|
| 238 |
|
| 239 | Example::
|
| 240 |
|
| 241 | kgdboc=kbd,ttyS0,115200
|
| 242 |
|
| 243 | 3. kdb with a keyboard::
|
| 244 |
|
| 245 | kgdboc=kbd
|
| 246 |
|
| 247 | 4. kdb with kernel mode setting::
|
| 248 |
|
| 249 | kgdboc=kms,kbd
|
| 250 |
|
| 251 | 5. kdb with kernel mode setting and kgdb over a serial port::
|
| 252 |
|
| 253 | kgdboc=kms,kbd,ttyS0,115200
|
| 254 |
|
| 255 | .. note::
|
| 256 |
|
| 257 | Kgdboc does not support interrupting the target via the gdb remote
|
| 258 | protocol. You must manually send a `SysRq-G` unless you have a proxy
|
| 259 | that splits console output to a terminal program. A console proxy has a
|
| 260 | separate TCP port for the debugger and a separate TCP port for the
|
| 261 | "human" console. The proxy can take care of sending the `SysRq-G`
|
| 262 | for you.
|
| 263 |
|
| 264 | When using kgdboc with no debugger proxy, you can end up connecting the
|
| 265 | debugger at one of two entry points. If an exception occurs after you
|
| 266 | have loaded kgdboc, a message should print on the console stating it is
|
| 267 | waiting for the debugger. In this case you disconnect your terminal
|
| 268 | program and then connect the debugger in its place. If you want to
|
| 269 | interrupt the target system and forcibly enter a debug session you have
|
| 270 | to issue a `Sysrq` sequence and then type the letter `g`. Then you
|
| 271 | disconnect the terminal session and connect gdb. Your options if you
|
| 272 | don't like this are to hack gdb to send the `SysRq-G` for you as well as
|
| 273 | on the initial connect, or to use a debugger proxy that allows an
|
| 274 | unmodified gdb to do the debugging.
|
| 275 |
|
| 276 | Kernel parameter: ``kgdboc_earlycon``
|
| 277 | -------------------------------------
|
| 278 |
|
| 279 | If you specify the kernel parameter ``kgdboc_earlycon`` and your serial
|
| 280 | driver registers a boot console that supports polling (doesn't need
|
| 281 | interrupts and implements a nonblocking read() function) kgdb will attempt
|
| 282 | to work using the boot console until it can transition to the regular
|
| 283 | tty driver specified by the ``kgdboc`` parameter.
|
| 284 |
|
| 285 | Normally there is only one boot console (especially that implements the
|
| 286 | read() function) so just adding ``kgdboc_earlycon`` on its own is
|
| 287 | sufficient to make this work. If you have more than one boot console you
|
| 288 | can add the boot console's name to differentiate. Note that names that
|
| 289 | are registered through the boot console layer and the tty layer are not
|
| 290 | the same for the same port.
|
| 291 |
|
| 292 | For instance, on one board to be explicit you might do::
|
| 293 |
|
| 294 | kgdboc_earlycon=qcom_geni kgdboc=ttyMSM0
|
| 295 |
|
| 296 | If the only boot console on the device was "qcom_geni", you could simplify::
|
| 297 |
|
| 298 | kgdboc_earlycon kgdboc=ttyMSM0
|
| 299 |
|
| 300 | Kernel parameter: ``kgdbwait``
|
| 301 | ------------------------------
|
| 302 |
|
| 303 | The Kernel command line option ``kgdbwait`` makes kgdb wait for a
|
| 304 | debugger connection during booting of a kernel. You can only use this
|
| 305 | option if you compiled a kgdb I/O driver into the kernel and you
|
| 306 | specified the I/O driver configuration as a kernel command line option.
|
| 307 | The kgdbwait parameter should always follow the configuration parameter
|
| 308 | for the kgdb I/O driver in the kernel command line else the I/O driver
|
| 309 | will not be configured prior to asking the kernel to use it to wait.
|
| 310 |
|
| 311 | The kernel will stop and wait as early as the I/O driver and
|
| 312 | architecture allows when you use this option. If you build the kgdb I/O
|
| 313 | driver as a loadable kernel module kgdbwait will not do anything.
|
| 314 |
|
| 315 | Kernel parameter: ``kgdbcon``
|
| 316 | -----------------------------
|
| 317 |
|
| 318 | The ``kgdbcon`` feature allows you to see printk() messages inside gdb
|
| 319 | while gdb is connected to the kernel. Kdb does not make use of the kgdbcon
|
| 320 | feature.
|
| 321 |
|
| 322 | Kgdb supports using the gdb serial protocol to send console messages to
|
| 323 | the debugger when the debugger is connected and running. There are two
|
| 324 | ways to activate this feature.
|
| 325 |
|
| 326 | 1. Activate with the kernel command line option::
|
| 327 |
|
| 328 | kgdbcon
|
| 329 |
|
| 330 | 2. Use sysfs before configuring an I/O driver::
|
| 331 |
|
| 332 | echo 1 > /sys/module/debug_core/parameters/kgdb_use_con
|
| 333 |
|
| 334 | .. note::
|
| 335 |
|
| 336 | If you do this after you configure the kgdb I/O driver, the
|
| 337 | setting will not take effect until the next point the I/O is
|
| 338 | reconfigured.
|
| 339 |
|
| 340 | .. important::
|
| 341 |
|
| 342 | You cannot use kgdboc + kgdbcon on a tty that is an
|
| 343 | active system console. An example of incorrect usage is::
|
| 344 |
|
| 345 | console=ttyS0,115200 kgdboc=ttyS0 kgdbcon
|
| 346 |
|
| 347 | It is possible to use this option with kgdboc on a tty that is not a
|
| 348 | system console.
|
| 349 |
|
| 350 | Run time parameter: ``kgdbreboot``
|
| 351 | ----------------------------------
|
| 352 |
|
| 353 | The kgdbreboot feature allows you to change how the debugger deals with
|
| 354 | the reboot notification. You have 3 choices for the behavior. The
|
| 355 | default behavior is always set to 0.
|
| 356 |
|
| 357 | .. tabularcolumns:: |p{0.4cm}|p{11.5cm}|p{5.6cm}|
|
| 358 |
|
| 359 | .. flat-table::
|
| 360 | :widths: 1 10 8
|
| 361 |
|
| 362 | * - 1
|
| 363 | - ``echo -1 > /sys/module/debug_core/parameters/kgdbreboot``
|
| 364 | - Ignore the reboot notification entirely.
|
| 365 |
|
| 366 | * - 2
|
| 367 | - ``echo 0 > /sys/module/debug_core/parameters/kgdbreboot``
|
| 368 | - Send the detach message to any attached debugger client.
|
| 369 |
|
| 370 | * - 3
|
| 371 | - ``echo 1 > /sys/module/debug_core/parameters/kgdbreboot``
|
| 372 | - Enter the debugger on reboot notify.
|
| 373 |
|
| 374 | Kernel parameter: ``nokaslr``
|
| 375 | -----------------------------
|
| 376 |
|
| 377 | If the architecture that you are using enables KASLR by default,
|
| 378 | you should consider turning it off. KASLR randomizes the
|
| 379 | virtual address where the kernel image is mapped and confuses
|
| 380 | gdb which resolves addresses of kernel symbols from the symbol table
|
| 381 | of vmlinux.
|
| 382 |
|
| 383 | Kernel parameter: ``rodata``
|
| 384 | ----------------------------
|
| 385 |
|
| 386 | ``CONFIG_STRICT_KERNEL_RWX`` is turned on by default and is not
|
| 387 | visible to menuconfig on some architectures (arm64 for example),
|
| 388 | you can pass ``rodata=off`` to the kernel in this case.
|
| 389 |
|
| 390 | Using kdb
|
| 391 | =========
|
| 392 |
|
| 393 | Quick start for kdb on a serial port
|
| 394 | ------------------------------------
|
| 395 |
|
| 396 | This is a quick example of how to use kdb.
|
| 397 |
|
| 398 | 1. Configure kgdboc at boot using kernel parameters::
|
| 399 |
|
| 400 | console=ttyS0,115200 kgdboc=ttyS0,115200 nokaslr
|
| 401 |
|
| 402 | OR
|
| 403 |
|
| 404 | Configure kgdboc after the kernel has booted; assuming you are using
|
| 405 | a serial port console::
|
| 406 |
|
| 407 | echo ttyS0 > /sys/module/kgdboc/parameters/kgdboc
|
| 408 |
|
| 409 | 2. Enter the kernel debugger manually or by waiting for an oops or
|
| 410 | fault. There are several ways you can enter the kernel debugger
|
| 411 | manually; all involve using the `SysRq-G`, which means you must have
|
| 412 | enabled ``CONFIG_MAGIC_SYSRQ=y`` in your kernel config.
|
| 413 |
|
| 414 | - When logged in as root or with a super user session you can run::
|
| 415 |
|
| 416 | echo g > /proc/sysrq-trigger
|
| 417 |
|
| 418 | - Example using minicom 2.2
|
| 419 |
|
| 420 | Press: `CTRL-A` `f` `g`
|
| 421 |
|
| 422 | - When you have telneted to a terminal server that supports sending
|
| 423 | a remote break
|
| 424 |
|
| 425 | Press: `CTRL-]`
|
| 426 |
|
| 427 | Type in: ``send break``
|
| 428 |
|
| 429 | Press: `Enter` `g`
|
| 430 |
|
| 431 | 3. From the kdb prompt you can run the ``help`` command to see a complete
|
| 432 | list of the commands that are available.
|
| 433 |
|
| 434 | Some useful commands in kdb include:
|
| 435 |
|
| 436 | =========== =================================================================
|
| 437 | ``lsmod`` Shows where kernel modules are loaded
|
| 438 | ``ps`` Displays only the active processes
|
| 439 | ``ps A`` Shows all the processes
|
| 440 | ``summary`` Shows kernel version info and memory usage
|
| 441 | ``bt`` Get a backtrace of the current process using dump_stack()
|
| 442 | ``dmesg`` View the kernel syslog buffer
|
| 443 | ``go`` Continue the system
|
| 444 | =========== =================================================================
|
| 445 |
|
| 446 | 4. When you are done using kdb you need to consider rebooting the system
|
| 447 | or using the ``go`` command to resuming normal kernel execution. If you
|
| 448 | have paused the kernel for a lengthy period of time, applications
|
| 449 | that rely on timely networking or anything to do with real wall clock
|
| 450 | time could be adversely affected, so you should take this into
|
| 451 | consideration when using the kernel debugger.
|
| 452 |
|
| 453 | Quick start for kdb using a keyboard connected console
|
| 454 | ------------------------------------------------------
|
| 455 |
|
| 456 | This is a quick example of how to use kdb with a keyboard.
|
| 457 |
|
| 458 | 1. Configure kgdboc at boot using kernel parameters::
|
| 459 |
|
| 460 | kgdboc=kbd
|
| 461 |
|
| 462 | OR
|
| 463 |
|
| 464 | Configure kgdboc after the kernel has booted::
|
| 465 |
|
| 466 | echo kbd > /sys/module/kgdboc/parameters/kgdboc
|
| 467 |
|
| 468 | 2. Enter the kernel debugger manually or by waiting for an oops or
|
| 469 | fault. There are several ways you can enter the kernel debugger
|
| 470 | manually; all involve using the `SysRq-G`, which means you must have
|
| 471 | enabled ``CONFIG_MAGIC_SYSRQ=y`` in your kernel config.
|
| 472 |
|
| 473 | - When logged in as root or with a super user session you can run::
|
| 474 |
|
| 475 | echo g > /proc/sysrq-trigger
|
| 476 |
|
| 477 | - Example using a laptop keyboard:
|
| 478 |
|
| 479 | Press and hold down: `Alt`
|
| 480 |
|
| 481 | Press and hold down: `Fn`
|
| 482 |
|
| 483 | Press and release the key with the label: `SysRq`
|
| 484 |
|
| 485 | Release: `Fn`
|
| 486 |
|
| 487 | Press and release: `g`
|
| 488 |
|
| 489 | Release: `Alt`
|
| 490 |
|
| 491 | - Example using a PS/2 101-key keyboard
|
| 492 |
|
| 493 | Press and hold down: `Alt`
|
| 494 |
|
| 495 | Press and release the key with the label: `SysRq`
|
| 496 |
|
| 497 | Press and release: `g`
|
| 498 |
|
| 499 | Release: `Alt`
|
| 500 |
|
| 501 | 3. Now type in a kdb command such as ``help``, ``dmesg``, ``bt`` or ``go`` to
|
| 502 | continue kernel execution.
|
| 503 |
|
| 504 | Using kgdb / gdb
|
| 505 | ================
|
| 506 |
|
| 507 | In order to use kgdb you must activate it by passing configuration
|
| 508 | information to one of the kgdb I/O drivers. If you do not pass any
|
| 509 | configuration information kgdb will not do anything at all. Kgdb will
|
| 510 | only actively hook up to the kernel trap hooks if a kgdb I/O driver is
|
| 511 | loaded and configured. If you unconfigure a kgdb I/O driver, kgdb will
|
| 512 | unregister all the kernel hook points.
|
| 513 |
|
| 514 | All kgdb I/O drivers can be reconfigured at run time, if
|
| 515 | ``CONFIG_SYSFS`` and ``CONFIG_MODULES`` are enabled, by echo'ing a new
|
| 516 | config string to ``/sys/module/<driver>/parameters/<option>``. The driver
|
| 517 | can be unconfigured by passing an empty string. You cannot change the
|
| 518 | configuration while the debugger is attached. Make sure to detach the
|
| 519 | debugger with the ``detach`` command prior to trying to unconfigure a
|
| 520 | kgdb I/O driver.
|
| 521 |
|
| 522 | Connecting with gdb to a serial port
|
| 523 | ------------------------------------
|
| 524 |
|
| 525 | 1. Configure kgdboc
|
| 526 |
|
| 527 | Configure kgdboc at boot using kernel parameters::
|
| 528 |
|
| 529 | kgdboc=ttyS0,115200
|
| 530 |
|
| 531 | OR
|
| 532 |
|
| 533 | Configure kgdboc after the kernel has booted::
|
| 534 |
|
| 535 | echo ttyS0 > /sys/module/kgdboc/parameters/kgdboc
|
| 536 |
|
| 537 | 2. Stop kernel execution (break into the debugger)
|
| 538 |
|
| 539 | In order to connect to gdb via kgdboc, the kernel must first be
|
| 540 | stopped. There are several ways to stop the kernel which include
|
| 541 | using kgdbwait as a boot argument, via a `SysRq-G`, or running the
|
| 542 | kernel until it takes an exception where it waits for the debugger to
|
| 543 | attach.
|
| 544 |
|
| 545 | - When logged in as root or with a super user session you can run::
|
| 546 |
|
| 547 | echo g > /proc/sysrq-trigger
|
| 548 |
|
| 549 | - Example using minicom 2.2
|
| 550 |
|
| 551 | Press: `CTRL-A` `f` `g`
|
| 552 |
|
| 553 | - When you have telneted to a terminal server that supports sending
|
| 554 | a remote break
|
| 555 |
|
| 556 | Press: `CTRL-]`
|
| 557 |
|
| 558 | Type in: ``send break``
|
| 559 |
|
| 560 | Press: `Enter` `g`
|
| 561 |
|
| 562 | 3. Connect from gdb
|
| 563 |
|
| 564 | Example (using a directly connected port)::
|
| 565 |
|
| 566 | % gdb ./vmlinux
|
| 567 | (gdb) set serial baud 115200
|
| 568 | (gdb) target remote /dev/ttyS0
|
| 569 |
|
| 570 |
|
| 571 | Example (kgdb to a terminal server on TCP port 2012)::
|
| 572 |
|
| 573 | % gdb ./vmlinux
|
| 574 | (gdb) target remote 192.168.2.2:2012
|
| 575 |
|
| 576 |
|
| 577 | Once connected, you can debug a kernel the way you would debug an
|
| 578 | application program.
|
| 579 |
|
| 580 | If you are having problems connecting or something is going seriously
|
| 581 | wrong while debugging, it will most often be the case that you want
|
| 582 | to enable gdb to be verbose about its target communications. You do
|
| 583 | this prior to issuing the ``target remote`` command by typing in::
|
| 584 |
|
| 585 | set debug remote 1
|
| 586 |
|
| 587 | Remember if you continue in gdb, and need to "break in" again, you need
|
| 588 | to issue an other `SysRq-G`. It is easy to create a simple entry point by
|
| 589 | putting a breakpoint at ``sys_sync`` and then you can run ``sync`` from a
|
| 590 | shell or script to break into the debugger.
|
| 591 |
|
| 592 | kgdb and kdb interoperability
|
| 593 | =============================
|
| 594 |
|
| 595 | It is possible to transition between kdb and kgdb dynamically. The debug
|
| 596 | core will remember which you used the last time and automatically start
|
| 597 | in the same mode.
|
| 598 |
|
| 599 | Switching between kdb and kgdb
|
| 600 | ------------------------------
|
| 601 |
|
| 602 | Switching from kgdb to kdb
|
| 603 | ~~~~~~~~~~~~~~~~~~~~~~~~~~
|
| 604 |
|
| 605 | There are two ways to switch from kgdb to kdb: you can use gdb to issue
|
| 606 | a maintenance packet, or you can blindly type the command ``$3#33``.
|
| 607 | Whenever the kernel debugger stops in kgdb mode it will print the
|
| 608 | message ``KGDB or $3#33 for KDB``. It is important to note that you have
|
| 609 | to type the sequence correctly in one pass. You cannot type a backspace
|
| 610 | or delete because kgdb will interpret that as part of the debug stream.
|
| 611 |
|
| 612 | 1. Change from kgdb to kdb by blindly typing::
|
| 613 |
|
| 614 | $3#33
|
| 615 |
|
| 616 | 2. Change from kgdb to kdb with gdb::
|
| 617 |
|
| 618 | maintenance packet 3
|
| 619 |
|
| 620 | .. note::
|
| 621 |
|
| 622 | Now you must kill gdb. Typically you press `CTRL-Z` and issue
|
| 623 | the command::
|
| 624 |
|
| 625 | kill -9 %
|
| 626 |
|
| 627 | Change from kdb to kgdb
|
| 628 | ~~~~~~~~~~~~~~~~~~~~~~~
|
| 629 |
|
| 630 | There are two ways you can change from kdb to kgdb. You can manually
|
| 631 | enter kgdb mode by issuing the kgdb command from the kdb shell prompt,
|
| 632 | or you can connect gdb while the kdb shell prompt is active. The kdb
|
| 633 | shell looks for the typical first commands that gdb would issue with the
|
| 634 | gdb remote protocol and if it sees one of those commands it
|
| 635 | automatically changes into kgdb mode.
|
| 636 |
|
| 637 | 1. From kdb issue the command::
|
| 638 |
|
| 639 | kgdb
|
| 640 |
|
| 641 | 2. At the kdb prompt, disconnect the terminal program and connect gdb in
|
| 642 | its place.
|
| 643 |
|
| 644 | Running kdb commands from gdb
|
| 645 | -----------------------------
|
| 646 |
|
| 647 | It is possible to run a limited set of kdb commands from gdb, using the
|
| 648 | gdb monitor command. You don't want to execute any of the run control or
|
| 649 | breakpoint operations, because it can disrupt the state of the kernel
|
| 650 | debugger. You should be using gdb for breakpoints and run control
|
| 651 | operations if you have gdb connected. The more useful commands to run
|
| 652 | are things like lsmod, dmesg, ps or possibly some of the memory
|
| 653 | information commands. To see all the kdb commands you can run
|
| 654 | ``monitor help``.
|
| 655 |
|
| 656 | Example::
|
| 657 |
|
| 658 | (gdb) monitor ps
|
| 659 | 1 idle process (state I) and
|
| 660 | 27 sleeping system daemon (state M) processes suppressed,
|
| 661 | use 'ps A' to see all.
|
| 662 | Task Addr Pid Parent [*] cpu State Thread Command
|
| 663 |
|
| 664 | 0xc78291d0 1 0 0 0 S 0xc7829404 init
|
| 665 | 0xc7954150 942 1 0 0 S 0xc7954384 dropbear
|
| 666 | 0xc78789c0 944 1 0 0 S 0xc7878bf4 sh
|
| 667 | (gdb)
|
| 668 |
|
| 669 | kgdb Test Suite
|
| 670 | ===============
|
| 671 |
|
| 672 | When kgdb is enabled in the kernel config you can also elect to enable
|
| 673 | the config parameter ``KGDB_TESTS``. Turning this on will enable a special
|
| 674 | kgdb I/O module which is designed to test the kgdb internal functions.
|
| 675 |
|
| 676 | The kgdb tests are mainly intended for developers to test the kgdb
|
| 677 | internals as well as a tool for developing a new kgdb architecture
|
| 678 | specific implementation. These tests are not really for end users of the
|
| 679 | Linux kernel. The primary source of documentation would be to look in
|
| 680 | the ``drivers/misc/kgdbts.c`` file.
|
| 681 |
|
| 682 | The kgdb test suite can also be configured at compile time to run the
|
| 683 | core set of tests by setting the kernel config parameter
|
| 684 | ``KGDB_TESTS_ON_BOOT``. This particular option is aimed at automated
|
| 685 | regression testing and does not require modifying the kernel boot config
|
| 686 | arguments. If this is turned on, the kgdb test suite can be disabled by
|
| 687 | specifying ``kgdbts=`` as a kernel boot argument.
|
| 688 |
|
| 689 | Kernel Debugger Internals
|
| 690 | =========================
|
| 691 |
|
| 692 | Architecture Specifics
|
| 693 | ----------------------
|
| 694 |
|
| 695 | The kernel debugger is organized into a number of components:
|
| 696 |
|
| 697 | 1. The debug core
|
| 698 |
|
| 699 | The debug core is found in ``kernel/debug/debug_core.c``. It
|
| 700 | contains:
|
| 701 |
|
| 702 | - A generic OS exception handler which includes sync'ing the
|
| 703 | processors into a stopped state on an multi-CPU system.
|
| 704 |
|
| 705 | - The API to talk to the kgdb I/O drivers
|
| 706 |
|
| 707 | - The API to make calls to the arch-specific kgdb implementation
|
| 708 |
|
| 709 | - The logic to perform safe memory reads and writes to memory while
|
| 710 | using the debugger
|
| 711 |
|
| 712 | - A full implementation for software breakpoints unless overridden
|
| 713 | by the arch
|
| 714 |
|
| 715 | - The API to invoke either the kdb or kgdb frontend to the debug
|
| 716 | core.
|
| 717 |
|
| 718 | - The structures and callback API for atomic kernel mode setting.
|
| 719 |
|
| 720 | .. note:: kgdboc is where the kms callbacks are invoked.
|
| 721 |
|
| 722 | 2. kgdb arch-specific implementation
|
| 723 |
|
| 724 | This implementation is generally found in ``arch/*/kernel/kgdb.c``. As
|
| 725 | an example, ``arch/x86/kernel/kgdb.c`` contains the specifics to
|
| 726 | implement HW breakpoint as well as the initialization to dynamically
|
| 727 | register and unregister for the trap handlers on this architecture.
|
| 728 | The arch-specific portion implements:
|
| 729 |
|
| 730 | - contains an arch-specific trap catcher which invokes
|
| 731 | kgdb_handle_exception() to start kgdb about doing its work
|
| 732 |
|
| 733 | - translation to and from gdb specific packet format to struct pt_regs
|
| 734 |
|
| 735 | - Registration and unregistration of architecture specific trap
|
| 736 | hooks
|
| 737 |
|
| 738 | - Any special exception handling and cleanup
|
| 739 |
|
| 740 | - NMI exception handling and cleanup
|
| 741 |
|
| 742 | - (optional) HW breakpoints
|
| 743 |
|
| 744 | 3. gdbstub frontend (aka kgdb)
|
| 745 |
|
| 746 | The gdbstub is located in ``kernel/debug/gdbstub.c``. It contains:
|
| 747 |
|
| 748 | - All the logic to implement the gdb serial protocol
|
| 749 |
|
| 750 | 4. kdb frontend
|
| 751 |
|
| 752 | The kdb debugger shell is broken down into a number of components.
|
| 753 | The kdb core is located in kernel/debug/kdb. There are a number of
|
| 754 | helper functions in some of the other kernel components to make it
|
| 755 | possible for kdb to examine and report information about the kernel
|
| 756 | without taking locks that could cause a kernel deadlock. The kdb core
|
| 757 | implements the following functionality.
|
| 758 |
|
| 759 | - A simple shell
|
| 760 |
|
| 761 | - The kdb core command set
|
| 762 |
|
| 763 | - A registration API to register additional kdb shell commands.
|
| 764 |
|
| 765 | - A good example of a self-contained kdb module is the ``ftdump``
|
| 766 | command for dumping the ftrace buffer. See:
|
| 767 | ``kernel/trace/trace_kdb.c``
|
| 768 |
|
| 769 | - For an example of how to dynamically register a new kdb command
|
| 770 | you can build the kdb_hello.ko kernel module from
|
| 771 | ``samples/kdb/kdb_hello.c``. To build this example you can set
|
| 772 | ``CONFIG_SAMPLES=y`` and ``CONFIG_SAMPLE_KDB=m`` in your kernel
|
| 773 | config. Later run ``modprobe kdb_hello`` and the next time you
|
| 774 | enter the kdb shell, you can run the ``hello`` command.
|
| 775 |
|
| 776 | - The implementation for kdb_printf() which emits messages directly
|
| 777 | to I/O drivers, bypassing the kernel log.
|
| 778 |
|
| 779 | - SW / HW breakpoint management for the kdb shell
|
| 780 |
|
| 781 | 5. kgdb I/O driver
|
| 782 |
|
| 783 | Each kgdb I/O driver has to provide an implementation for the
|
| 784 | following:
|
| 785 |
|
| 786 | - configuration via built-in or module
|
| 787 |
|
| 788 | - dynamic configuration and kgdb hook registration calls
|
| 789 |
|
| 790 | - read and write character interface
|
| 791 |
|
| 792 | - A cleanup handler for unconfiguring from the kgdb core
|
| 793 |
|
| 794 | - (optional) Early debug methodology
|
| 795 |
|
| 796 | Any given kgdb I/O driver has to operate very closely with the
|
| 797 | hardware and must do it in such a way that does not enable interrupts
|
| 798 | or change other parts of the system context without completely
|
| 799 | restoring them. The kgdb core will repeatedly "poll" a kgdb I/O
|
| 800 | driver for characters when it needs input. The I/O driver is expected
|
| 801 | to return immediately if there is no data available. Doing so allows
|
| 802 | for the future possibility to touch watchdog hardware in such a way
|
| 803 | as to have a target system not reset when these are enabled.
|
| 804 |
|
| 805 | If you are intent on adding kgdb architecture specific support for a new
|
| 806 | architecture, the architecture should define ``HAVE_ARCH_KGDB`` in the
|
| 807 | architecture specific Kconfig file. This will enable kgdb for the
|
| 808 | architecture, and at that point you must create an architecture specific
|
| 809 | kgdb implementation.
|
| 810 |
|
| 811 | There are a few flags which must be set on every architecture in their
|
| 812 | ``asm/kgdb.h`` file. These are:
|
| 813 |
|
| 814 | - ``NUMREGBYTES``:
|
| 815 | The size in bytes of all of the registers, so that we
|
| 816 | can ensure they will all fit into a packet.
|
| 817 |
|
| 818 | - ``BUFMAX``:
|
| 819 | The size in bytes of the buffer GDB will read into. This must
|
| 820 | be larger than NUMREGBYTES.
|
| 821 |
|
| 822 | - ``CACHE_FLUSH_IS_SAFE``:
|
| 823 | Set to 1 if it is always safe to call
|
| 824 | flush_cache_range or flush_icache_range. On some architectures,
|
| 825 | these functions may not be safe to call on SMP since we keep other
|
| 826 | CPUs in a holding pattern.
|
| 827 |
|
| 828 | There are also the following functions for the common backend, found in
|
| 829 | ``kernel/kgdb.c``, that must be supplied by the architecture-specific
|
| 830 | backend unless marked as (optional), in which case a default function
|
| 831 | maybe used if the architecture does not need to provide a specific
|
| 832 | implementation.
|
| 833 |
|
| 834 | .. kernel-doc:: include/linux/kgdb.h
|
| 835 | :internal:
|
| 836 |
|
| 837 | kgdboc internals
|
| 838 | ----------------
|
| 839 |
|
| 840 | kgdboc and uarts
|
| 841 | ~~~~~~~~~~~~~~~~
|
| 842 |
|
| 843 | The kgdboc driver is actually a very thin driver that relies on the
|
| 844 | underlying low level to the hardware driver having "polling hooks" to
|
| 845 | which the tty driver is attached. In the initial implementation of
|
| 846 | kgdboc the serial_core was changed to expose a low level UART hook for
|
| 847 | doing polled mode reading and writing of a single character while in an
|
| 848 | atomic context. When kgdb makes an I/O request to the debugger, kgdboc
|
| 849 | invokes a callback in the serial core which in turn uses the callback in
|
| 850 | the UART driver.
|
| 851 |
|
| 852 | When using kgdboc with a UART, the UART driver must implement two
|
| 853 | callbacks in the struct uart_ops.
|
| 854 | Example from ``drivers/8250.c``::
|
| 855 |
|
| 856 |
|
| 857 | #ifdef CONFIG_CONSOLE_POLL
|
| 858 | .poll_get_char = serial8250_get_poll_char,
|
| 859 | .poll_put_char = serial8250_put_poll_char,
|
| 860 | #endif
|
| 861 |
|
| 862 |
|
| 863 | Any implementation specifics around creating a polling driver use the
|
| 864 | ``#ifdef CONFIG_CONSOLE_POLL``, as shown above. Keep in mind that
|
| 865 | polling hooks have to be implemented in such a way that they can be
|
| 866 | called from an atomic context and have to restore the state of the UART
|
| 867 | chip on return such that the system can return to normal when the
|
| 868 | debugger detaches. You need to be very careful with any kind of lock you
|
| 869 | consider, because failing here is most likely going to mean pressing the
|
| 870 | reset button.
|
| 871 |
|
| 872 | kgdboc and keyboards
|
| 873 | ~~~~~~~~~~~~~~~~~~~~~~~~
|
| 874 |
|
| 875 | The kgdboc driver contains logic to configure communications with an
|
| 876 | attached keyboard. The keyboard infrastructure is only compiled into the
|
| 877 | kernel when ``CONFIG_KDB_KEYBOARD=y`` is set in the kernel configuration.
|
| 878 |
|
| 879 | The core polled keyboard driver for PS/2 type keyboards is in
|
| 880 | ``kernel/debug/kdb/kdb_keyboard.c``. This driver is hooked into the debug core
|
| 881 | when kgdboc populates the callback in the array called
|
| 882 | :c:expr:`kdb_poll_funcs[]`. The kdb_get_kbd_char() is the top-level
|
| 883 | function which polls hardware for single character input.
|
| 884 |
|
| 885 | kgdboc and kms
|
| 886 | ~~~~~~~~~~~~~~~~~~
|
| 887 |
|
| 888 | The kgdboc driver contains logic to request the graphics display to
|
| 889 | switch to a text context when you are using ``kgdboc=kms,kbd``, provided
|
| 890 | that you have a video driver which has a frame buffer console and atomic
|
| 891 | kernel mode setting support.
|
| 892 |
|
| 893 | Every time the kernel debugger is entered it calls
|
| 894 | kgdboc_pre_exp_handler() which in turn calls con_debug_enter()
|
| 895 | in the virtual console layer. On resuming kernel execution, the kernel
|
| 896 | debugger calls kgdboc_post_exp_handler() which in turn calls
|
| 897 | con_debug_leave().
|
| 898 |
|
| 899 |
|
| 900 |
|
| 901 | Credits
|
| 902 | =======
|
| 903 |
|
| 904 | The following people have contributed to this document:
|
| 905 |
|
| 906 | 1. Amit Kale <[email protected]>
|
| 907 |
|
| 908 | 2. Tom Rini <[email protected]>
|
| 909 |
|
| 910 | In March 2008 this document was completely rewritten by:
|
| 911 |
|
| 912 | - Jason Wessel <[email protected]>
|
| 913 |
|
| 914 | In Jan 2010 this document was updated to include kdb.
|
| 915 |
|
| 916 | - Jason Wessel <[email protected]>
|