| 1 | .. highlight:: none
|
| 2 |
|
| 3 | Debugging kernel and modules via gdb
|
| 4 | ====================================
|
| 5 |
|
| 6 | The kernel debugger kgdb, hypervisors like QEMU or JTAG-based hardware
|
| 7 | interfaces allow to debug the Linux kernel and its modules during runtime
|
| 8 | using gdb. Gdb comes with a powerful scripting interface for python. The
|
| 9 | kernel provides a collection of helper scripts that can simplify typical
|
| 10 | kernel debugging steps. This is a short tutorial about how to enable and use
|
| 11 | them. It focuses on QEMU/KVM virtual machines as target, but the examples can
|
| 12 | be transferred to the other gdb stubs as well.
|
| 13 |
|
| 14 |
|
| 15 | Requirements
|
| 16 | ------------
|
| 17 |
|
| 18 | - gdb 7.2+ (recommended: 7.4+) with python support enabled (typically true
|
| 19 | for distributions)
|
| 20 |
|
| 21 |
|
| 22 | Setup
|
| 23 | -----
|
| 24 |
|
| 25 | - Create a virtual Linux machine for QEMU/KVM (see www.linux-kvm.org and
|
| 26 | www.qemu.org for more details). For cross-development,
|
| 27 | https://landley.net/aboriginal/bin keeps a pool of machine images and
|
| 28 | toolchains that can be helpful to start from.
|
| 29 |
|
| 30 | - Build the kernel with CONFIG_GDB_SCRIPTS enabled, but leave
|
| 31 | CONFIG_DEBUG_INFO_REDUCED off. If your architecture supports
|
| 32 | CONFIG_FRAME_POINTER, keep it enabled.
|
| 33 |
|
| 34 | - Install that kernel on the guest, turn off KASLR if necessary by adding
|
| 35 | "nokaslr" to the kernel command line.
|
| 36 | Alternatively, QEMU allows to boot the kernel directly using -kernel,
|
| 37 | -append, -initrd command line switches. This is generally only useful if
|
| 38 | you do not depend on modules. See QEMU documentation for more details on
|
| 39 | this mode. In this case, you should build the kernel with
|
| 40 | CONFIG_RANDOMIZE_BASE disabled if the architecture supports KASLR.
|
| 41 |
|
| 42 | - Enable the gdb stub of QEMU/KVM, either
|
| 43 |
|
| 44 | - at VM startup time by appending "-s" to the QEMU command line
|
| 45 |
|
| 46 | or
|
| 47 |
|
| 48 | - during runtime by issuing "gdbserver" from the QEMU monitor
|
| 49 | console
|
| 50 |
|
| 51 | - cd /path/to/linux-build
|
| 52 |
|
| 53 | - Start gdb: gdb vmlinux
|
| 54 |
|
| 55 | Note: Some distros may restrict auto-loading of gdb scripts to known safe
|
| 56 | directories. In case gdb reports to refuse loading vmlinux-gdb.py, add::
|
| 57 |
|
| 58 | add-auto-load-safe-path /path/to/linux-build
|
| 59 |
|
| 60 | to ~/.gdbinit. See gdb help for more details.
|
| 61 |
|
| 62 | - Attach to the booted guest::
|
| 63 |
|
| 64 | (gdb) target remote :1234
|
| 65 |
|
| 66 |
|
| 67 | Examples of using the Linux-provided gdb helpers
|
| 68 | ------------------------------------------------
|
| 69 |
|
| 70 | - Load module (and main kernel) symbols::
|
| 71 |
|
| 72 | (gdb) lx-symbols
|
| 73 | loading vmlinux
|
| 74 | scanning for modules in /home/user/linux/build
|
| 75 | loading @0xffffffffa0020000: /home/user/linux/build/net/netfilter/xt_tcpudp.ko
|
| 76 | loading @0xffffffffa0016000: /home/user/linux/build/net/netfilter/xt_pkttype.ko
|
| 77 | loading @0xffffffffa0002000: /home/user/linux/build/net/netfilter/xt_limit.ko
|
| 78 | loading @0xffffffffa00ca000: /home/user/linux/build/net/packet/af_packet.ko
|
| 79 | loading @0xffffffffa003c000: /home/user/linux/build/fs/fuse/fuse.ko
|
| 80 | ...
|
| 81 | loading @0xffffffffa0000000: /home/user/linux/build/drivers/ata/ata_generic.ko
|
| 82 |
|
| 83 | - Set a breakpoint on some not yet loaded module function, e.g.::
|
| 84 |
|
| 85 | (gdb) b btrfs_init_sysfs
|
| 86 | Function "btrfs_init_sysfs" not defined.
|
| 87 | Make breakpoint pending on future shared library load? (y or [n]) y
|
| 88 | Breakpoint 1 (btrfs_init_sysfs) pending.
|
| 89 |
|
| 90 | - Continue the target::
|
| 91 |
|
| 92 | (gdb) c
|
| 93 |
|
| 94 | - Load the module on the target and watch the symbols being loaded as well as
|
| 95 | the breakpoint hit::
|
| 96 |
|
| 97 | loading @0xffffffffa0034000: /home/user/linux/build/lib/libcrc32c.ko
|
| 98 | loading @0xffffffffa0050000: /home/user/linux/build/lib/lzo/lzo_compress.ko
|
| 99 | loading @0xffffffffa006e000: /home/user/linux/build/lib/zlib_deflate/zlib_deflate.ko
|
| 100 | loading @0xffffffffa01b1000: /home/user/linux/build/fs/btrfs/btrfs.ko
|
| 101 |
|
| 102 | Breakpoint 1, btrfs_init_sysfs () at /home/user/linux/fs/btrfs/sysfs.c:36
|
| 103 | 36 btrfs_kset = kset_create_and_add("btrfs", NULL, fs_kobj);
|
| 104 |
|
| 105 | - Dump the log buffer of the target kernel::
|
| 106 |
|
| 107 | (gdb) lx-dmesg
|
| 108 | [ 0.000000] Initializing cgroup subsys cpuset
|
| 109 | [ 0.000000] Initializing cgroup subsys cpu
|
| 110 | [ 0.000000] Linux version 3.8.0-rc4-dbg+ (...
|
| 111 | [ 0.000000] Command line: root=/dev/sda2 resume=/dev/sda1 vga=0x314
|
| 112 | [ 0.000000] e820: BIOS-provided physical RAM map:
|
| 113 | [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
|
| 114 | [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved
|
| 115 | ....
|
| 116 |
|
| 117 | - Examine fields of the current task struct(supported by x86 and arm64 only)::
|
| 118 |
|
| 119 | (gdb) p $lx_current().pid
|
| 120 | $1 = 4998
|
| 121 | (gdb) p $lx_current().comm
|
| 122 | $2 = "modprobe\000\000\000\000\000\000\000"
|
| 123 |
|
| 124 | - Make use of the per-cpu function for the current or a specified CPU::
|
| 125 |
|
| 126 | (gdb) p $lx_per_cpu("runqueues").nr_running
|
| 127 | $3 = 1
|
| 128 | (gdb) p $lx_per_cpu("runqueues", 2).nr_running
|
| 129 | $4 = 0
|
| 130 |
|
| 131 | - Dig into hrtimers using the container_of helper::
|
| 132 |
|
| 133 | (gdb) set $next = $lx_per_cpu("hrtimer_bases").clock_base[0].active.next
|
| 134 | (gdb) p *$container_of($next, "struct hrtimer", "node")
|
| 135 | $5 = {
|
| 136 | node = {
|
| 137 | node = {
|
| 138 | __rb_parent_color = 18446612133355256072,
|
| 139 | rb_right = 0x0 <irq_stack_union>,
|
| 140 | rb_left = 0x0 <irq_stack_union>
|
| 141 | },
|
| 142 | expires = {
|
| 143 | tv64 = 1835268000000
|
| 144 | }
|
| 145 | },
|
| 146 | _softexpires = {
|
| 147 | tv64 = 1835268000000
|
| 148 | },
|
| 149 | function = 0xffffffff81078232 <tick_sched_timer>,
|
| 150 | base = 0xffff88003fd0d6f0,
|
| 151 | state = 1,
|
| 152 | start_pid = 0,
|
| 153 | start_site = 0xffffffff81055c1f <hrtimer_start_range_ns+20>,
|
| 154 | start_comm = "swapper/2\000\000\000\000\000\000"
|
| 155 | }
|
| 156 |
|
| 157 |
|
| 158 | List of commands and functions
|
| 159 | ------------------------------
|
| 160 |
|
| 161 | The number of commands and convenience functions may evolve over the time,
|
| 162 | this is just a snapshot of the initial version::
|
| 163 |
|
| 164 | (gdb) apropos lx
|
| 165 | function lx_current -- Return current task
|
| 166 | function lx_module -- Find module by name and return the module variable
|
| 167 | function lx_per_cpu -- Return per-cpu variable
|
| 168 | function lx_task_by_pid -- Find Linux task by PID and return the task_struct variable
|
| 169 | function lx_thread_info -- Calculate Linux thread_info from task variable
|
| 170 | lx-dmesg -- Print Linux kernel log buffer
|
| 171 | lx-lsmod -- List currently loaded modules
|
| 172 | lx-symbols -- (Re-)load symbols of Linux kernel and currently loaded modules
|
| 173 |
|
| 174 | Detailed help can be obtained via "help <command-name>" for commands and "help
|
| 175 | function <function-name>" for convenience functions.
|