Commit | Line | Data |
---|---|---|
5f096274 JC |
1 | .. highlight:: none |
2 | ||
bda1a921 JK |
3 | Debugging kernel and modules via gdb |
4 | ==================================== | |
5 | ||
6 | The kernel debugger kgdb, hypervisors like QEMU or JTAG-based hardware | |
7 | interfaces allow to debug the Linux kernel and its modules during runtime | |
8 | using gdb. Gdb comes with a powerful scripting interface for python. The | |
9 | kernel provides a collection of helper scripts that can simplify typical | |
10 | kernel debugging steps. This is a short tutorial about how to enable and use | |
11 | them. It focuses on QEMU/KVM virtual machines as target, but the examples can | |
12 | be transferred to the other gdb stubs as well. | |
13 | ||
14 | ||
15 | Requirements | |
16 | ------------ | |
17 | ||
5f096274 JC |
18 | - gdb 7.2+ (recommended: 7.4+) with python support enabled (typically true |
19 | for distributions) | |
bda1a921 JK |
20 | |
21 | ||
22 | Setup | |
23 | ----- | |
24 | ||
5f096274 JC |
25 | - Create a virtual Linux machine for QEMU/KVM (see www.linux-kvm.org and |
26 | www.qemu.org for more details). For cross-development, | |
27 | http://landley.net/aboriginal/bin keeps a pool of machine images and | |
28 | toolchains that can be helpful to start from. | |
bda1a921 | 29 | |
5f096274 JC |
30 | - Build the kernel with CONFIG_GDB_SCRIPTS enabled, but leave |
31 | CONFIG_DEBUG_INFO_REDUCED off. If your architecture supports | |
32 | CONFIG_FRAME_POINTER, keep it enabled. | |
bda1a921 | 33 | |
5f096274 JC |
34 | - Install that kernel on the guest. |
35 | Alternatively, QEMU allows to boot the kernel directly using -kernel, | |
36 | -append, -initrd command line switches. This is generally only useful if | |
37 | you do not depend on modules. See QEMU documentation for more details on | |
38 | this mode. | |
bda1a921 | 39 | |
5f096274 | 40 | - Enable the gdb stub of QEMU/KVM, either |
bda1a921 | 41 | |
bda1a921 | 42 | - at VM startup time by appending "-s" to the QEMU command line |
5f096274 JC |
43 | |
44 | or | |
45 | ||
bda1a921 JK |
46 | - during runtime by issuing "gdbserver" from the QEMU monitor |
47 | console | |
48 | ||
5f096274 | 49 | - cd /path/to/linux-build |
bda1a921 | 50 | |
5f096274 | 51 | - Start gdb: gdb vmlinux |
bda1a921 | 52 | |
5f096274 JC |
53 | Note: Some distros may restrict auto-loading of gdb scripts to known safe |
54 | directories. In case gdb reports to refuse loading vmlinux-gdb.py, add:: | |
bda1a921 JK |
55 | |
56 | add-auto-load-safe-path /path/to/linux-build | |
57 | ||
5f096274 JC |
58 | to ~/.gdbinit. See gdb help for more details. |
59 | ||
60 | - Attach to the booted guest:: | |
bda1a921 | 61 | |
bda1a921 JK |
62 | (gdb) target remote :1234 |
63 | ||
64 | ||
65 | Examples of using the Linux-provided gdb helpers | |
66 | ------------------------------------------------ | |
67 | ||
5f096274 JC |
68 | - Load module (and main kernel) symbols:: |
69 | ||
bda1a921 JK |
70 | (gdb) lx-symbols |
71 | loading vmlinux | |
72 | scanning for modules in /home/user/linux/build | |
73 | loading @0xffffffffa0020000: /home/user/linux/build/net/netfilter/xt_tcpudp.ko | |
74 | loading @0xffffffffa0016000: /home/user/linux/build/net/netfilter/xt_pkttype.ko | |
75 | loading @0xffffffffa0002000: /home/user/linux/build/net/netfilter/xt_limit.ko | |
76 | loading @0xffffffffa00ca000: /home/user/linux/build/net/packet/af_packet.ko | |
77 | loading @0xffffffffa003c000: /home/user/linux/build/fs/fuse/fuse.ko | |
78 | ... | |
79 | loading @0xffffffffa0000000: /home/user/linux/build/drivers/ata/ata_generic.ko | |
80 | ||
5f096274 JC |
81 | - Set a breakpoint on some not yet loaded module function, e.g.:: |
82 | ||
bda1a921 JK |
83 | (gdb) b btrfs_init_sysfs |
84 | Function "btrfs_init_sysfs" not defined. | |
85 | Make breakpoint pending on future shared library load? (y or [n]) y | |
86 | Breakpoint 1 (btrfs_init_sysfs) pending. | |
87 | ||
5f096274 JC |
88 | - Continue the target:: |
89 | ||
bda1a921 JK |
90 | (gdb) c |
91 | ||
5f096274 JC |
92 | - Load the module on the target and watch the symbols being loaded as well as |
93 | the breakpoint hit:: | |
94 | ||
bda1a921 JK |
95 | loading @0xffffffffa0034000: /home/user/linux/build/lib/libcrc32c.ko |
96 | loading @0xffffffffa0050000: /home/user/linux/build/lib/lzo/lzo_compress.ko | |
97 | loading @0xffffffffa006e000: /home/user/linux/build/lib/zlib_deflate/zlib_deflate.ko | |
98 | loading @0xffffffffa01b1000: /home/user/linux/build/fs/btrfs/btrfs.ko | |
99 | ||
100 | Breakpoint 1, btrfs_init_sysfs () at /home/user/linux/fs/btrfs/sysfs.c:36 | |
101 | 36 btrfs_kset = kset_create_and_add("btrfs", NULL, fs_kobj); | |
102 | ||
5f096274 JC |
103 | - Dump the log buffer of the target kernel:: |
104 | ||
bda1a921 JK |
105 | (gdb) lx-dmesg |
106 | [ 0.000000] Initializing cgroup subsys cpuset | |
107 | [ 0.000000] Initializing cgroup subsys cpu | |
108 | [ 0.000000] Linux version 3.8.0-rc4-dbg+ (... | |
109 | [ 0.000000] Command line: root=/dev/sda2 resume=/dev/sda1 vga=0x314 | |
110 | [ 0.000000] e820: BIOS-provided physical RAM map: | |
111 | [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable | |
112 | [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved | |
113 | .... | |
114 | ||
5f096274 JC |
115 | - Examine fields of the current task struct:: |
116 | ||
bda1a921 JK |
117 | (gdb) p $lx_current().pid |
118 | $1 = 4998 | |
119 | (gdb) p $lx_current().comm | |
120 | $2 = "modprobe\000\000\000\000\000\000\000" | |
121 | ||
5f096274 JC |
122 | - Make use of the per-cpu function for the current or a specified CPU:: |
123 | ||
bda1a921 JK |
124 | (gdb) p $lx_per_cpu("runqueues").nr_running |
125 | $3 = 1 | |
126 | (gdb) p $lx_per_cpu("runqueues", 2).nr_running | |
127 | $4 = 0 | |
128 | ||
5f096274 JC |
129 | - Dig into hrtimers using the container_of helper:: |
130 | ||
bda1a921 JK |
131 | (gdb) set $next = $lx_per_cpu("hrtimer_bases").clock_base[0].active.next |
132 | (gdb) p *$container_of($next, "struct hrtimer", "node") | |
133 | $5 = { | |
134 | node = { | |
135 | node = { | |
136 | __rb_parent_color = 18446612133355256072, | |
137 | rb_right = 0x0 <irq_stack_union>, | |
138 | rb_left = 0x0 <irq_stack_union> | |
139 | }, | |
140 | expires = { | |
141 | tv64 = 1835268000000 | |
142 | } | |
143 | }, | |
144 | _softexpires = { | |
145 | tv64 = 1835268000000 | |
146 | }, | |
147 | function = 0xffffffff81078232 <tick_sched_timer>, | |
148 | base = 0xffff88003fd0d6f0, | |
149 | state = 1, | |
150 | start_pid = 0, | |
151 | start_site = 0xffffffff81055c1f <hrtimer_start_range_ns+20>, | |
152 | start_comm = "swapper/2\000\000\000\000\000\000" | |
153 | } | |
154 | ||
155 | ||
156 | List of commands and functions | |
157 | ------------------------------ | |
158 | ||
159 | The number of commands and convenience functions may evolve over the time, | |
5f096274 | 160 | this is just a snapshot of the initial version:: |
bda1a921 JK |
161 | |
162 | (gdb) apropos lx | |
163 | function lx_current -- Return current task | |
164 | function lx_module -- Find module by name and return the module variable | |
165 | function lx_per_cpu -- Return per-cpu variable | |
166 | function lx_task_by_pid -- Find Linux task by PID and return the task_struct variable | |
167 | function lx_thread_info -- Calculate Linux thread_info from task variable | |
168 | lx-dmesg -- Print Linux kernel log buffer | |
169 | lx-lsmod -- List currently loaded modules | |
170 | lx-symbols -- (Re-)load symbols of Linux kernel and currently loaded modules | |
171 | ||
172 | Detailed help can be obtained via "help <command-name>" for commands and "help | |
173 | function <function-name>" for convenience functions. |