How to use KGDB
Summary
KGDB is a source-level debugger for Linux kernel. To make it simpler, KGDB can be regarded as another GDB agent, which resides in the Linux kernel. It is used along with GDB to debug a Linux kernel. GDB can be used to "break in" to the running kernel to inspect memory and variables and look through call stack information, similar to what an application developer would use GDB for. It is possible to place breakpoints in kernel code and perform some limited execution stepping.
A target board and a host machine are required for using KGDB. The kernel to be debugged runs on the target board. The host machine runs an instance of gdb against the vmlinux file which contains the symbols (not boot image such as bzImage, zImage, uImage...). There should be a connection between the target system and the host which depends on the availability of KGDB I/O modules. In the following example we'll use the serial line available on the target system also used as Linux console, feature known in KGDB as "over console" (kgdboc). Kgdboc is designed to work with a single serial port.
Prerequisites
The kernel to be debugged should be configured with KGDB support. Configure the kernel and re-build the kernel with the following included.
$ make kernel-menuconfig
[*] Include Kernel Hacking -> Kernel Debugging [*] Include Kernel Hacking -> Kernel Debugging -> KGDB: Kernel debugger [*] Include Kernel Hacking -> Kernel Debugging -> KGDB: Kernel debugger -> use kgdb over serial console [*] Include Kernel Hacking -> Compile the Kernel with debug info [*] General setup -> Configure standard kernel features (expert users) [*] General setup -> Load all symbols for debugging / ksymoops [*] General setup -> Load all symbols for debugging / ksymoops -> Include all symbols in kallsyms
If you want to debug the kernel during boot, select the Use kgdb over serial console to be compiled in kernel (*) and not as module (M).
Boot Arguments
The Kernel command line option kgdbwait makes KGDB wait for a debugger connection during booting of a kernel as soon as the I/O driver has been activated. You can only use this option you compiled a KGDB I/O driver into the kernel and you specified the I/O driver configuration as a kernel command line option. The kgdbwait parameter should always follow the configuration parameter for the KGDB I/O driver in the kernel command line else the I/O driver will not be configured prior to asking the kernel to use it to wait. At run time, the kgdboc driver must be activated and instructed how to operate which is achieved by providing the kernel command line option kgdboc=<tty-device>,[baud].
Example:
Bootargs — Vybrid Platform:
setenv bootargs mem=128M console=ttymxc1 ip=192.168.1.2:192.168.1.1:192.168.1.1:255.255.255.0 root=/dev/nfs rw nfsroot=192.168.1.1:/tftpboot/rfs kgdboc=ttymxc1,115200 kgdbwait
Connecting GDB
Boot the target with the respective uImage after providing the appropriate Bootargs. Observe that the target kernel boot process stop with the following message:
. . console [ttymxc1] enabled kgdb: Registered I/O driver kgdboc. kgdb: Waiting for connection from remote gdb...
If you are using the same serial line for both console and debug, now is the time to disconnect the terminal application (e.g. minicom). Then, start the host debugger using the vmlinux elf from your Linux build as input file. Once connected, you can debug a kernel the way you would debug an application program.
$gdb ./vmlinux
(gdb) set serial baud 115200
(gdb) target remote <tty-device>
Example:
$ export PATH=$PATH:/media/Data/Timesys/Faraday/FreescaleRelease3/factory/build_armv7l-timesys-linux-gnueabi/toolchain/bin
$ cd /media/Data/Timesys/Faraday/FreescaleRelease3/factory/build_armv7l-timesys-linux-gnueabi/linux-3.0
$ armv7l-timesys-linux-gnueabi-gdb vmlinux GNU gdb (Timesys 20121022) 7.4.1 Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "--host=i686-linux-gnu --target=armv7l-timesys-linux-gnueabi". For bug reporting instructions, please see: <https://linuxlink.timesys.com/support>... Reading symbols from /media/Data/Timesys/Faraday/FreescaleRelease3/factory/build_armv7l-timesys-linux-gnueabi/linux-3.0/vmlinux...done. (gdb)
Now setup GDB:
(gdb) set serial baud 115200
(gdb) target remote /dev/ttyUSB0 Remote debugging using /dev/ttyUSB0 kgdb_breakpoint () at kernel/debug/debug_core.c:959 959 arch_kgdb_breakpoint();
Step Debugging:
(gdb) next 960 wmb(); /* Sync point after breakpoint */
Set Breakpoint in Kernel Source/Built-in Drivers:
(gdb) break fsl_nfc.c:fsl_nfc_init
Breakpoint 1 at 0x8001b244: file drivers/mtd/nand/fsl_nfc.c, line 976.
(gdb) continue
Continuing.
Breakpoint 1, fsl_nfc_init () at drivers/mtd/nand/fsl_nfc.c:976
976 {
(gdb)
Continue the Kernel boot process:
(gdb) continue Continuing.
Other GDB commands:
break <symbol> list info thread thread <id> backtrace frame
Note: If you are having problems connecting or something is going seriously wrong while debugging, it will most often be the case that you want to enable GDB to be verbose about its target communications. You do this prior to issuing the target remote command by typing in the following on the Host machine:
set remote debug 1
Debugging a kernel module using KGDB
While the kernel body resides at well known virtual addresses (same as the load addresses from the vmlinux elf file), a loadable module (.ko) is just an archive, a collection of sections with no valid load address. The kernel loader allocates memory and decides where to load each loadable section of such a module. This means that, before module load (modprobe, insmod), we don't know where in the virtual memory it will stay, but we can find out immediately after the module load. The file kernel/module.c is the loader. Looking into this file, observe that some features are guarded by CONFIG_KALLSYMS definition. In order to export section load information to userspace, the kernel must be compiled with this macro enabled.
[*] General setup -> Load all symbols for debugging / ksymoops -> Include all symbols in kallsyms
So per-section load information can be availed as follows:
# cd /opt
# insmod modex.ko
# ls /sys/module/modex/sections/.* /sys/module/modex/sections/.bss /sys/module/modex/sections/.data /sys/module/modex/sections/.exit.text /sys/module/modex/sections/.gnu.linkonce.this_module /sys/module/modex/sections/.init.text /sys/module/modex/sections/.note.gnu.build-id /sys/module/modex/sections/.rodata.str1.4 /sys/module/modex/sections/.strtab /sys/module/modex/sections/.symtab /sys/module/modex/sections/.text /sys/module/modex/sections/.: /sys/module/modex/sections/..: holders initstate notes refcnt sections srcversion version
# cat /sys/module/modex/sections/.bss 0x7f006558
# cat /sys/module/modex/sections/.data 0x7f0063cc
# cat /sys/module/modex/sections/.init.text 0x7f008000
# cat /sys/module/modex/sections/.text 0x7f006000
# cat /sys/module/modex/sections/.rodata.str1.4 0x7f006270
# cat /sys/module/modex/sections/.strtab 0x7f008534
# cat /sys/module/modex/sections/.symtab 0x7f0080a4
# cat /sys/module/modex/sections/.exit.text 0x7f006220
# cat /sys/module/modex/sections/.gnu.linkonce.this_module 0x7f006434
# cat /sys/module/modex/sections/.note.gnu.build-id 0x7f0063a8
# echo "ttymxc1,115200" > /sys/module/kgdboc/parameters/kgdboc kgdb: Unregistered I/O driver kgdboc, debugger disabled. kgdb: Registered I/O driver kgdboc.
# echo g > /proc/sysrq-triggerSysRq : DEBUG Entering KGDB
Each file starting with the following characters: "." contains the virtual load address of a loadable section. Some of those are data sections (e.g. for variables), others are code sections. At the very least, use the values for .text, .data, .rodata and .bss to instruct the GDB debugger about the load addresses:
(gdb) add-symbol-file <filename.ko> <text_section_load_address> [-s .<SECT> <SECT_LOAD_ADDRESS>]*
On Host Machine:
nodeax@Nodeax:/media/Data/Timesys/Faraday/FreescaleRelease3/factory/build_armv7l-timesys-linux-gnueabi/linux-3.0$ armv7l-timesys-linux-gnueabi-gdb GNU gdb (Timesys 20121022) 7.4.1 Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "--host=i686-linux-gnu --target=armv7l-timesys-linux-gnueabi". For bug reporting instructions, please see: <https://linuxlink.timesys.com/support>.
(gdb) add-symbol-file /media/Data/Timesys/Faraday/Vybrid-DS5/kernel_module/generic/modex.ko 0x7f008000 -s .data 0x7f0063cc -s .bss 0x7f006558 -s .rodata.str1.4 0x7f006270
add symbol table from file "/media/Data/Timesys/Faraday/Vybrid-DS5/kernel_module/generic/modex.ko" at
.text_addr = 0x7f006000
.data_addr = 0x7f0063cc
.bss_addr = 0x7f006558
.init.text_addr = 0x7f008000
.rodata.str1.4_addr = 0x7f006270
(y or n) y
Reading symbols from /media/Data/Timesys/Faraday/Vybrid-DS5/kernel_module/generic/modex.ko...done.(gdb) set serial baud 115200
(gdb) target remote /dev/ttyUSB0
You should now be able to set breakpoints in the loadable module and debug it in the same session of a kernel debug.
Debugging a kernel module init section using KGDB
To debug the kernel module init section, set a kernel breakpoint in the file kernel/module.c at the place where the kernel calls the module's init function (do_one_initcall(mod->init)). At this point, all module sections would have been loaded into virtual memory. Inspect kernel variables' values and find the load address for .init.text section and provide it to the debugger in the add-symbol-file command. Thereafter you should then be able to set a breakpoint in the module's init function.
nodeax@Nodeax:/media/Data/Timesys/DM3730LogicPD-Compilation-Issue/factory-20130321-logicpd-2.3-2/build_armv7l-timesys-linux-gnueabi/toolchain/bin$ ./armv7l-timesys-linux-gnueabi-gdb /media/Data/Timesys/DM3730LogicPD-Compilation-Issue/factory-20130321-logicpd-2.3-2/build_armv7l-timesys-linux-gnueabi/linux-3.0/vmlinux GNU gdb (Timesys 20130318) 7.4.1 Copyright (C) 2012 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "--host=i686-linux-gnu --target=armv7l-timesys-linux-gnueabi". For bug reporting instructions, please see: <https://linuxlink.timesys.com/support>... Reading symbols from /media/Data/Timesys/DM3730LogicPD-Compilation-Issue/factory-20130321-logicpd-2.3-2/build_armv7l-timesys-linux-gnueabi/linux-3.0/vmlinux...done.
(gdb) set serial baud 115200
(gdb) target remote /dev/ttyUSB0 Remote debugging using /dev/ttyUSB0 kgdb_breakpoint () at kernel/debug/debug_core.c:959 959 arch_kgdb_breakpoint();
(gdb) br module.c:2915 Breakpoint 1 at 0xc00c3548: file kernel/module.c, line 2915.
(gdb) c Continuing.
Now on the Host machine, setup an SSH connection with the target board after boot process is complete and install the module.
nodeax@Nodeax:/media/Data/Timesys/DM3730LogicPD-Compilation-Issue/factory-20130321-logicpd-2.3-2/build_armv7l-timesys-linux-gnueabi/linux-3.0$ ssh root@192.168.1.21 root@192.168.1.21's password: BusyBox v1.20.2 (2013-04-02 06:53:27 EDT) built-in shell (ash) Enter 'help' for a list of built-in commands.
# cd /opt # ls modex.ko # insmod modex.ko
Now the Kernel execution should stop at module.c:
[New Thread 341] [Switching to Thread 341] Breakpoint 1, sys_init_module (umod=<optimized out>, len=<optimized out>, uargs=0x0) at kernel/module.c:2915 2915 if (mod->init != NULL)
(gdb) next 2916 ret = do_one_initcall(mod->init);
(gdb) step do_one_initcall (fn=0xbf002000) at init/main.c:671 671 if (initcall_debug)
(gdb) add-symbol-file /media/Data/Timesys/Faraday/Vybrid-DS5/kernel_module/generic/modex.ko 0xbf000000
add symbol table from file "/media/Data/Timesys/Faraday/Vybrid-DS5/kernel_module/generic/modex.ko" at
.text_addr = 0xbf000000(gdb)
Now you can step through the init section of the module / any function in the module and add breakpoints in the module source.
Debugging the Kernel After Boot Sysrq-g
KGDB I/O driver can also be re-configured from the sysfs interface after the kernel boots by echo’ing a new config string to /sys/module/kgdboc/parameters/kgdboc. The driver can be un-configured by passing an empty string. You cannot change the configuration while the debugger is attached. Make sure to detach the debugger with the detach command prior to trying un-configure a KGDB I/O driver.
echo <tty-device> > /sys/module/kgdboc/parameters/kgdboc
Example:
echo "ttymxc1,115200" > /sys/module/kgdboc/parameters/kgdboc kgdb: Unregistered I/O driver kgdboc, debugger disabled. kgdb: Registered I/O driver kgdboc.
When using kgdboc there is no way the user can trigger an interrupt from host debugger in order to enter a debug session, so a sysrq-g sequence should be initiated on target. It will be followed by a confirmation message:
echo g > /proc/sysrq-trigger SysRq : DEBUG Entering KGDB
Now the target waits for the host debugger to connect and the same steps from Debugging the kernel boot phase should be followed.
Additional Resources
http://kernel.org/pub/linux/kernel/people/jwessel/kdb/
