A KDB / KGDB session on the popular Raspberry Pi embedded Linux board

Assumptions / Pre-reqs

For this post to be useful, you should:

– know how to build a Linux kernel from source

– know something about Linux kernel programming, writing kernel module code, etc

– have some familiarity with setting up and using KDB and KGDB (a bit of this is covered here, not all); also, see some useful Resources just below..

– have an R Pi (I use the Rev B R Pi) with an SD card

– have a custom Linux kernel running on it (need to be able to modify kernel configuration and rebuild at will)

– the R Pi does not have a dedicated physical serial port; we require one to get (and send) console I/O (so that we can see kernel printk’s and interact via the keyboard). I find a simple and efficient way to do this is to make use of the GPIO pins 14 (TXD) and 15 (RXD) on the board, connecting them to a simple FTDI
USBTTL serial breakout board. I’m using FTDI’s FT232R Breakout board; it works very well indeed.

My R Pi (Model B) attached to a FTDI FT232R USB-to-TTL breakout board
My R Pi (Model B) attached to a FTDI FT232R USB-to-TTL breakout board

Above pic: My R Pi (Model B) attached to a FTDI FT232R USB-to-TTL breakout board.
Connections: (see photo)
          R Pi                                   FTDI
TXD (GPIO 14) RX-I              (RX-I and TX-O pins are at the front of the FTDI
RXD (GPIO 15) TX-O              board (directly opp the USB mini connector))
GND (GPIO 6)   GND

Yeah, quite a few pre-reqs huh 🙂

Resources

– Raspberry Pi on Wikipedia

– Using kgdb, kdb and the kernel debugger internals

– A good tutorial on building-from-scratch for the R Pi root filesystem and Linux kernel, using the excellent Buildroot tool,
can be found here.

Hi folks,

The R Pi (that’s Raspberry Pi for those of you still not wakey-wakey) is a fun powerful embedded computer to play with. Yes, it runs Linux (the Raspbian distro for R Pi is particularly popular), and yes, Linux does run into trouble now and then, especially when we geeky types fiddle with kernel-level touch-me-not’s.

So: when you do have a kernel bug, how does one proceed? A little googling will quickly reveal some powerful Linux kernel debugging tools- often  near the top of the list is KDB and KGDB [here’s a pretty neat page covering this on the awesome stackoverflow site].

This blog post is not intended to cover it all in gruesome detail; rather, it gives you a flavor of how to setup and use these tools on the R Pi running a custom (yes, that’s right – your very own compiled-from-source) Linux kernel.

First, configure the R Pi kernel source tree with the “usual” (described earlier) configuration options for KGDB. These usually include the following:

CONFIG_DEBUG_INFO=y    [Kernel Hacking / Compile-time checks and compiler options]
CONFIG_MAGIC_SYSRQ=y   [Kernel Hacking / Magic SysRq key] 

# CONFIG_DEBUG_RODATA is not set
CONFIG_FRAME_POINTER=y   [Kernel hacking /
Compile-time checks and compiler options]

CONFIG_KGDB=y            [Kernel hacking / KGDB: kernel debugging]
CONFIG_KGDB_SERIAL_CONSOLE=y

and, in addition for KDB, under the same menu:

CONFIG_KGDB_KDB=y
CONFIG_KDB_KEYBOARD=y

For the discussion below, we assume that the kernel has been successfully configured, built (and works) for the R Pi.

Scenario I  |  Using KDB at early Kernel Boot-time

To try this, setup the R Pi kernel to boot with the ‘kgdbwait‘ kernel parameter. How? Edit the ‘/boot/cmdline.txt‘ file in the boot partition of the R Pi SD card (usually mounted via the device file /dev/mmcblk0p1 on a Linux box, mount point usually is /boot). Here’s the kernel command-line am currently using :

# cat /boot/cmdline.txt
dwc_otg.lpm_enable=0 dwc_otg.speed=1 debug console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 kgdbwait console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait 
#

[Note:

1. Want more details? See Documentation/kernel-parameters. . Also, the “debug” param is not strictly required..

2. You will require a terminal emulator app running on the Linux host system- I’d highly recommend ‘screen’ over ‘minicom’:

# screen /dev/ttyUSB0 115200

]

Boot the R Pi!

Uncompressing Linux... done, booting the kernel.
*** start_kernel:487 :       << just my debug printk's; ignore em >>
Booting Linux on physical CPU 0
*** start_kernel:496 : 
Initializing cgroup subsys cpu
*** start_kernel:507 : 
Linux version 3.6.11+ (kaiwan@kaiwan-ThinkPad-X220) (gcc version 4.7.3 (Buildroot 2013.05) ) #13 PREEMPT Thu Jul 4 13:02:28 IST 2013
CPU: ARMv6-compatible processor [410fb767] revision 7 (ARMv7), cr=00c5387d
CPU: PIPT / VIPT nonaliasing data cache, VIPT nonaliasing instruction cache
Machine: BCM2708
--snip--
Console: switching to colour frame buffer device 82x26
kgdb: Registered I/O driver kgdboc.
kgdb: Waiting for connection from remote gdb...

<< KDB is entered automatically, as soon as the kgdb I/O modules are setup;
this is due to the ‘kgdbwait‘ kernel parameter! >>

Entering kdb (current=0xdb81abc0, pid 1) due to Keyboard Entry

kdb> ps
17 sleeping system daemon (state M) processes suppressed,
use 'ps A' to see all.
Task Addr Pid Parent [*] cpu State Thread Command
0xdb81abc0 1 0 1 0 R 0xdb81ada4 *swapper
0xdb81abc0 1 0 1 0 R 0xdb81ada4 *swapper
kdb> help
Command Usage Description
----------------------------------------------------------
md  Display Memory Contents, also mdWcN, e.g. md8c1
mdr   Display Raw Memory
mdp   Display Physical Memory
mds  Display Memory Symbolically
mm   Modify Memory Contents
go [] Continue Execution
rd Display Registers
rm   Modify Registers
ef  Display exception frame
bt [] Stack traceback
btp  Display stack for process 
bta [DRSTCZEUIMA] Display stack all processes
btc Backtrace current process on each cpu
btt  Backtrace process given its struct task address
ll   Switch to new cpu
kgdb Enter kgdb mode
ps [|A] Display active task list
pid  Switch to another task
reboot Reboot the machine immediately
lsmod List loaded kernel modules
sr  Magic SysRq key
dmesg [lines] Display syslog buffer
defcmd name "usage" "help" Define a set of commands, down to endefcmd
kill   Send a signal to a process
summary Summarize the system
per_cpu  [] [] Set/Display breakpoints
bl [] Display breakpoints
bc  Clear Breakpoint
be  Enable Breakpoint
bd  Disable Breakpoint
ss Single Step
ssb Single step to branch/call
dumpcommon Common kdb debugging
dumpall First line debugging
dumpcpu Same as dumpall but only tasks on cpus
kdb> 
kdb> bt
Stack traceback for pid 1
0xdb81abc0 1 0 1 0 R 0xdb81ada4 *swapper
Backtrace: 
[] (dump_backtrace+0x0/0x114) from [] (show_stack+0x20/0x24)
 r6:00000000 r5:c0550118 r4:c05a3458 r3:db82be68
[] (show_stack+0x0/0x24) from [] (kdb_show_stack+0x4c/0x68)
[] (kdb_show_stack+0x0/0x68) from [] (kdb_bt1.isra.0+0x98/0xe0)
 r8:00000032 r7:00000000 r6:00000000 r5:ffffffff r4:db81abc0
r3:c05a2de0
[] (kdb_bt1.isra.0+0x0/0xe0) from [] (kdb_bt+0x2a8/0x34c)
 r7:00000001 r6:00000000 r5:c05a3e1c r4:c05a3d44
[] (kdb_bt+0x0/0x34c) from [] (kdb_parse+0x2e0/0x684)
 r7:c05a3d44 r6:c05a373c r5:c05a3664 r4:00000009
[] (kdb_parse+0x0/0x684) from [] (kdb_main_loop+0x4f8/0x730)
[] (kdb_main_loop+0x0/0x730) from [] (kdb_stub+0x170/0x3a4)
[] (kdb_stub+0x0/0x3a4) from [] (kgdb_handle_exception+0x34c/0x6c0)
[] (kgdb_handle_exception+0x0/0x6c0) from [] (kgdb_compiled_brk_fn+0x34/0x40)
[] (kgdb_compiled_brk_fn+0x0/0x40) from [] (do_undefinstr+0x104/0x1e4)
[] (do_undefinstr+0x0/0x1e4) from [] (__und_svc_finish+0x0/0x34)
Exception stack(0xdb82be68 to 0xdb82beb0)
be60: 00000000 00000000 c059ef3c c059ef38 c059ef38 00000000
be80: c056c224 c056c1c8 c052bfc0 c052bfb4 c0522074 db82befc db82bf00 db82bef0
bea0: c006d484 c006c440 60000013 ffffffff
[] (kgdb_breakpoint+0x0/0x5c) from [] (kgdb_register_io_module+0xd0/0x1b8)
[] (kgdb_register_io_module+0x0/0x1b8) from [] (configure_kgdboc+0x104/0x1c4)
 r6:c056c1c8 r5:c0611a1c r4:c056c168 r3:00000000
[] (configure_kgdboc+0x0/0x1c4) from [] (init_kgdboc+0x20/0x30)
 r7:db82a000 r6:c0579ea0 r5:00000006 r4:c0541698
[] (init_kgdboc+0x0/0x30) from [] (do_one_initcall+0x130/0x1b0)
[] (do_one_initcall+0x0/0x1b0) from [] (kernel_init+0x13c/0x24c)
[] (kernel_init+0x0/0x24c) from [] (do_exit+0x0/0x7a8)
kdb> 
kdb> md 0xc0000000   << dump memory at kernel virtual address 0xc000 0000 >>
0xc0000000 ea000006 e1a00000 e1a00000 e1a00000 ................
0xc0000010 e1a00000 e1a00000 e1a00000 e1a00000 ................
0xc0000020 e3a00000 e3a01042 e3811c0c e59f2000 ....B........ ..
0xc0000030 e59ff000 00000100 00008000 00000000 ................
0xc0000040 00000000 00000000 00000000 00000000 ................
0xc0000050-0xc000006f zero suppressed
0xc0000070 00000000 00000000 00000000 00000000 ................
kdb> mdp 0x0   << dump memory at kernel physical address 0xc000 0000 >> 
phys 0x00000000 ea000006 e1a00000 e1a00000 e1a00000 ................
phys 0x00000010 e1a00000 e1a00000 e1a00000 e1a00000 ................
phys 0x00000020 e3a00000 e3a01042 e3811c0c e59f2000 ....B........ ..
phys 0x00000030 e59ff000 00000100 00008000 00000000 ................
phys 0x00000040 00000000 00000000 00000000 00000000 ................
0x00000050-0x0000006f zero suppressed
phys 0x00000070 00000000 00000000 00000000 00000000 ................
kdb>
<< The content is the same, proving the 3:1 GB virtual-address split feature; 
(virtual addr) va 0xc0000000 == (maps to) phy addr 0x0 >>

 kdb> go

brd: module loaded
loop: module loaded
--snip--
vchiq: vchiq_init_state: slot_zero = 0xdc804000, is_master = 0
usbcore: registered new interface driver smsc95xx
usbcore: registered new interface driver cdc_ncm
--snip--

Scenario II  |  Using KDB later (while apps are running)


Okay, to start with, get rid of the ‘kgdbwait’ kernel command-line parameter (typically accessible via /boot/cmdline.txt) and reboot your Pi. Get to the shell prompt (am using the Raspbian user-land here but with my own custom kernel):

--snip-- 
Debian GNU/Linux 7.0 raspberrypi ttyAMA0 
raspberrypi login: pi
Password: 
Last login: Wed Jul 3 19:17:40 UTC 2013 on ttyAMA0 
Linux raspberrypi 3.6.11+ #13 PREEMPT Thu Jul 4 13:02:28 IST 2013 armv6l 
The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. 

pi@raspberrypi:~$ 

How do we “break into” KDB from here?

<< Approach 1: Easy: get a root shell and enter KDB interactively via the kernel
‘Magic SysRq’ feature (we assume it’s enabled) by writing ‘g’ into /proc/sysrq-trigger >>

pi@raspberrypi:~$ sudo /bin/sh
# echo g > /proc/sysrq-trigger
SysRq : DEBUG
Entering kdb (current=0xdbbaa900, pid 1941) due to Keyboard Entry
kdb>

<< Approach 2: Using the ‘screen’ terminal emulator ‘hotkey’ approach:

To “send” the hotkey (on the PC we use the Alt-SysRq- combo), we make use of the terminal emulator’s “Send Break key” feature:

on minicom (not recommended!)  : ^A f g
on screen (recommended!)            : ^A b g

(^A => press Ctrl and keeping it pressed, press the ‘a’ key).
>>

So, while in the ‘screen’ terminal emulator app, I do

^A b g
…and immediately see…

RPi $ SysRq : DEBUG
Entering kdb (current=0xc054ca80, pid 0) due to Keyboard Entry
kdb>

Voila, we’re in!
As user-land is fully functional, KDB’s ‘ps A’ command reveals quite a few processes (output culled for readability):

kdb> ps A
Task Addr Pid Parent [*] cpu State Thread Command
0xc054ca80 0 0 1 0 R 0xc054cc64 *swapper
0xdb81abc0 1 0 0 0 S 0xdb81ada4 init
0xdb81a8e0 2 0 0 0 M 0xdb81aac4 kthreadd
0xdb81a600 3 2 0 0 M 0xdb81a7e4 ksoftirqd/0
0xdb81a320 4 2 0 0 M 0xdb81a504 kworker/0:0
0xdb81a040 5 2 0 0 M 0xdb81a224 kworker/0:0H
--snip--
0xdbb42180 1874 1 0 0 S 0xdbb42364 getty
0xda437980 1875 1 0 0 S 0xda437b64 getty
0xdbbdbbe0 1876 1 0 0 S 0xdbbdbdc4 login
0xda58e100 1879 1876 0 0 S 0xda58e2e4 bash
kdb> 
kdb> schedule
schedule = 0xc03cea9c (schedule)
kdb>

Scenario III  |  Using KGDB at early Kernel Boot-time

To try this, setup the R Pi kernel to boot with the ‘kgdbwait‘ kernel parameter. How? Edit the ‘cmdline.txt’ file in the boot partition of the R Pi SD card (usually mounted via the device file /dev/mmcblk0p1 on a Linux box). Here’s the kernel command-line am currently using :
# cat /boot/cmdline.txt

 dwc_otg.lpm_enable=0 dwc_otg.speed=1 debug console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 kgdbwait console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait #

[Want more details? See Documentation/kernel-parameters.txt . Also, the “debug” param is not strictly required..]

Boot the R Pi!

Uncompressing Linux... done, booting the kernel.
*** start_kernel:487 :      << >
Booting Linux on physical CPU 0
*** start_kernel:496 : 
Initializing cgroup subsys cpu
*** start_kernel:507 : 
Linux version 3.6.11+ (kaiwan@kaiwan-ThinkPad-X220) (gcc version 4.7.3 (Buildroot 2013.05) ) #13 PREEMPT Thu Jul 4 13:02:28 IST 2013
CPU: ARMv6-compatible processor [410fb767] revision 7 (ARMv7), cr=00c5387d
CPU: PIPT / VIPT nonaliasing data cache, VIPT nonaliasing instruction cache
Machine: BCM2708
--snip--
Console: switching to colour frame buffer device 82x26
kgdb: Registered I/O driver kgdboc.
kgdb: Waiting for connection from remote gdb...

<< KDB is entered automatically, as soon as the kgdb I/O modules are setup; this is
due to the ‘kgdbwait’ kernel parameter! >>

Entering kdb (current=0xdb81abc0, pid 1) due to Keyboard Entry
kdb>
kdb> kgdb
Entering please attach debugger or use $D#44+ or $3#33
<< ... waits here ... >>

<< ON THE HOST:
From another terminal window on the host system, navigate to the 
   folder where the (cross-compiled) kernel source tree for the 
   R Pi lives and run the cross-compile gdb passing along the kernel 
   debug image file 'vmlinux' as a parameter >>

# cd 
# head -n5 Makefile 
VERSION = 3
PATCHLEVEL = 6
SUBLEVEL = 11
EXTRAVERSION = 
NAME = Terrified Chipmunk
#

<< Am using kernel ver 3.6.11 here.. >>

# ls -lh ./vmlinux
-rwxrwxr-x 1 kaiwan kaiwan 64M Jul 4 13:02 ./vmlinux*
# 

<< Make sure to run the cross-compile gdb passing along the 
kernel debug image file 'vmlinux' as a parameter >>

# arm-buildroot-linux-uclibcgnueabi-gdb ./vmlinux
GNU gdb (GDB) 7.5.1
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=i686-pc-linux-gnu --target=arm-buildroot-linux-uclibcgnueabi".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /linux-3.6.11-rpi/vmlinux...done.
(gdb) set remotebaud 115200
(gdb) target remote /dev/ttyUSB0
Remote debugging using /dev/ttyUSB0
kgdb_breakpoint () at kernel/debug/debug_core.c:986
986 arch_kgdb_breakpoint();
(gdb) bt                             << lets look up the stack >>
#0 kgdb_breakpoint () at kernel/debug/debug_core.c:986
#1 0xc006d484 in kgdb_initial_breakpoint () at kernel/debug/debug_core.c:885
#2 kgdb_register_io_module (new_dbg_io_ops=new_dbg_io_ops@entry=0xc056c224 ) at kernel/debug/debug_core.c:927
#3 0xc0266c5c in configure_kgdboc () at drivers/tty/serial/kgdboc.c:197
#4 0xc0522094 in init_kgdboc () at drivers/tty/serial/kgdboc.c:219
#5 0xc000869c in do_one_initcall (fn=0xc0522074 ) at init/main.c:720
#6 0xc0510a88 in do_initcall_level (level=) at init/main.c:791
#7 do_initcalls () at init/main.c:799
#8 do_basic_setup () at init/main.c:818
#9 kernel_init (unused=) at init/main.c:917
#10 0xc0023684 in spin_unlock (lock=) at include/linux/spinlock.h:325
#11 task_unlock (p=) at include/linux/sched.h:2420
#12 exit_files (tsk=0xc051094c , tsk@entry=) at kernel/exit.c:557

<< TIP:
A quick edit: before going further, I'd like to point this out: 
I was having a lot of issues getting KGDB to work- it would appear to hang
and I'd get no I/O on the terminal emulator. After a lot of trying & reading up 
email lists, I realized that the issue is with the 'minicom' terminal emulator 
program! Man. Once I switched to using 'screen', the issues disappeared; it 
worked just fine! Thought I'd point this out for the benefit of those who might 
face similar issues...
>>

<< Setup breakpoints as required >>

(gdb) b panic
Breakpoint 1 at 0xc03c9958: file kernel/panic.c, line 70.
(gdb) b do_sync_
do_sync_read do_sync_readv_writev do_sync_work do_sync_write 
(gdb) b sys_sync
Breakpoint 2 at 0xc0100fe8: file fs/sync.c, line 103.
(gdb) b vfs_read
Breakpoint 3 at 0xc00d4004: file fs/read_write.c, line 367.
(gdb) info breakpoints 
Num Type Disp Enb Address What
1 breakpoint keep y 0xc03c9958 in panic at kernel/panic.c:70
2 breakpoint keep y 0xc0100fe8 in sys_sync at fs/sync.c:103
3 breakpoint keep y 0xc00d4004 in vfs_read at fs/read_write.c:367 
(gdb) c
Continuing.
<< gdb now idles until a breakpoint is hit; here it happens to be 
the 'vfs_read()' function; once hit, gdb is back in play... >>

Breakpoint 3, vfs_read (file=0xdb9ce7a0, buf=buf@entry=0xdb9b7980 "", count=count@entry=0x80, pos=pos@entry=0xdb82be88) at fs/read_write.c:367
367 {
(gdb) bt
#0 vfs_read (file=0xdb9ce7a0, buf=buf@entry=0xdb9b7980 "", count=count@entry=0x80, pos=pos@entry=0xdb82be88) at fs/read_write.c:367
#1 0xc00d9b4c in kernel_read (file=, offset=0x0, addr=addr@entry=0xdb9b7980 "", count=count@entry=0x80) at fs/exec.c:809
#2 0xc00d9c50 in prepare_binprm (bprm=bprm@entry=0xdb9b7980) at fs/exec.c:1329
#3 0xc00db288 in do_execve_common (regs=0xdb82bf08, envp=..., argv=..., filename=0xc04a20ec "/sbin/init") at fs/exec.c:1549
#4 do_execve (filename=filename@entry=0xc04a20ec "/sbin/init", __argv=__argv@entry=0xc054c7d0 , 
 __envp=__envp@entry=0xc054c858 , regs=regs@entry=0xdb82bf08) at fs/exec.c:1613
#5 0xc00118c0 in kernel_execve (filename=filename@entry=0xc04a20ec "/sbin/init", argv=argv@entry=0xc054c7d0 , 
 envp=envp@entry=0xc054c858 ) at arch/arm/kernel/sys_arm.c:92
#6 0xc03c8df4 in run_init_process (init_filename=init_filename@entry=0xc04a20ec "/sbin/init") at init/main.c:833
#7 0xc03c8ee4 in init_post () at init/main.c:872
#8 0xc0510b40 in kernel_init (unused=) at init/main.c:947
#9 0xc0023684 in spin_unlock (lock=) at include/linux/spinlock.h:325
#10 task_unlock (p=) at include/linux/sched.h:2420
#11 exit_files (tsk=0xc051094c , tsk@entry=) at kernel/exit.c:557
(gdb) l
362 }
363 
364 EXPORT_SYMBOL(do_sync_read);
365 
366 ssize_t vfs_read(struct file *file, char __user *buf, size_t count, loff_t *pos)
367 {
368 ssize_t ret;
369 
370 if (!(file->f_mode & FMODE_READ))
371 return -EBADF;
(gdb) [Enter]
372 if (!file->f_op || (!file->f_op->read && !file->f_op->aio_read))
373 return -EINVAL;
374 if (unlikely(!access_ok(VERIFY_WRITE, buf, count)))
375 return -EFAULT;
376 
377 ret = rw_verify_area(READ, file, pos, count);
378 if (ret >= 0) {
379 count = ret;
380 if (file->f_op->read)
381 ret = file->f_op->read(file, buf, count, pos);
(gdb) n    << n = next => step over >>
370 if (!(file->f_mode & FMODE_READ))
(gdb) p file
$1 = (struct file *) 0xdb9ce7a0
(gdb) p *file   << lets look up the 'file' structure - the Linux equivalent to the 
                  classic UNIX OFDT (Open File Descriptor Table) >>
$22 = {f_u = {fu_list = {next = 0xdb9d0070, prev = 0xdb9d0070}, fu_rcuhead = {next = 0xdb9d0070, func = 0xdb9d0070}}, f_path = {mnt = 0xdb8111d0, 
 dentry = 0xdb495778}, f_op = 0xc03e6ac0 , f_lock = {{rlock = {raw_lock = {}}}}, f_count = {counter = 0x1}, 
 f_flags = 0x20020, f_mode = 0x1d, f_pos = 0x0, f_owner = {lock = {raw_lock = {}}, pid = 0x0, pid_type = PIDTYPE_PID, uid = 0x0, 
 euid = 0x0, signum = 0x0}, f_cred = 0xdb81c820, f_ra = {start = 0x0, size = 0x0, async_size = 0x0, ra_pages = 0x20, mmap_miss = 0x0, 
 prev_pos = 0xffffffffffffffff}, f_version = 0x0, private_data = 0x0, f_ep_links = {next = 0xdb9ce80c, prev = 0xdb9ce80c}, f_tfile_llink = {
 next = 0xdb9ce814, prev = 0xdb9ce814}, f_mapping = 0xdb4993a4}
(gdb) set print pretty     << make it look pretty >>
(gdb) p *file
$23 = {
 f_u = {
 fu_list = {
 next = 0xdb9d0070, 
 prev = 0xdb9d0070
 }, 
 fu_rcuhead = {
 next = 0xdb9d0070, 
 func = 0xdb9d0070
 }
 }, 
 f_path = {
 mnt = 0xdb8111d0, 
 dentry = 0xdb495778
 }, 
 f_op = 0xc03e6ac0 , 
 f_lock = {
 {
 rlock = {
 raw_lock = {}
 }
 }
 }, 
 f_count = {
 counter = 0x1
 }, 
 f_flags = 0x20020, 
 f_mode = 0x1d, 
 f_pos = 0x0, 
 f_owner = {
 lock = {
 raw_lock = {}
 }, 
 pid = 0x0, 
 pid_type = PIDTYPE_PID, 
---Type  to continue, or q  to quit---q
Quit

<< Lets lookup the stack frame we're currently at, 
ARM CPU register values & disable the 'vfs_read' breakpoint >>

(gdb) info frame 
Stack level 0, frame at 0xdb82be88:
 pc = 0xc00d4020 in vfs_read (fs/read_write.c:370); saved pc 0xc00d9b4c
 called by frame at 0xdb82bea8
 source language c.
 Arglist at 0xdb82be84, args: file=0xdb9ce7a0, buf=buf@entry=0xdb9b7980 "", count=count@entry=0x80, pos=pos@entry=0xdb82be88
 Locals at 0xdb82be84, Previous frame's sp at 0xdb82be7c
 Saved registers:
 r4 at 0xdb82be64, r5 at 0xdb82be68, r6 at 0xdb82be6c, r7 at 0xdb82be70, r8 at 0xdb82be74, r11 at 0xdb82be78, lr at 0xdb82be80
(gdb) info registers 
r0 0xdb9ce7a0 0xdb9ce7a0
r1 0xdb9b7980 0xdb9b7980
r2 0x80 0x80
r3 0xdb82be88 0xdb82be88
r4 0x0 0x0
r5 0xc04a20ec 0xc04a20ec
r6 0xdb9b7980 0xdb9b7980
r7 0xc054c858 0xc054c858
r8 0xdb82bf08 0xdb82bf08
r9 0x0 0x0
r10 0xdb9b7980 0xdb9b7980
r11 0xdb82be84 0xdb82be84
r12 0xdb82be88 0xdb82be88
sp 0xdb82be58 0xdb82be58
lr 0xc00d9b4c 0xc00d9b4c
pc 0xc00d4020 0xc00d4020 <vfs_read+28>
cpsr 0x60000113 0x60000113
(gdb) info breakpoints 
Num Type Disp Enb Address What
1 breakpoint keep y 0xc03c9958 in panic at kernel/panic.c:70
2 breakpoint keep y 0xc0100fe8 in sys_sync at fs/sync.c:103
3 breakpoint keep y 0xc00d4004 in vfs_read at fs/read_write.c:367
 breakpoint already hit 1 time
(gdb) disable 3
(gdb) info breakpoints 
Num Type Disp Enb Address What
1 breakpoint keep y 0xc03c9958 in panic at kernel/panic.c:70
2 breakpoint keep y 0xc0100fe8 in sys_sync at fs/sync.c:103
3 breakpoint keep n 0xc00d4004 in vfs_read at fs/read_write.c:367
 breakpoint already hit 1 time
(gdb)

<< Useful! Can run a KDB command from within gdb using the 'monitor' prefix >>

(gdb) monitor ps
28 sleeping system daemon (state M) processes suppressed,
use 'ps A' to see all.
Task Addr Pid Parent [*] cpu State Thread Command
0xdb81abc0 1 0 1 0 R 0xdb81ada4 *swapper
0xdb81abc0 1 0 1 0 R 0xdb81ada4 *swapper
0xdb8a5920 12 2 0 0 R 0xdb8a5b04 khubd
(gdb) monitor 0xc00d4020
0xc00d4020 = 0xc00d4020 (vfs_read+0x1c)
(gdb) monitor go    << tell the R Pi to continue ...                        (or we could just use gdb's 'continue' cmd) >>
go must execute on the entry cpu, please use "cpu -1" and then execute go
(gdb)    << In spite of the above message, it seems              to work, continuing execution ... >>

Scenario IV  |  Using KGDB later (while apps are running)

(The details are very similar to the section “Scenario II | Using KDB later (while apps are running)” above).

To start with, get rid of the ‘kgdbwait’ kernel command-line parameter (typically accessible via /boot/cmdline.txt) and reboot your Pi. Get to the shell prompt (am using the Raspbian user-land here but with my own custom kernel):

--snip-- 
Debian GNU/Linux 7.0 raspberrypi ttyAMA0 
raspberrypi login: pi
Password: 
Last login: Wed Jul 3 19:17:40 UTC 2013 on ttyAMA0 
Linux raspberrypi 3.6.11+ #13 PREEMPT Thu Jul 4 13:02:28 IST 2013 armv6l 
The programs included with the Debian GNU/Linux system are free software; the exact distribution terms for each program are described in the individual files in /usr/share/doc/*/copyright. Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. 

pi@raspberrypi:~$ 

<< Just fyi, lets look-up, using the powerful proc filesystem, the kernel version, 
kernel command-line and processor info >>
pi@raspberrypi:~$ cat /proc/version 
Linux version 3.6.11+ (kaiwan@kaiwan-ThinkPad-X220) (gcc version 4.7.3 (Buildroot 2013.05) ) #13 PREEMPT Thu Jul 4 13:02:28 IST 2013

pi@raspberrypi:~$ cat /proc/cmdline 
dma.dmachans=0x7f35 bcm2708_fb.fbwidth=656 bcm2708_fb.fbheight=416 bcm2708.boardrev=0xf bcm2708.serial=0x7986e1c5 smsc95xx.macaddr=B8:27:EB:86:E1:C5 sdhci-bcm2708.emmc_clock_freq=100000000 vc_mem.mem_base=0x1ec00000 vc_mem.mem_size=0x20000000 dwc_otg.lpm_enable=0 dwc_otg.speed=1 debug console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 elevator=deadline rootwait

pi@raspberrypi:~$ cat /proc/cpuinfo 
Processor : ARMv6-compatible processor rev 7 (v6l)
BogoMIPS : 697.95
Features : swp half thumb fastmult vfp edsp java tls 
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x0
CPU part : 0xb76
CPU revision : 7
Hardware : BCM2708
Revision : 000f
Serial : 000000007986e1c5
pi@raspberrypi:~$

To enter KGDB now (i.e. while user-land applications are alive and kicking), it’s really a very similar procedure to the “Scenario II” above: first enter KDB using the Magic-SysRq method, and then from within KDB invoke KGDB using the ‘kgdb’ kdb command! That’s it. Everything else remains the same…

<< Approach 1: Easy: get a root shell and enter KDB interactively via
Magic SysRq by writing ‘g’ into /proc/sysrq-trigger >>

pi@raspberrypi:~$ sudo /bin/sh
# echo g > /proc/sysrq-trigger
SysRq : DEBUG
Entering kdb (current=0xdbbaa900, pid 1941) due to Keyboard Entry
kdb>

<< Approach 2: Using the ‘screen’ terminal emulator: On the keyboard type

^A b g
>>

SysRq : DEBUG
Entering kdb (current=0xc054ca80, pid 0) due to Keyboard Entry
kdb> kgdb
Entering please attach debugger or use $D#44+ or $3#33

The rest of the discussion is the same as “Scenario III | Using KGDB …” above. HTH, Enjoy!

Please do leave your comments; and ‘Like’ my post (if you like it)! 🙂

Advertisements

6 thoughts on “A KDB / KGDB session on the popular Raspberry Pi embedded Linux board”

  1. is it possible to kernel debug with a usb-to-serial-converter attached to the rpi and the host machine connected with a null modem cable?

      1. OK I will try with “screen”. Nevertheless I did a test with ttyAMA0 only for kgdboc / KGDB and tty1 (console on graphic screen).
        thx for your answer 🙂

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s