Category Archives: programming

Setting up Kdump and Crash for ARM-32 – an Ongoing Saga

Author: Kaiwan N Billimoria, kaiwanTECH
Date: 13 July 2017

DUT (Device Under Test):
Hardware platform: Qemu-virtualized Versatile Express Cortex-A9.
Software platform: mainline linux kernel ver 4.9.1, kexec-tools, crash utility.

First, my attempt at setting up the Raspberry Pi 3 failed; mostly due to recurring issues with the bloody MMC card; probably a power issue! (see this link).

Anyway. Then switched to doing the same on the always-reliable Qemu virtualizer; I prefer to setup the Vexpress-CA9.

In fact, a supporting project I maintain on github – the SEALS project – is proving extremely useful for building the ARM-32 hardware/software platform quickly and efficiently. (Fun fact: SEALS = Simple Embedded Arm Linux System).

So, I cloned the above-mentioned git repo for SEALS into a new working folder.

The way SEALS work is simple: edit a configuration file (build.config) to your satisfaction, to reflect the PATH to and versions of the cross-compiler, kernel, kernel command-line parameters, busybox, rootfs size, etc.

Setup the SEALS build.config file.

Screenshot: the build_SEALS.sh script initial screen displaying the current build config:kdumpcr1

<<
Relevant Info reproduced below for clarity:

Toolchain prefix : arm-none-linux-gnueabi-
Toolchain version: (Sourcery CodeBench Lite 2014.05-29) 4.8.3 20140320 (prerelease)

Staging folder : <…>/SEALS_staging
ARM Platform : Versatile Express (A9)

Platform RAM : 512 MB
RootFS force rebuild : 0
RootFS size : 768 MB

Linux kernel to use : 4.9.1
Linux kernel codebase location : <…>/SEALS_staging/linux-4.9.1
Kernel command-line : “console=ttyAMA0 root=/dev/mmcblk0 init=/sbin/init crashkernel=32M”

Busybox to use : 1.26.2
Busybox codebase location : <…>/SEALS_staging/busybox-1.26.2

>>

Screenshot: build_SEALS.sh second GUI screen, allowing the user to select actions to takekdumpcr2

Upon clicking ‘OK’, the build process starts:

I Boot Kernel Setup

  • kernel config: must carefully configure the Linux kernel. Please follow the kernel documentation in detail:
    https://www.kernel.org/doc/Documentation/kdump/kdump.txt [1]In brief, ensure these are set:
    CONFIG_KEXEC=y
    CONFIG_SYSFS=y << should be >>
    CONFIG_DEBUG_INFO=y
    CONFIG_CRASH_DUMP=y
    CONFIG_PROC_VMCORE=y

Dump-capture kernel config options (Arch Dependent, arm)
To use a relocatable kernel, Enable “AUTO_ZRELADDR” support under “Boot” options:      

             AUTO_ZRELADDR=y”

  • kexec
    • We require to build kexec (kexec-tools is the package). But: the package does not seem to be directly available for A-32 (ARM-32), so had to build from source.
    • Did not succeed
    • then saw this gist:

https://gist.github.com/Gnurou/7191098

which succinctly got it working!

  • Copy the ‘kexec’ binary into the root filesystem (staging tree) under it’s sbin/ folder
  • We build a relocatable kernel so that we can use the same ‘zImage’ 
    for the dump kernel as well as the primary boot kernel:
     “Or use the system kernel binary itself as dump-capture kernel and there is 
    no need to build a separate dump-capture kernel. 
    This is possible  only with the architectures which support a 
    relocatable kernel. As  of today, i386, x86_64, ppc64, ia64 and 
    arm architectures support relocatable kernel. ...”
    
  • the build system will proceed to build the kernel using the cross-compiler specified
  • went through just fine.

II Load dump-capture (or kdump) kernel into boot kernel’s RAM

Do read [1], but to cut a long story short

  • Create a small shell script kx.sh - a wrapper over kexec – in the root filesystem:
     
     #!/bin/sh
    DUMPK_CMDLINE="console=ttyAMA0 root=/dev/mmcblk0 rootfstype=ext4 rootwait init=/sbin/init maxcpus=1 reset_devices"
    kexec --type zImage \
    -p ./zImage-4.9.1-crk \
    --dtb=./vexpress-v2p-ca9.dtb \
    --append="${DUMPK_CMDLINE}" 
    [ $? -ne 0 ] && { 
        echo "kexec failed." ; exit 1
    }
    echo "$0: kexec: success, dump kernel loaded."
    exit 0
    
  • Run it. It will only work (in my experience) when:
    • you’ve passed the kernel parameter ‘crashkernel=32M’
    • verified that indeed the boot kernel has reserved 32MB RAM for the dump-capture kernel/system:
RUN: Running qemu-system-arm now ...

qemu-system-arm -m 512 -M vexpress-a9 -kernel <...>/images/zImage \
-drive file=<...>/images/rfs.img,if=sd,format=raw \
-append "console=ttyAMA0 root=/dev/mmcblk0 init=/sbin/init crashkernel=32M" \
-nographic -no-reboot -dtb <...>/linux-4.9.1/arch/arm/boot/dts/vexpress-v2p-ca9.dtb

Booting Linux on physical CPU 0x0
Linux version 4.9.1-crk (hk@hk) (gcc version 4.8.3 20140320 (prerelease) (Sourcery CodeBench Lite 2014.05-29) ) #2 SMP Wed Jul 12 19:41:08 IST 2017
CPU: ARMv7 Processor [410fc090] revision 0 (ARMv7), cr=10c5387d
CPU: PIPT / VIPT nonaliasing data cache, VIPT nonaliasing instruction cache
OF: fdt:Machine model: V2P-CA9
...
ARM / $ dmesg |grep -i crash
Reserving 32MB of memory at 1920MB for crashkernel (System RAM: 512MB)
Kernel command line: console=ttyAMA0 root=/dev/mmcblk0 init=/sbin/init crashkernel=32M
ARM / $ id
uid=0 gid=0
ARM / $ ./kx.sh
./kx.sh: kexec: success, dump kernel loaded.
ARM / $ 

Ok, the dump-capture kernel has loaded up.
Now to test it!

III Test the soft boot into the dump-capture kernel

On the console of the (emulated) ARM-32:

ARM / $ echo c > /proc/sysrq-trigger 
sysrq: SysRq : Trigger a crash
Unhandled fault: page domain fault (0x81b) at 0x00000000
pgd = 9ee44000
[00000000] *pgd=7ee30831, *pte=00000000, *ppte=00000000
Internal error: : 81b [#1] SMP ARM
Modules linked in:
CPU: 0 PID: 724 Comm: sh Not tainted 4.9.1-crk #2
Hardware name: ARM-Versatile Express
task: 9f589600 task.stack: 9ee40000
PC is at sysrq_handle_crash+0x24/0x2c
LR is at arm_heavy_mb+0x1c/0x38
pc : [<804060d8>] lr : [<80114bd8>] psr: 60000013
sp : 9ee41eb8 ip : 00000000 fp : 00000000

...

[<804060d8>] (sysrq_handle_crash) from [<804065bc>] (__handle_sysrq+0xa8/0x170)
[<804065bc>] (__handle_sysrq) from [<80406ab8>] (write_sysrq_trigger+0x54/0x64)
[<80406ab8>] (write_sysrq_trigger) from [<80278588>] (proc_reg_write+0x58/0x90)
[<80278588>] (proc_reg_write) from [<802235c4>] (__vfs_write+0x28/0x10c)
[<802235c4>] (__vfs_write) from [<80224098>] (vfs_write+0xb4/0x15c)
[<80224098>] (vfs_write) from [<80224d30>] (SyS_write+0x40/0x80)
[<80224d30>] (SyS_write) from [<801074a0>] (ret_fast_syscall+0x0/0x3c)

Code: f57ff04e ebf43aba e3a03000 e3a02001 (e5c32000) 

Loading crashdump kernel...
Bye!
Booting Linux on physical CPU 0x0

Linux version 4.9.1-crk (hk@hk) (gcc version 4.8.3 20140320 (prerelease) (Sourcery CodeBench Lite 2014.05-29) ) #2 SMP Wed Jul 12 19:41:08 IST 2017
CPU: ARMv7 Processor [410fc090] revision 0 (ARMv7), cr=10c5387d
CPU: PIPT / VIPT nonaliasing data cache, VIPT nonaliasing instruction cache
OF: fdt:Machine model: V2P-CA9
OF: fdt:Ignoring memory range 0x60000000 - 0x78000000
Memory policy: Data cache writeback
CPU: All CPU(s) started in SVC mode.
percpu: Embedded 14 pages/cpu @81e76000 s27648 r8192 d21504 u57344
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 7874
Kernel command line: console=ttyAMA0 root=/dev/mmcblk0 rootfstype=ext4 rootwait 
init=/sbin/init maxcpus=1 reset_devices elfcorehdr=0x79f00000 mem=31744K

...
ARM / $ ls -l /proc/vmcore            << the dump image (480 MB here) >>
-r-------- 1 0 0 503324672 Jul 13 12:22 /proc/vmcore
ARM / $ 

Copy the dump file (with cp or scp, whatever), 
get it to the host system.

cp /proc/vmcore <dump-file>
ARM / $ halt
ARM / $ EXT4-fs (mmcblk0): re-mounted. Opts: (null)
The system is going down NOW!
Sent SIGTERM to all processes
Sent SIGKILL to all processes
Requesting system halt
reboot: System halted
QEMU: Terminated
^A-X  << type Ctrl-a followed by x to exit qemu >>
... and done.

build_SEALS.sh: all done, exiting.
Thank you for using SEALS! We hope you like it.
There is much scope for improvement of course; would love to hear your feedback, ideas, and contribution!
Please visit : https://github.com/kaiwan/seals . 


IV Analyse the kdump image with the crash utility

CORE ANALYSIS SUITE

The core analysis suite is a self-contained tool that can be used to
investigate either live systems, kernel core dumps created from dump
creation facilities such as kdump, kvmdump, xendump, the netdump and
diskdump packages offered by Red Hat, the LKCD kernel patch, the mcore
kernel patch created by Mission Critical Linux, as well as other formats
created by manufacturer-specific firmware.

...

A whitepaper with complete documentation concerning the use of this utility
can be found here:
http://people.redhat.com/anderson/crash_whitepaper [3]
...

The crash binary can only be used on systems of the same architecture as
the host build system. There are a few optional manners of building the
crash binary:

o On an x86_64 host, a 32-bit x86 binary that can be used to analyze
32-bit x86 dumpfiles may be built by typing "make target=X86".
o On an x86 or x86_64 host, a 32-bit x86 binary that can be used to analyze
 32-bit arm dumpfiles may be built by typing "make target=ARM".
...

Ah. To paraphrase, Therein lies the devil, in the details.

[UPDATE : 14 July ’17
I do have it building successfully now. The trick apparently – on x86_64 Ubuntu 17.04 – was to install the 
lib32z1-dev package! Once I did, it built just fine. Many thanks to Dave Anderson (RedHat) who promptly replied to my query on the crash mailing list.]

I cloned the ‘crash’ git repo, did ‘make target=ARM’, it fails with:

...
 ../readline/libreadline.a ../opcodes/libopcodes.a ../bfd/libbfd.a
../libiberty/libiberty.a ../libdecnumber/libdecnumber.a -ldl
-lncurses -lm ../libiberty/libiberty.a build-gnulib/import/libgnu.a
 -lz -ldl -rdynamic
/usr/bin/ld: cannot find -lz
collect2: error: ld returned 1 exit status
Makefile:1174: recipe for target 'gdb' failed
...

Still trying to debug this!

Btw, if you’re unsure, pl see crash’s github Readme on how to build it.
So, now, with a ‘crash’ binary that works, lets get to work:

$ file crash
crash: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.6.32, …

$ ./crash

crash 7.1.9++
Copyright (C) 2002-2017 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
[…]

crash: compiled for the ARM architecture
$

To examine a kernel dump (kdump) file, invoke crash like so:

crash <path-to-vmlinux-with-debug-symbols> <path-to-kernel-dumpfile>

$ <...>/crash/crash \
  <...>/SEALS_staging/linux-4.9.1/vmlinux ./kdump.img

crash 7.1.9++
Copyright (C) 2002-2017 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
[...]
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
[...]
WARNING: cannot find NT_PRSTATUS note for cpu: 1
WARNING: cannot find NT_PRSTATUS note for cpu: 2
WARNING: cannot find NT_PRSTATUS note for cpu: 3

 KERNEL: <...>/SEALS_staging/linux-4.9.1/vmlinux
 DUMPFILE: ./kdump.img
 CPUS: 4 [OFFLINE: 3]
 DATE: Thu Jul 13 00:38:39 2017
 UPTIME: 00:00:42
LOAD AVERAGE: 0.00, 0.00, 0.00
 TASKS: 56
 NODENAME: (none)
 RELEASE: 4.9.1-crk
 VERSION: #2 SMP Wed Jul 12 19:41:08 IST 2017
 MACHINE: armv7l (unknown Mhz)
 MEMORY: 512 MB
 PANIC: "sysrq: SysRq : Trigger a crash"
 PID: 735
 COMMAND: "echo"
 TASK: 9f6af900 [THREAD_INFO: 9ee48000]
 CPU: 0
 STATE: TASK_RUNNING (SYSRQ)

crash> ps
 PID PPID CPU TASK ST %MEM VSZ RSS COMM
 0 0 0 80a05c00 RU 0.0 0 0 [swapper/0]
> 0 0 1 9f4ab700 RU 0.0 0 0 [swapper/1]
> 0 0 2 9f4abc80 RU 0.0 0 0 [swapper/2]
> 0 0 3 9f4ac200 RU 0.0 0 0 [swapper/3]
 1 0 0 9f4a8000 IN 0.1 3344 1500 init
[...]
722 2 0 9f6ac200 IN 0.0 0 0 [ext4-rsv-conver]
728 1 0 9f6ab180 IN 0.1 3348 1672 sh
> 735 728 0 9f6af900 RU 0.1 3344 1080 echo
crash> bt
PID: 735 TASK: 9f6af900 CPU: 0 COMMAND: "echo"
 #0 [<804060d8>] (sysrq_handle_crash) from [<804065bc>]
 #1 [<804065bc>] (__handle_sysrq) from [<80406ab8>]
 #2 [<80406ab8>] (write_sysrq_trigger) from [<80278588>]
 #3 [<80278588>] (proc_reg_write) from [<802235c4>]
 #4 [<802235c4>] (__vfs_write) from [<80224098>]
 #5 [<80224098>] (vfs_write) from [<80224d30>]
 #6 [<80224d30>] (sys_write) from [<801074a0>]
 pc : [<76e8d7ec>] lr : [<0000f9dc>] psr: 60000010
 sp : 7ebdcc7c ip : 00000000 fp : 00000000
 r10: 0010286c r9 : 7ebdce68 r8 : 00000020
 r7 : 00000004 r6 : 00103008 r5 : 00000001 r4 : 00102e2c
 r3 : 00000000 r2 : 00000002 r1 : 00103008 r0 : 00000001
 Flags: nZCv IRQs on FIQs on Mode USER_32 ISA ARM
crash>

And so on …

Another thing we can do is use gdb – to a limited extent – to analyse the dump file:

From [1]:

Before analyzing the dump image, you should reboot into a stable kernel.

You can do limited analysis using GDB on the dump file copied out of
/proc/vmcore. Use the debug vmlinux built with -g and run the following
command:
  gdb vmlinux <dump-file>

Stack trace for the task on processor 0, register display, and memory
display work fine.

Also, [3] is an excellent whitepaper on using crash. Do read it.

All right, hope that helps!

Advertisements

Low-Level Software Design

[Please note, this article isn’t about formal design methods (LLD), UML, Design Patterns, nor about object-oriented design, etc. It’s written with a view towards the kind of software project I typically get to work on – embedded / Linux OS related, with the primary programming language being ‘C’ and/or scripting (typically with bash).]

When one looks back, all said and done, it isn’t that hard to get a decent software design and architecture. Obviously, the larger your project, the more the thought and analysis that goes into building a robust system. (Certainly, the more the years of experience, the easier it seems).

However, I am of the view that certain fundamentals never change: get them right and many of the pieces auto-slot into place. Work on a project enough and one always comes away with a  “feel” for the architecture and codebase – it’s robust, will work, or it’s just not.

So what are these “fundamentals”? Well, here’s the interesting thing: you already know them! But in the heat and dust of release pressures (“I don’t care that you need another half-day, check it in now!!!”), deadlines, production, we tend to forget the basics. Sounds familiar? 🙂

The points below are definitely nothing new, but always worth reiterating:

Low-level Design and Software Architecture

  • Jot down the requirements: why are we doing this? what do we hope to achieve?
  • Draw an overall diagram of the project, the data structures, the code flow, as you visualise it. You don’t really need fancy software tools- pencil and paper will do, especially at first.
    pencil20on20notebook20-20writing20concept
    Arrive, gently, at the software architecture.
  • Layering helps (but one can overdo it)
    • To paraphrase- “adding a layer can be used to solve any problem in computer science” 🙂 Of course, one can quite easily add new problems too; careful!
  • It evolves – don’t be afraid to iterate, to use trial and error
    • “Be ready to throw the first one away – you’re going to anyway” – paraphrased from that classic book “The Mythical Man Month”
    • “There is no silver bullet” – again from the same book of wisdom. There is no one solution to all your problems – you’ll have to weigh options, make trade-offs. It’s like life y’know 😉
  • Design the code to be modular, structured
  • A function encapsulates an intention
    • Requirement-driven code: why is the function there?
  • Each function does exactly one thing
    • This is really important. If you can do this well, you will greatly reduce bugs, and thus, the need to debug.
  • Use configuration files (Edit: preferably in plain ASCII text format).

Coding

  • Insert function stubs – code it in detail later, get the overall low-level design and function interfacing correct first. What parameters, return value(s)? 
  • Avoid globals
    • use parameters, return values
    • in multithreaded / multiprocess environments, using any kind of global implies using a synchronization primitive of some sort (mutex, semaphore, spinlock, etc) to take care of concurrency concerns, races. Be aware – beware! – this is often a huge source of performance bottlenecks!
      Edit: When writing MT software, use powerful techniques TLS and TSD to further avoid globals.
  • Keep it minimal, and clean: Careful! don’t end up using too many (nested functions) layers – leads to “lasagna / spaghetti code” that’s hard to follow and thus understand
  • If a function’s code exceeds a ‘page’, re-look, redesign.

Of course, a project is not a dead static thing – at least it shouldn’t be. It evolves over time. Expect requirements, and thus your low-level design and code, to change. The better thought out the overall architecture though, the more resilient it will be to constant flux.

For example: you’re writing a device driver and a “read” method is attempting to read data from the ‘device’ (whatever the heck it is), but there is no data available right now, what should we do? Abort, returning an error code? Wait for data? Retry the operation thrice and see?

The “correct” answer: follow the standard. Assuming we’re working on a POSIX-compliant OS (Unix/Linux), the standard says that blocking calls must do precisely that: block, wait for data until it becomes available. So just wait for data. “But I don’t want to wait forever!” cries the application! Okay, implement a non-blocking open in that case (there’s a reason for that O_NONBLOCK flag folks!). Or a timeout feature, if it makes sense.

Shouldn’t the driver method “retry” the operation if it does not succeed at first? Short answer, No. Follow the Unix design philosophy: “provide mechanism, not policy”. Let the application define the policy (should we retry, if yes, how often; should we timeout, if yes, what’s the timeout, etc etc). The mechanism part of it – your driver’s read method implementation must work independent of, in fact ignore, such concerns. (But hey, it must be written to be concurrent and reentrant -safe. A post on that another day perhaps?).

Configuration

Using the same example: lets say we do want the “read” method of our device driver to timeout after, say, 1 second. Where do we specify this value? Recollect, “provide mechanism, not policy”. So we’ll do so in the application, not the driver. But where in the app? Ah. I’d suggest we don’t hardcode the value; instead, keep it in a simple ASCII-text configuration file:

app_config
   read_timeout=1

Of course, with ‘C’ one would usually put stuff like this into a header file. Fair enough, but please keep a separate header – say, app_config.h .

Try and do some crystal-ball-gazing: at some remote (or not-so-remote) point in the future, what if the project requires a GUI front-end? Probably, as an example, we will want to let the end-user view and set configuration – change the timeout, etc – easily via the GUI. Then, you will see the sense of using a simple ASCII-text configuration file to hold config values – reading and updating the values now becomes simple and clean.
Finally, nothing said above is sacred – we learn to take things on a case-by-case basis, use judgement.


A few Resources