Why the printk can be so slow on a serial line

I got the idea for this rather silly post when I was trying to understand something regarding the Linux driver model and the DT (Device Tree). So, with Linux, a device driver (here am using a platform driver), need not hard-code the (platform) device stuff into the driver; in fact it mustn’t. That’s exactly what the DT is for. So, the driver can register itself as a platform driver and an entry is made in the DT for the platform device; then the DT source (.dts) is compiled (to .dtb) and passed along at boot (I tried all this on a Raspberry Pi 3 Model B+).

I was trying to empirically see for myself how exactly the kernel invokes our platform driver’s probe method? Lets use ftrace to find out (kudos Steven Rostedt & team)! To make it easier, we use the super trace-cmd(1) front-end to ftrace.

Get the trace-cmd source:


Build & install:

&& sudo make install

Use it: wanted to capture the work of platform_driver_register() within the insmod:

First, lets make it easier by ourselves emitting a message into the ftrace log buffer (via the trace_printk()), around the area of interest; this way, we can just search the (typically huge) report for this string!

trace_printk("@@@@@ %s: MARKER 1: platform_driver_register() begin\n", DRVNAME);
ret_val = platform_driver_register(&my_platform_driver);
trace_printk("@@@@@ %s: MARKER 2: platform_driver_register() done\n", DRVNAME);

To have ftrace trace the functions of a particular kernel module (very very useful!), use the syntax:

echo ":mod:<kernel-module-name>" >> /sys/kernel/debug/tracing/set_ftrace_filter

(Actually here I do not use this filter as otherwise we will lose the printk output).

trace-cmd record -p function_graph -F taskset 01 insmod ./dtdemo_platdrv.ko
report -I -S -l > ftrc_rep.txt

Notice how we use taskset(1) to ensure that the process we’re interested in ftrace-ing (insmod(8)) runs on exactly one CPU (cpumask set to 01).

So back to the original question: Why is the printk slow? Because it executes a hell of a lot of code, in a loop. Why? Because it emits the content of the string passed character-by-character to the serial device. We actually see this by gleaning the voluminous trace-cmd report output below; I’ve just put three screenshots of the trace-cmd report output below, else the wrapping is far too much and ugly; also, all the output shown below in the screenshots may not be perfectly contiguous (though that’s the intention):

[…] the above repeats many times!

This, below, is the close brace of the above printk() (yes, it’s wrapped around, ignore that):

insmod-12410   0....  6164.594698: funcgraph_exit:       # 11726.146 us |                              }

So, in this particular case, it took around 117.26 ms to execute! Quite a bit…
A few additional (useless) stats:

The trace-cmd(1) (ftrace wrapper front-end) snippet of only the single printk is over 6000 lines (I extracted only the printk stuff into another text file):

$ wc -l the_ftraced_printk.txt 
6546 the_ftraced_printk.txt

The function that actually writes a single character to the serial device here is serial8250_console_putchar(); lets see how many times it was called within this one printk:

$ grep "serial8250_console_putchar" the_ftraced_printk.txt |wc -l

Whew. So, when one must output a lot of printk’s and it’s a time-constrained code path (think interrupts), what must one do? Ah: ftrace provides the versatile trace_printk() function. It’s functionally equivalent (better!) to the printk, but does not write to the kernel log buffer nor to the – critical point – console device; it writes to the ftrace buffer only (which, btw, is tunable – you can make it very large unlike the printk buffer). (An aside: check out my post A HEADER OF CONVENIENCEit shows several macros. The MSG() macro emits a debug message: one can internally set the preference – to use the regular printk or the trace_printk().


Working on the Console with the Raspberry Pi

The Raspberry Pi (I’m currently using the R Pi 3 Model B) is a fantastic embedded Linux prototyping and development platform for hobbyists and tinkerers.

However, one of the thing that bugs me about it – by default, one can either hook it up to to a digital monitor or TV via HDMI (not ideal for developers, not for me at least!), or, much better, attach it to your local router via ethernet RJ45 cable and log in to it over ssh(1) from your laptop.

The Raspberry Pi 3 Model B: notice the power cable (bottom-rt) and ethernet RJ45 cable (top rt)

(My other pet grudge against  the R Pi is the completely closed bootloader; why can’t we just install and use U-Boot!?; can we?).

Another key point: for a long while I was using an inappropriate power supply and, for the lack of one, was destroying my (micro)SD cards! Using a proper “known to work” power supply for the R Pi is critical; I now use a “Elementz Engineers Guild 5V 3A USB to MicroUSB Charger for Raspberry Pi 3 Model B/RPi 2 Model B/B+/A+ (White)” power supply (purchased from Amazon India) and it works just great.

Why a Linux Console

So, while the ssh login is fine for most purposes, there are circumstances where there is no alternative but to have a Linux console device available right from boot up. Typical reasons include:

  • require access to the bootloader monitor or command line
  • kernel debug work (using KGDB or KDB, etc)
  • want  to work on kernel code within a console device (so that, for example, all printk’s appear immediately like printf’s
  • … etc

How to do it

Essentially, to get a console, we require the old-style serial port and a simple RS-232 compatible ‘crossover’ cable – Tx<–>Rx :: target board <–> laptop serial port.

Only one problem: modern laptops (and even PCs) no longer have serial ports; now that’s not an earth-shaking point, we know this. So, of course, USB to the rescue: many USB-to-serial adapter cables exist, allowing us to relive the good ‘ol days! That’s exactly what I did – purchased and setup a nice, quite cheap and quite reliable little USB-to-serial cable; here are the gory details:

1. Purchased the “Imported USB To RS232 TTL UART PL2303HX Converter USB to COM Cable Adapter Module” from Amazon, India.

Imported USB To RS232 TTL UART PL2303HX Converter USB to COM Cable Adapter Module” from Amazon, India

2. Relevant technical info on amazon for the product:

“… Description:

Built in TTL COM PC PL2303HX Chip Standard USB type A male and TTL 4 Pin connector
Cable Length: Approx. 1m
Color: Black
Function: Serial communication Router or ADSL firmware upgrade GPS serial communication Hard drive firmware upgrade super terminal in use; Commonly used serial debugging tools All kinds of satellite machine for upgrading

Black cable-----GND
Green cable-----TXD
White cable-----RXD
Red cable-------VCC
Package includes: 1x USB to TTL Module"

The “Wiring” line above is what’s important to us..

3. For the R Pi 3 , must do an additional step:
If you want to use the UART console during booting on a Pi3 then you need to add the line `enable_uart=1` to config.txt [on the SDcard boot partition]. Also, you can use the normal UART pins for other functions without pinctrl getting in the way.”

4. So, lookup the R Pi’s GPIO header (particularly for the R Pi 3 as that’s what we’re using).

Useful ref: Interfacing hardware with the Raspberry Pi

R Pi 3 : GPIO Pinout ; image source

Connect the wires as follows:    

RPi3 GPIO           USB<->RS232 TTL adapter
pin# (Func)        Color of wire (Func)
--------------     ------------------------
2 (5V Vcc)          Red wire   (VCC)
6 (GND)             Black wire (GND)
8 (UART0 TX)        White wire (RXD)
10 (UART0 RX)        Green wire (TXD)

First attach the wires to the GPIO ensuring no power is applied to the R Pi device.

The USB-to-serial adapter’s four wires in the GPIO pins 2,6,8,10 of the R PI

5. Add a Linux console:
(Backup /boot/cmdline.txt first and then)
Edit the /boot/cmdline.txt to have:

dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 console=ttyS0 root=PARTUUID=db8dd2db-02 rootfstype=ext4 elevator=deadline fsck.repair=yes rootwait 3

6. (Ref: RPi Serial Connection, eLinux)
6.1 Plug in the USB-to-serial adapter’s USB side to the PC/laptop, for power. We do this first so that the host (laptop) kernel detects the device and assigns it a device node (via udev; typically, /dev/ttyUSB0). Can check this via dmesg: (on my host):

$ dmesg
[94933.835317] usb 1-1: new full-speed USB device number 8 using xhci_hcd
[94933.984283] usb 1-1: New USB device found, idVendor=067b, idProduct=2303, bcdDevice= 3.00
[94933.984294] usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[94933.984302] usb 1-1: Product: USB-Serial Controller
[94933.984308] usb 1-1: Manufacturer: Prolific Technology Inc.
[94933.986152] pl2303 1-1:1.0: pl2303 converter detected
[94933.987003] usb 1-1: pl2303 converter now attached to ttyUSB0

6.2 On the PC/laptop, in a terminal window run a terminal emulator app – typically, screen or minicom (there are other choices as well, though am happy with minicom).

6.3 Setup the terminal emulator serial parameters correctly:
115200 8N1, Hardware and Software Flow Control Off.
For minicom: Ensure your login belongs to the ‘dialout’ group:

sudo usermod -a -G dialout username
sudo minicom -b 115200 -o -D <Port_Name>
; where Port_Name is usually /dev/ttyUSB0

If all okay, you should see the login (agetty) prompt from the Raspberry Pi (and, if you did it fast enough, the console printk’s from the kernel as it boots). Login (username: pi, default pswd: raspberry).

minicom terminal emulator, with R Pi


  • In my experience, using just the USB rail for power to the R Pi causes unstable behavior; it reboots all of a sudden. So, attaching up a good 3.0A power supply (via it’s miniUSB port) is important
  • Minicom settings menu: Ctrl+a+z.

My book – Hands-On System Programming with Linux

Hello, pleased that my first book has been recently released (on 31 Oct 2018). The publisher is Packt (based in Birmingham, UK).

Quick Links

The book is available in various formats here:

The book has an open-to-public GitHub repository as well.

A chapter of the book is freely available online:  File IO Essentials. Do download and check it out.

A fairly detailed description of the book, and what it attempts to cover is given below. Obviously, am very grateful to my readers. A request: once you’ve gone through the book, please take five minutes to write a review for the book on Amazon.

Hands-On System Programming with Linux

The Linux OS has grown from being a one-person tiny hobby project to a fundamental and integral part of a stunning variety of software products and projects; it is found in tiny industrial robots, consumer electronic devices (smartphones, tablets, music players); it is the powerhouse behind enterprise-scale servers, network switches and data centres all over the Earth.

This book is about a fundamental and key aspect of Linux – systems programming at the library and system call layers of the Linux stack. It will cement for you, in a fast-paced yet deeply technical manner, both the conceptual “why” and with a hands-on approach the practical “how” of systems programming on the Linux OS. Real-world relevant and detailed code examples, tested on the latest 4.x kernel distros, are found throughout the book, with added emphasis on key aspects such as programming for security. Linux was never more relevant in industry; are you?

The book’s style is one of making complex topics and ideas easy to understand; we take great pains to introduce the reader to relevant theoretical concepts such that the fundamentals are crystal clear before moving on to APIs and code. Then, the book goes on to provide detailed practical, hands-on and relevant code examples to solidify the concepts. All the code shown in the book (several example for each chapter) is available ready-to-build-and-try on the book’s GitHub repository here : https://github.com/PacktPublishing/Hands-on-System-Programming-with-Linux. Not only that, each chapter has suggested assignments for the budding programmer to try; some of these assignments have solutions provided as well (find them in the GitHub repo).

This book covers a huge amount of ground within the Linux system programming domain; nevertheless, it does so in a unique way. Several features make this book useful and unique; some of them are enumerated below:

  • How many books on programming have you read that, besides the typical ‘Hello, world’ ‘C’ program, also include a couple of assembly language examples, describe what the CPU ABI is, and a whole lot more, including a detailed description of system architecture, system calls, etc in the very first chapter
  • While we do not delve into intricate details on their usage, this book uses, and briefly shows how to use, plenty of useful and relevant tools – from ltrace and strace to perf, to LTTng and Ftrace
  • The book is indeed detailed; for example, it does not shy away from even delving (to the appropriate degree) into kernel internals wherever required. The understanding and description of an important topic such as Virtual Memory in Chapter 2, serves as a good example. Fairly advanced concepts such as looking into the process stack (gstack, GDB) and what the process VM-split is, are covered as well
  • Ch 4 on Dynamic Memory Allocation goes well beyond the usual malloc API family; here, once we get that covered (in depth of course), the book moves into more advanced topics such as internal malloc behavior, demand paging, explicitly locking and protecting memory pages, using the alloca API, etc
  • Ch 5 on Memory Issues, leads with a detailed description (and actual code test cases) of common memory defects (bugs), that novice (and even experienced!) programmers often overlook. These include the OOB (Out Of Bounds) memory accesses (read/write overflows/underflows), UAF, UAR, leakage and double free
  • Ch 6 then then continues logically forward with a detailed discussion on Tools to detect the afore-mentioned memory defects with; here, we focus on using the well known Valgrind (memcheck) and on the newer, exciting Sanitizer toolset (compiler-based tools). We compare them point for point, and explain how the modern C/C++ developer must tackle memory defects
  • File I/O: a vast topic by itself; we divide the discussion into two parts. The reader is advised to first delve into Appendix A – File IO Essentials (available online) which covers the basic concepts. Our Ch 18 on Advanced File I/O goes deeper; most programmers are aware of and frequently use the “usual” read/write system calls. We show how to optimize performance in various ways- using the MT (multithreaded) optimal pread/pwrite APIs, using scatter-gather (SG-IO). An explanation (with detailed diagrams) on the kernel picture for the block IO code path, shows the reader how I/O actually works – the kernel page cache and related components are shown. Leveraging this knowledge with the posix_fadvise and the readahead APIs is described as well. Memory mapping as a powerful “zero-copy” technique for file IO is explained with sample code. More advanced areas such as DIO and AIO are briefly delved into as well, all with the idea that the developer can leverage the system for maximum I/O performance
  • Most programmers are (dimly) aware of the traditional Unix “permissions model”; we cover this in detail from a system programming perspective in Ch 7 – Process Credentials (we explain setuid/setgid/saved-set-ID programs). We emphasize though, that there is a superior modern approach – the POSIX Capabilities model – which is covered in Ch 8. Security concerns and how these get addressed form the meat of the discussion here
  • Being a book on system programming, we obviously cover Process Execution (Ch 9) and Process Creation (Ch 10) in depth; the fork system call and all its subtleties and intricacies. Towards this end, we encode the “The rules of fork” – a set of seven rules that help the developer truly understand how this important system call actually behaves. As part of this chapter, the wait API and its variations, and the famous Unix “orphan” and “zombie” process are covered as well
  • Again, as expected, we cover the topic of Signaling in depth over two whole chapters (11 and 12). Besides the basics, the book delves into practical details (with code, of course) in covering, for example, various techniques by which one can handle a very high  volume of signals continually bombarding a process. The notion of software reentrancy (and the related reentrant-safety concept) is covered. Ways to prevent the zombie, using an alternate signal stack, etc are covered as well. The next chapter on Signaling delves into the intricacies of handling fatal signals in a real-world Linux application – a must-do! The book provides an effective “template code” to conveniently fulfil this purpose. Using real-time signals for IPC, and synchronous APIs for signaling is covered too
  • The chapter (13) on Timers covers both the traditional and the more recent powerful POSIX timers; here, we explain concepts via two interesting mini-projects – building a “how fast can you react” (CLI) game and a small “run:walk” timer app
  • Multithreading with Pthreads is again a vast area; this book devotes three whole chapters (14, 15, 16) to this important topic. In the first of this trilogy, we delve into the thread concept, how exactly it differs from a process and importantly, why threading is useful (with code examples of course). The next chapter deals in depth with the key topics of concurrency and synchronization (within the Pthreads domain, covering mutex locks and CVs); here, we use the (old but) very interesting Mars Pathfinder mission as a quick case study on priority inversion and how it can be prevented. Next, the really important topics of thread safety, cancellation and cleanup, are covered. The chapter ends with a brief “Multi processing vs MT” discussion and some typical FAQs. (We even include a ‘pthreads_app_template.c’ program in the book’s GitHub tree)
  • Ch 17 delves into intricacies of CPU Scheduling on the Linux OS; key concepts – the Linux state machine, what real-time is, POSIX scheduling policies available, etc are covered. We demonstrate a MT app that switches two worker threads to become (soft) real-time
  • The book ends with a small chapter (19) devoted to Trobleshooting and Best Practices to follow – a small but really key chapter!

Scattered throughout the book are interesting examples and reader assignments (some of which have solutions provided). Also, we try not to be completely x86 specific; in a few examples we cross-compile for the popular ARM-32 CPU; we mention the SEALS project (allowing one to quickly prototype apps on a Qemu emulated ARM/Linux system).

Who this book will benefit:

  • The professional Linux application developer
  • Application architects, leads, technical managers, consultants
  • Linux QA professionals
  • Students desirious of gaining an industry-relevant edge
  • Anybody interested in Linux programming and crafting good software in general.

Advice to a Young Firmware Developer – by Jack Ganssle; and Assembly

I hope Jack Ganssle forgives my directly copying his content; the only reason I do so is that these thoughts of his are precious and I wish for more of us to read and appreciate them. (The discussion is definitely biased towards firmware/embedded developers that work primarily on a hardware platform using ‘C’ as the language. Just a small part of Jack’s excellent Embedded Muse newsletter is shown below; do check out the full article and subscribe to his newsletter). 


Directly copied from here: The Embedded Muse, Issue #362 by jack Ganssle, 19 Nov 2018.

Advice to a Young Firmware Developer

… Learn, in detail, how a computer works. You should be able to draw a detailed block diagram of one. Even if you have no interest in the hardware, it’s impossible to understand assembly language and other important aspects of creating firmware without understanding program counters, registers, busses and the like.

Learn an assembly language. Write real programs in it, and make them work. Absent a grounding in assembly much of the operation of a computer will be mysterious. In real life you’ll have to delve into the assembly at least occasionally, at least to work on the startup code, and to find some classes of bugs.

A recent article in IEEE Spectrum surveyed language use and C didn’t even make the cut. Java, Javascript, HMTL and Python were ranked as the most in-demand languages in the USA. Yet around 70% of firmware people work in C. C++ makes up another 20%. For better or worse, all of the other embedded languages are in the noise. Master C, pointers and all. (Rust is increasingly popular, yet, despite the hype, has under a 1% share in the embedded space).

But do learn some other languages. Python can be useful for scripting. Ada gives a discipline I wish more had.

Work in a cross-development environment with an embedded target board. It’s very different from using Visual Studio.

Get comfortable with a Linux shell. With sed, awk, and a hundred other tools you can do incredible things without writing any code.

Take the time to think through what you’re building. It’s tempting to start coding early. Design is critically important. Remember the old saying: “if you think good design is expensive, consider the cost of bad design.”

Monitor your bug rates. Forever. Skip this and you’ll never know two things: if you’re improving, and how you compare to the industry. We all think we’re great developers, but metrics can be a cold shower.

Always be on the prowl for tools. Some are free, others expensive, but great tools will make you more productive and less error-prone. These change all the time so figure on constantly refining your toolbox.

Did you know the average firmware person reads one technical book a year? Yet this field evolves at the pace of Moore’s Law. Constantly study software engineering. We do have a Body of Knowledge. Every year new ideas appear. Some are brilliant, others whacky, but they all make you think.

Learn about the hardware. At least get a general understanding. An engineer who can use an oscilloscope and logic analyzer for troubleshooting code is a valuable addition to a software team. Digital and analog hardware is cool and fascinating. …”

So, okay, that’s the part of the article I wanted to show.

Learning Assembly Language – Resources

How does one just learn assembly language then? Well, there are resources of course that help – books, online articles; here’s a few: 

Pthreads Dev – Common Programming Mistakes to Avoid

Disclaimer: Okay, let me straight away say this: most of these points below are from various books and online sources. One that stands out in my mind, an excellent tome, though quite old now, is “Multithreaded Programming with Pthreads”, Bil Lewis and Daniel J. Berg.

Common Programming Errors one (often?) makes when programming MT apps with Pthreads

  • Failure to check return values for errors
  • Using errno without actually checking that an error has occurred
WRONG                       Correct
syscall_foo();              if (syscall_foo() < 0) {
if (errno) { ... }              if (errno) { ... } }

(Also, note that all Pthread APIs may not set errno)

  • Not joining on joinable threads
  • A critical one: Failure to verify that library calls are MT Safe
    Use the foo_r API version if it exists, over the foo.
    Use TLS, TSD, etc.
  • Falling off the bottom of main()
    must call pthread_exit() ; yes, in main as well!
  • Forgetting to include the POSIX_C_SOURCE flag
  • Depending upon Scheduling Order
    Write your programs to depend upon synchronization. Don’t do :

          sleep(5); /* Enough time for manager to start */

Instead, wait until the event in question actually occurs; synchronize on it, perhaps using CVs (condition variables)

  • Not Recognizing Shared Data
    Especially true when manipulating complex data structures – such as lists in which each element (or node) as well as the entire list has a separate mutex for protection; so to search the list, you would have to obtain, then release, each lock as the thread moved down the list. It would work, but be very expensive. Reconsider the design perhaps?
  • Assuming bit, byte or word stores are atomic
    Maybe, maybe not. Don’t assume – protect shared data
  • Not blocking signals when using sigwait(3)
    Any signals you’re blocking upon with the sigwait(3) should never be delivered asynchronously to another thread; block ’em and use ’em
  • Passing pointers to data on the stack to another thread

(Case a) Simple – an integer value:

Correct (below):

 // thread creation loop
 for (i=0; i<NUM_THREADS; i++) {
    thread_create(&thrd[i], &attr, worker, i);
 // join loop...


The integer is passed to each thread as a ‘literal value’; no issues.

Wrong approach (below):

 // thread creation loop
 for (i=0; i<NUM_THREADS; i++) {
     thread_create(&thrd[i], &attr, worker, &i);
  // join loop...


Passing the integer by address implies that some thread B could be reading it while thread main is writing to it! A race, a bug.

(Case b) More complex – a data structure:

Correct (below):

my_struct *pstr;
// thread creation loop
for (i=0; i<NUM_THREADS; i++) {
   pstr = (my_struct *) malloc(...);
   pstr->data = <whatever>;
   pstr->... = ...; // and so on...
   pthread_create(&thrd[i], &attr, worker, pstr);
// in the join loop..


The malloc ensures the memory is accessible to the particular thread it’s being passed to. Thread Safe.

Wrong! (below)

my_struct *pstr = malloc(...);

for (i=0; i<NUM_THREADS; i++) {
   pstr->data = <whatever>;
   pstr->... = ...; // and so on...
   pthread_create(&thrd[i], &attr, worker, pstr);
// join


If you do this (the wrong one, above), then the global pointer (one instance of the data structure only) is being passed around without protection – threads will step on “each other’s toes” corrupting the data and the app. Thread Unsafe.

  • Avoid the above problems; use the TLS (Thread-Local Storage) – a simple and elegant approach to making your code thread safe.

Resource: GCC page on TLS.



Application Binary Interface (ABI) Docs and Their Meaning

Have you, the programmer, ever really thought about how it all actually works? Am sure you have…

We write

printf("Hello, world! value = %d\n", 41+1);

and it works. But it’s ‘C’ code – the microprocessor cannot possibly understand it; all it  “understands” is a stream of binary digits – machine language. So, who or what transforms source code into this machine language?

The compiler of course! How? It just does (cheeky). So who wrote the compiler? How?
Ah. Compiler authors figure out how by reading a document provided by the microprocessor (cpu) folks – the ABI – Application Binary Interface.

People often ask “But what exactly is an ABI?”. I like the answer provided here by JesperE:

"... If you know assembly and how things work at the OS-level, you are conforming to a certain ABI. The ABI govern things like
how parameters are passed, where return values are placed. For many platforms there is only one ABI to choose from, and in those
cases the ABI is just "how things work".

However, the ABI also govern things like how classes/objects are laid out in C++. This is necessary if you want to be able to pass
object references across module boundaries or if you want to mix code compiled with different compilers. ..."

Another way to state it:
The ABI describes the underlying nuts and bolts of the mechanisms  that systems software such as the compiler, linker, loader – IOW, the toolchain – needs to be aware of: data representation, function calling and return conventions, register usage conventions, stack construction, stack frame layout, argument passing – formal linkage, encoding of object files (eg. ELF), etc.

Having a minimal understanding of :

  • a CPU’s ABI – which includes stuff like
    • it’s procedure calling convention
    • stack frame layout
    • ISA (Instruction Set Architecture)
    • registers and their internal usage, and,
  • bare minimal assembly language for that CPU,

helps to no end when debugging a complex situation at the level of the “metal”.

With this in mind, here are a few links to various CPU ABI documents, and other related tutorials:

However, especially for folks new to it, reading the ABI docs can be quite a daunting task! Below, I hope to provide some simplifications which help one gain the essentials without getting completely lost in details (that probably do not matter).

Often, when debugging, one finds that the issue lies with how exactly a function is being called – we need to examine the function parameters, locals, return value. This can even be done when all we have is a binary dump – like the well known core file (see man 5 core for details).

Intel x86 – the IA-32

On the IA-32, the stack is used for function calling, parameter passing, locals.

Stack Frame Layout on IA-32

[...                            <-- Bottom; higher addresses.
RET addr 
[SFP]                      <-- SFP = pointer to previous stack frame [EBP] [optional]
...]                           <-- ESP: Top of stack; in effect, lowest stack address

Intel 64-bit – the x86_64

On this processor family, the situation is far more optimized. Registers are used to pass along the first six arguments to a function; the seventh onwards is passed on the stack. The stack layout is very similar to that on IA-32.

Register Set


<Original image: from Intel manuals>

Actually, the above register-set image applies to all x86 processors – it’s an overlay model:

  • the 32-bit registers are literally “half” the size and their prefix changes from R to E
  • the 16-bit registers are half the size of the 32-bit and their prefix changes from E to A
  • the 8-bit registers are half the size of the 16-bit and their prefix changes from A to AH, AL.

The first six arguments are passed in the following registers as follows:


(By the way, looking up the registers is easy from within GDB: just use it’s info registers command).

An example from this excellent blog “Stack frame layout on x86-64” will help illustrate:

On the x86_64, call a function that receives 8 parameters – ‘a, b, c, d, e, f, g, h’. The situation looks like this now:


What is this “red zone” thing above? From the ABI doc:

The 128-byte area beyond the location pointed to by %rsp is considered to be reserved and shall not be modified by signal or interrupt handlers. Therefore, functions may use this area for temporary data that is not needed across function calls. In particular, leaf functions may use this area for their entire stack frame, rather than adjusting the stack pointer in the prologue and epilogue. This area is known as the red zone.

Basically it’s an optimization for the compiler folks: when a ‘leaf’ function is called (one that does not invoke any other functions), the compiler will generate code to use the 128 byte area as ‘scratch’ for the locals. This way we save two machine instructions to lower and raise the stack on function prologue (entry) and epilogue (return).

ARM-32 (Aarch32)

<Credits: some pics shown below are from here : ‘ARM University Program’, YouTube. Please see it for details>.

The Aarch32 processor family has seven modes of operation: of these, six of them are privileged and only one – ‘User’ – is the non-privileged mode, in which user application processes run.


When a process or thread makes a system call, the compiler has the code issue the SWI machine instruction which puts the CPU into Supervisor (SVC) mode.

The Aarch32 Register Set:


Register usage conventions are mentioned below.

Function Calling on the ARM-32

The Aarch32 ABI reveals that it’s registers are used as follows:

Register APCS name Purpose
R0 a1 Argument registerspassing values, don’t need to be preserved,
results are usually returned in R0
R1 a2
R2 a3
R3 a4
R4 v1 Variable registers, used internally by functions, must be preserved if used. Essentially, r4 to r9 hold local variables as register variables.

(Also, in case of the SWI machine instruction (syscall), r7 holds the syscall #).
R5 v2
R6 v3
R7 v4
R8 v5
R9 v6
R10 sl Stack Limit / stack chunk handle
R11 fp Frame Pointer, contains zero, or points to stack backtrace structure
R12 ip Procedure entry temporary workspace
R13 sp Stack Pointer, fully descending stack, points to lowest free word
R14 lr Link Register, return address at function exit
R15 pc Program Counter

(APCS = ARM Procedure Calling Standard)

When a function is called on the ARM-32 family, the compiler generates assembly code such that the first four integer or pointer arguments are placed in the registers r0, r1, r2 and r3. If the function is to receive more than four parameters, the fifth one onwards goes onto the stack. If enabled, the frame pointer (very useful for accurate stack unwinding/backtracing) is in r11. The last three registers are always used for special purposes:

  • r13: stack pointer register
  • r14: link register; in effect, return (text/code) address
  • r15: the program counter (the PC)


The PSR – Processor State Register – holds the system ‘state’; it is constructed like this:



<TODO: Aarch64>

Hope this helps!

Setting up Kdump and Crash for ARM-32 – an Ongoing Saga

Author: Kaiwan N Billimoria, kaiwanTECH
Date: 13 July 2017

DUT (Device Under Test):
Hardware platform: Qemu-virtualized Versatile Express Cortex-A9.
Software platform: mainline linux kernel ver 4.9.1, kexec-tools, crash utility.

First, my attempt at setting up the Raspberry Pi 3 failed; mostly due to recurring issues with the bloody MMC card; probably a power issue! (see this link).

Anyway. Then switched to doing the same on the always-reliable Qemu virtualizer; I prefer to setup the Vexpress-CA9.

In fact, a supporting project I maintain on github – the SEALS project – is proving extremely useful for building the ARM-32 hardware/software platform quickly and efficiently. (Fun fact: SEALS = Simple Embedded Arm Linux System).

So, I cloned the above-mentioned git repo for SEALS into a new working folder.

The way SEALS work is simple: edit a configuration file (build.config) to your satisfaction, to reflect the PATH to and versions of the cross-compiler, kernel, kernel command-line parameters, busybox, rootfs size, etc.

Setup the SEALS build.config file.

Screenshot: the build_SEALS.sh script initial screen displaying the current build config:kdumpcr1

Relevant Info reproduced below for clarity:

Toolchain prefix : arm-none-linux-gnueabi-
Toolchain version: (Sourcery CodeBench Lite 2014.05-29) 4.8.3 20140320 (prerelease)

Staging folder : <…>/SEALS_staging
ARM Platform : Versatile Express (A9)

Platform RAM : 512 MB
RootFS force rebuild : 0
RootFS size : 768 MB

Linux kernel to use : 4.9.1
Linux kernel codebase location : <…>/SEALS_staging/linux-4.9.1
Kernel command-line : “console=ttyAMA0 root=/dev/mmcblk0 init=/sbin/init crashkernel=32M”

Busybox to use : 1.26.2
Busybox codebase location : <…>/SEALS_staging/busybox-1.26.2


Screenshot: build_SEALS.sh second GUI screen, allowing the user to select actions to takekdumpcr2

Upon clicking ‘OK’, the build process starts:

I Boot Kernel Setup

  • kernel config: must carefully configure the Linux kernel. Please follow the kernel documentation in detail:
    https://www.kernel.org/doc/Documentation/kdump/kdump.txt [1]In brief, ensure these are set:
    CONFIG_SYSFS=y << should be >>

Dump-capture kernel config options (Arch Dependent, arm)
To use a relocatable kernel, Enable “AUTO_ZRELADDR” support under “Boot” options:      



which succinctly got it working!

  • Copy the ‘kexec’ binary into the root filesystem (staging tree) under it’s sbin/ folder
  • We build a relocatable kernel so that we can use the same ‘zImage’ 
    for the dump kernel as well as the primary boot kernel:
     “Or use the system kernel binary itself as dump-capture kernel and there is 
    no need to build a separate dump-capture kernel. 
    This is possible  only with the architectures which support a 
    relocatable kernel. As  of today, i386, x86_64, ppc64, ia64 and 
    arm architectures support relocatable kernel. ...”
  • the SEALS build system will proceed to build the kernel using the cross-compiler specified
  • went through just fine.

II Load dump-capture (or kdump) kernel into boot kernel’s RAM

Do read [1], but to cut a long story short

  • Create a small shell script kx.sh - a wrapper over kexec – in the root filesystem:
    DUMPK_CMDLINE="console=ttyAMA0 root=/dev/mmcblk0 rootfstype=ext4 rootwait init=/sbin/init maxcpus=1 reset_devices"
    kexec --type zImage \
    -p ./zImage-4.9.1-crk \
    --dtb=./vexpress-v2p-ca9.dtb \
    [ $? -ne 0 ] && { 
        echo "kexec failed." ; exit 1
    echo "$0: kexec: success, dump kernel loaded."
    exit 0
  • Run it. It will only work (in my experience) when:
    • you’ve passed the kernel parameter ‘crashkernel=32M’
    • verified that indeed the boot kernel has reserved 32MB RAM for the dump-capture kernel/system:
RUN: Running qemu-system-arm now ...

qemu-system-arm -m 512 -M vexpress-a9 -kernel <...>/images/zImage \
-drive file=<...>/images/rfs.img,if=sd,format=raw \
-append "console=ttyAMA0 root=/dev/mmcblk0 init=/sbin/init crashkernel=32M" \
-nographic -no-reboot -dtb <...>/linux-4.9.1/arch/arm/boot/dts/vexpress-v2p-ca9.dtb

Booting Linux on physical CPU 0x0
Linux version 4.9.1-crk (hk@hk) (gcc version 4.8.3 20140320 (prerelease) (Sourcery CodeBench Lite 2014.05-29) ) #2 SMP Wed Jul 12 19:41:08 IST 2017
CPU: ARMv7 Processor [410fc090] revision 0 (ARMv7), cr=10c5387d
CPU: PIPT / VIPT nonaliasing data cache, VIPT nonaliasing instruction cache
OF: fdt:Machine model: V2P-CA9
ARM / $ dmesg |grep -i crash
Reserving 32MB of memory at 1920MB for crashkernel (System RAM: 512MB)
Kernel command line: console=ttyAMA0 root=/dev/mmcblk0 init=/sbin/init crashkernel=32M
ARM / $ id
uid=0 gid=0
ARM / $ ./kx.sh
./kx.sh: kexec: success, dump kernel loaded.
ARM / $ 

Ok, the dump-capture kernel has loaded up.
Now to test it!

III Test the soft boot into the dump-capture kernel

On the console of the (emulated) ARM-32:

ARM / $ echo c > /proc/sysrq-trigger 
sysrq: SysRq : Trigger a crash
Unhandled fault: page domain fault (0x81b) at 0x00000000
pgd = 9ee44000
[00000000] *pgd=7ee30831, *pte=00000000, *ppte=00000000
Internal error: : 81b [#1] SMP ARM
Modules linked in:
CPU: 0 PID: 724 Comm: sh Not tainted 4.9.1-crk #2
Hardware name: ARM-Versatile Express
task: 9f589600 task.stack: 9ee40000
PC is at sysrq_handle_crash+0x24/0x2c
LR is at arm_heavy_mb+0x1c/0x38
pc : [<804060d8>] lr : [<80114bd8>] psr: 60000013
sp : 9ee41eb8 ip : 00000000 fp : 00000000


[<804060d8>] (sysrq_handle_crash) from [<804065bc>] (__handle_sysrq+0xa8/0x170)
[<804065bc>] (__handle_sysrq) from [<80406ab8>] (write_sysrq_trigger+0x54/0x64)
[<80406ab8>] (write_sysrq_trigger) from [<80278588>] (proc_reg_write+0x58/0x90)
[<80278588>] (proc_reg_write) from [<802235c4>] (__vfs_write+0x28/0x10c)
[<802235c4>] (__vfs_write) from [<80224098>] (vfs_write+0xb4/0x15c)
[<80224098>] (vfs_write) from [<80224d30>] (SyS_write+0x40/0x80)
[<80224d30>] (SyS_write) from [<801074a0>] (ret_fast_syscall+0x0/0x3c)

Code: f57ff04e ebf43aba e3a03000 e3a02001 (e5c32000) 

Loading crashdump kernel...
Booting Linux on physical CPU 0x0

Linux version 4.9.1-crk (hk@hk) (gcc version 4.8.3 20140320 (prerelease) (Sourcery CodeBench Lite 2014.05-29) ) #2 SMP Wed Jul 12 19:41:08 IST 2017
CPU: ARMv7 Processor [410fc090] revision 0 (ARMv7), cr=10c5387d
CPU: PIPT / VIPT nonaliasing data cache, VIPT nonaliasing instruction cache
OF: fdt:Machine model: V2P-CA9
OF: fdt:Ignoring memory range 0x60000000 - 0x78000000
Memory policy: Data cache writeback
CPU: All CPU(s) started in SVC mode.
percpu: Embedded 14 pages/cpu @81e76000 s27648 r8192 d21504 u57344
Built 1 zonelists in Zone order, mobility grouping on. Total pages: 7874
Kernel command line: console=ttyAMA0 root=/dev/mmcblk0 rootfstype=ext4 rootwait 
init=/sbin/init maxcpus=1 reset_devices elfcorehdr=0x79f00000 mem=31744K

ARM / $ ls -l /proc/vmcore            << the dump image (480 MB here) >>
-r-------- 1 0 0 503324672 Jul 13 12:22 /proc/vmcore
ARM / $ 

Copy the dump file (with cp or scp, whatever), 
get it to the host system.

cp /proc/vmcore <dump-file>
ARM / $ halt
ARM / $ EXT4-fs (mmcblk0): re-mounted. Opts: (null)
The system is going down NOW!
Sent SIGTERM to all processes
Sent SIGKILL to all processes
Requesting system halt
reboot: System halted
QEMU: Terminated
^A-X  << type Ctrl-a followed by x to exit qemu >>
... and done.

build_SEALS.sh: all done, exiting.
Thank you for using SEALS! We hope you like it.
There is much scope for improvement of course; would love to hear your feedback, ideas, and contribution!
Please visit : https://github.com/kaiwan/seals . 

IV Analyse the kdump image with the crash utility


The core analysis suite is a self-contained tool that can be used to
investigate either live systems, kernel core dumps created from dump
creation facilities such as kdump, kvmdump, xendump, the netdump and
diskdump packages offered by Red Hat, the LKCD kernel patch, the mcore
kernel patch created by Mission Critical Linux, as well as other formats
created by manufacturer-specific firmware.


A whitepaper with complete documentation concerning the use of this utility
can be found here:
http://people.redhat.com/anderson/crash_whitepaper [3]

The crash binary can only be used on systems of the same architecture as
the host build system. There are a few optional manners of building the
crash binary:

o On an x86_64 host, a 32-bit x86 binary that can be used to analyze
32-bit x86 dumpfiles may be built by typing "make target=X86".
o On an x86 or x86_64 host, a 32-bit x86 binary that can be used to analyze
 32-bit arm dumpfiles may be built by typing "make target=ARM".

Ah. To paraphrase, Therein lies the devil, in the details.

[UPDATE : 14 July ’17
I do have it building successfully now. The trick apparently – on x86_64 Ubuntu 17.04 – was to install the 
lib32z1-dev package! Once I did, it built just fine. Many thanks to Dave Anderson (RedHat) who promptly replied to my query on the crash mailing list.]

I cloned the ‘crash’ git repo, did ‘make target=ARM’, it fails with:

 ../readline/libreadline.a ../opcodes/libopcodes.a ../bfd/libbfd.a
../libiberty/libiberty.a ../libdecnumber/libdecnumber.a -ldl
-lncurses -lm ../libiberty/libiberty.a build-gnulib/import/libgnu.a
 -lz -ldl -rdynamic
/usr/bin/ld: cannot find -lz
collect2: error: ld returned 1 exit status
Makefile:1174: recipe for target 'gdb' failed

Still trying to debug this!

Btw, if you’re unsure, pl see crash’s github Readme on how to build it.
So, now, with a ‘crash’ binary that works, lets get to work:

$ file crash
crash: ELF 32-bit LSB shared object, Intel 80386, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux.so.2, for GNU/Linux 2.6.32, …

$ ./crash

crash 7.1.9++
Copyright (C) 2002-2017 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation

crash: compiled for the ARM architecture

To examine a kernel dump (kdump) file, invoke crash like so:

crash <path-to-vmlinux-with-debug-symbols> <path-to-kernel-dumpfile>

$ <...>/crash/crash \
  <...>/SEALS_staging/linux-4.9.1/vmlinux ./kdump.img

crash 7.1.9++
Copyright (C) 2002-2017 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
WARNING: cannot find NT_PRSTATUS note for cpu: 1
WARNING: cannot find NT_PRSTATUS note for cpu: 2
WARNING: cannot find NT_PRSTATUS note for cpu: 3

 KERNEL: <...>/SEALS_staging/linux-4.9.1/vmlinux
 DUMPFILE: ./kdump.img
 DATE: Thu Jul 13 00:38:39 2017
 UPTIME: 00:00:42
LOAD AVERAGE: 0.00, 0.00, 0.00
 TASKS: 56
 NODENAME: (none)
 RELEASE: 4.9.1-crk
 VERSION: #2 SMP Wed Jul 12 19:41:08 IST 2017
 MACHINE: armv7l (unknown Mhz)
 PANIC: "sysrq: SysRq : Trigger a crash"
 PID: 735
 COMMAND: "echo"
 TASK: 9f6af900 [THREAD_INFO: 9ee48000]
 CPU: 0

crash> ps
 0 0 0 80a05c00 RU 0.0 0 0 [swapper/0]
> 0 0 1 9f4ab700 RU 0.0 0 0 [swapper/1]
> 0 0 2 9f4abc80 RU 0.0 0 0 [swapper/2]
> 0 0 3 9f4ac200 RU 0.0 0 0 [swapper/3]
 1 0 0 9f4a8000 IN 0.1 3344 1500 init
722 2 0 9f6ac200 IN 0.0 0 0 [ext4-rsv-conver]
728 1 0 9f6ab180 IN 0.1 3348 1672 sh
> 735 728 0 9f6af900 RU 0.1 3344 1080 echo
crash> bt
PID: 735 TASK: 9f6af900 CPU: 0 COMMAND: "echo"
 #0 [<804060d8>] (sysrq_handle_crash) from [<804065bc>]
 #1 [<804065bc>] (__handle_sysrq) from [<80406ab8>]
 #2 [<80406ab8>] (write_sysrq_trigger) from [<80278588>]
 #3 [<80278588>] (proc_reg_write) from [<802235c4>]
 #4 [<802235c4>] (__vfs_write) from [<80224098>]
 #5 [<80224098>] (vfs_write) from [<80224d30>]
 #6 [<80224d30>] (sys_write) from [<801074a0>]
 pc : [<76e8d7ec>] lr : [<0000f9dc>] psr: 60000010
 sp : 7ebdcc7c ip : 00000000 fp : 00000000
 r10: 0010286c r9 : 7ebdce68 r8 : 00000020
 r7 : 00000004 r6 : 00103008 r5 : 00000001 r4 : 00102e2c
 r3 : 00000000 r2 : 00000002 r1 : 00103008 r0 : 00000001
 Flags: nZCv IRQs on FIQs on Mode USER_32 ISA ARM

And so on …

Another thing we can do is use gdb – to a limited extent – to analyse the dump file:

From [1]:

Before analyzing the dump image, you should reboot into a stable kernel.

You can do limited analysis using GDB on the dump file copied out of
/proc/vmcore. Use the debug vmlinux built with -g and run the following
  gdb vmlinux <dump-file>

Stack trace for the task on processor 0, register display, and memory
display work fine.

Also, [3] is an excellent whitepaper on using crash. Do read it.

All right, hope that helps!

Low-Level Software Design

[Please note, this article isn’t about formal design methods (LLD), UML, Design Patterns, nor about object-oriented design, etc. It’s written with a view towards the kind of software project I typically get to work on – embedded / Linux OS related, with the primary programming language being ‘C’ and/or scripting (typically with bash).]

When one looks back, all said and done, it isn’t that hard to get a decent software design and architecture. Obviously, the larger your project, the more the thought and analysis that goes into building a robust system. (Certainly, the more the years of experience, the easier it seems).

However, I am of the view that certain fundamentals never change: get them right and many of the pieces auto-slot into place. Work on a project enough and one always comes away with a  “feel” for the architecture and codebase – it’s robust, will work, or it’s just not.

So what are these “fundamentals”? Well, here’s the interesting thing: you already know them! But in the heat and dust of release pressures (“I don’t care that you need another half-day, check it in now!!!”), deadlines, production, we tend to forget the basics. Sounds familiar? 🙂

The points below are definitely nothing new, but always worth reiterating:

Low-level Design and Software Architecture

  • Jot down the requirements: why are we doing this? what do we hope to achieve?
  • Draw an overall diagram of the project, the data structures, the code flow, as you visualise it. You don’t really need fancy software tools- pencil and paper will do, especially at first.
    Arrive, gently, at the software architecture.
  • Layering helps (but one can overdo it)
    • To paraphrase- “adding a layer can be used to solve any problem in computer science” 🙂 Of course, one can quite easily add new problems too; careful!
  • It evolves – don’t be afraid to iterate, to use trial and error
    • “Be ready to throw the first one away – you’re going to anyway” – paraphrased from that classic book “The Mythical Man Month”
    • “There is no silver bullet” – again from the same book of wisdom. There is no one solution to all your problems – you’ll have to weigh options, make trade-offs. It’s like life y’know 😉
  • Design the code to be modular, structured
  • A function encapsulates an intention
    • Requirement-driven code: why is the function there?
  • Each function does exactly one thing
    • This is really important. If you can do this well, you will greatly reduce bugs, and thus, the need to debug.
  • Use configuration files (Edit: preferably in plain ASCII text format).


  • Insert function stubs – code it in detail later, get the overall low-level design and function interfacing correct first. What parameters, return value(s)? 
  • Avoid globals
    • use parameters, return values
    • in multithreaded / multiprocess environments, using any kind of global implies using a synchronization primitive of some sort (mutex, semaphore, spinlock, etc) to take care of concurrency concerns, races. Be aware – beware! – this is often a huge source of performance bottlenecks!
      Edit: When writing MT software, use powerful techniques TLS and TSD to further avoid globals.
  • Keep it minimal, and clean: Careful! don’t end up using too many (nested functions) layers – leads to “lasagna / spaghetti code” that’s hard to follow and thus understand
  • If a function’s code exceeds a ‘page’, re-look, redesign.

Of course, a project is not a dead static thing – at least it shouldn’t be. It evolves over time. Expect requirements, and thus your low-level design and code, to change. The better thought out the overall architecture though, the more resilient it will be to constant flux.

For example: you’re writing a device driver and a “read” method is attempting to read data from the ‘device’ (whatever the heck it is), but there is no data available right now, what should we do? Abort, returning an error code? Wait for data? Retry the operation thrice and see?

The “correct” answer: follow the standard. Assuming we’re working on a POSIX-compliant OS (Unix/Linux), the standard says that blocking calls must do precisely that: block, wait for data until it becomes available. So just wait for data. “But I don’t want to wait forever!” cries the application! Okay, implement a non-blocking open in that case (there’s a reason for that O_NONBLOCK flag folks!). Or a timeout feature, if it makes sense.

Shouldn’t the driver method “retry” the operation if it does not succeed at first? Short answer, No. Follow the Unix design philosophy: “provide mechanism, not policy”. Let the application define the policy (should we retry, if yes, how often; should we timeout, if yes, what’s the timeout, etc etc). The mechanism part of it – your driver’s read method implementation must work independent of, in fact ignore, such concerns. (But hey, it must be written to be concurrent and reentrant -safe. A post on that another day perhaps?).


Using the same example: lets say we do want the “read” method of our device driver to timeout after, say, 1 second. Where do we specify this value? Recollect, “provide mechanism, not policy”. So we’ll do so in the application, not the driver. But where in the app? Ah. I’d suggest we don’t hardcode the value; instead, keep it in a simple ASCII-text configuration file:


Of course, with ‘C’ one would usually put stuff like this into a header file. Fair enough, but please keep a separate header – say, app_config.h .

Try and do some crystal-ball-gazing: at some remote (or not-so-remote) point in the future, what if the project requires a GUI front-end? Probably, as an example, we will want to let the end-user view and set configuration – change the timeout, etc – easily via the GUI. Then, you will see the sense of using a simple ASCII-text configuration file to hold config values – reading and updating the values now becomes simple and clean.
Finally, nothing said above is sacred – we learn to take things on a case-by-case basis, use judgement.

A few Resources

Linux Kernel Version Timeline

I wanted to quickly look up Linux kernel release dates by version number.

All the info is on kernelnewbies.org . I’ve just copied it below…

Click on the version # links (below) to see details of that version (redirects to the kernelnewbies website).

Source: http://kernelnewbies.org/LinuxVersions
Last Updated: 01 Apr 2016


Linux 4.5 Released 13 March, 2016 (63 days)

Linux 4.4 Released 10 January, 2016 (70 days)

Linux 4.3 Released 1 November, 2015 (63 days)

Linux 4.2 Released 30 August, 2015 (70 days)

Linux 4.1 Released 21 June, 2015 (70 days)

Linux 4.0 Released 12 April, 2015 (63 days)


Linux 3.19 Released 8 February, 2015 (63 days)

Linux 3.18 Released 7 December, 2014 (63 days)

Linux 3.17 Released 5 October, 2014 (63 days)

Linux 3.16 Released 3 August, 2014 (56 days)

Linux 3.15 Released 8 June, 2014 (70 days)

Linux 3.14 Released 30 March, 2014 (70 days)

Linux 3.13 Released 19 January, 2014 (78 days)

Linux 3.12 Released 2 November, 2013 (61 days)

Linux 3.11 Released 2 September, 2013 (64 days)

Linux 3.10 Released 30 June, 2013 (63 days)

Linux 3.9 Released 28 April, 2013 (69 days)

Linux 3.8 Released 18 February, 2013 70 ( days)

Linux 3.7 Released 10 December 2012 (71 days)

Linux 3.6 Released 30 September, 2012 (71 days)

Linux 3.5 Released 21 July, 2012 (62 days)

Linux 3.4 Released 20 May, 2012 (63 days)

Linux 3.3 Released 18 March, 2012 (74 days)

Linux 3.2 Released 4 January, 2012 (72 days)

Linux 3.1 Released 24 October, 2011 (95 days)

Linux 3.0 Released 21 July, 2011 (64 days)


Linux 2.6.39 Released 18 May, 2011 (65 days)

Linux 2.6.38 Released 14 March, 2011 (69 days)

Linux 2.6.37 Released 4 January, 2011 (76 days)

Linux 2.6.36 Released 20 October, 2010 (80 days)

Linux 2.6.35 Released 1 August, 2010 (76 days)

Linux 2.6.34 Released 16 May, 2010 (81 days)

Linux 2.6.33 Released 24 February, 2010 (83 days)

Linux 2.6.32 Released 3 December, 2009 (84 days)

Linux 2.6.31 Released 9 September, 2009 (92 days)

Linux 2.6.30 Released 9 June, 2009 (77 days)

Linux 2.6.29 Released 24 March, 2009 (89 days)

Linux 2.6.28 Released 25 December, 2008 (77 days)

Linux 2.6.27 Released 9 October, 2008 (88 days)

Linux 2.6.26 Released 13 July, 2008 (87 days)

Linux 2.6.25 Released 17 April, 2008 (84 days)

Linux 2.6.24 Released 24 January, 2008 (107 days)

Linux 2.6.23 Released 9 October, 2007 (93 days)

Linux 2.6.22 Released 8 July, 2007 (73 days)

Linux 2.6.21 Released 26 April, 2007 (80 days)

Linux 2.6.20 Released 5 February, 2007 (68 days)

Linux 2.6.19 Released 29 November, 2006 (70 days)

Linux 2.6.18 Released 20 September, 2006 (95 days)

Linux 2.6.17 Released 17 June, 2006 (88 days)

Linux 2.6.16 Released 20 March, 2006 (76 days)

Linux 2.6.15 Released 3 January, 2006 (68 days)

Linux 2.6.14 Released 27 October, 2005 (59 days)

Linux 2.6.13 Released 29 August, 2005 (73 days)

Linux 2.6.12 Released 17 June, 2005 (107 days)

Linux 2.6.11 Released 2 March, 2005 (68 days)

Linux 2.6.10 Released 24 December, 2004 (66 days)

Linux 2.6.9 Released 19 October, 2004 (66 days)

Linux 2.6.8 Released 14 August, 2004 (59 days)

Linux 2.6.7 Released 16 June, 2004 (37 days)

Linux 2.6.6 Released 10 May, 2004 (36 days)

Linux 2.6.5 Released 4 April, 2004 (24 days)

Linux 2.6.4 Released 11 March, 2004 (22 days)

Linux 2.6.3 Released 18 February, 2004 (14 days)

Linux 2.6.2 Released 4 February, 2004 (26 days)

Linux 2.6.1 Released 9 January, 2004 (22 days)

Linux 2.6.0 Released 18 December, 2003

LApTaC – Learn Appreciate Teach and Contribute

For a score and 5 years – my adult working life so far – I’ve had the good fortune to pursue things and areas that interest me deeply.

Systems and assembly programming on the DOS platform in the early ’90s, Kyokushin Karate on the side, UNIX, and that too in DEC (Digital Equipment Corp, now a part of HP) in the mid-90’s, consulting and corporate training on the Linux OS after that (still going strong), with long distance running on the side. Now, last couple of years, Linux training/consulting continues, product development (my own product, yeah!), managing a product team working on an analystics database project, and of course, distance running.

Yawn, yeah, I know, they interest me not you. Nevertheless, a good ride, and still riding!
Which brings me to the title:

LApTaC :
Teach, and

An oft heard complaint, the lament of the modern person- “I don’t feel fulfilled in my life”.
Perhaps one ought to step back, reassess goals and priorities (striving too hard for how our society defines “success”?), and LApTaC in your life!

Its true you know, it never stops. Bored? Learn something new; we now know that age is certainly no barrier. That stuff we used to hear “your brain cells will die as you age and never regenerate”, turns out to be BS.

Enough said; appreciate what you have and the new things you learn everyday. I like Mark Manson’s practical and straightforward writing; read his essay “Stop Trying to be Happy”, among many others.

Also take a gander at this book: “18 Minutes: Find Your Focus, Master Distraction and Get the Right Things Done” by Peter Bregman.

A wise man once said, “If you want to see if you truly understand something, teach it”. Over a decade of training experience shows me this truth; also, the simpler you can explain it, the better you understand it. Jargon rarely, if ever, works.
Volunteer to teach something you know: the “fulfilment” payback you get from voluntary work cannot be overestimated.

and, Contribute:

A passage from the book – “A Return to Love : Reflections on the Principles of a Course in Miracles” by Marianne Williamson – has become popular as an inspirational quote:

Our deepest fear is not that we are inadequate. Our deepest fear is that we are powerful beyond measure. It is our light, not our darkness, that most frightens us. We ask ourselves, who am I to be brilliant, gorgeous, talented, fabulous? Actually, who are you not to be? You are a child of God. Your playing small doesn’t serve the world. There’s nothing enlightened about shrinking so that other people won’t feel insecure around you. We are all meant to shine, as children do. We were born to make manifest the glory of God that is within us. It’s not just in some of us; it’s in everyone. And as we let our own light shine, we unconsciously give other people permission to do the same. As we’re liberated from our own fear, our presence automatically liberates others.
So go full steam ahead and LApTaC your beautiful life!


Tech musings, hands-on, mostly Linux