Category Archives: SysAd

Simple System Monitoring for a Linux Desktop

The Problem

What exactly is eating into my HDD / processor / network right now??

Yeah! On the (Linux) desktop, we’d like to know why things crawl along sometimes. Which process(es) is the culprit behind that disk activity, or the memory hogger, or eating up network bandwidth?

Many tools exist that can help us pinpoint these facts. Sometimes, though, it’s just easier if someone shows us a quick easy way to get relevant facts; so here goes:

Continue reading Simple System Monitoring for a Linux Desktop

Linux Tools for the serious Systems Programmer

Tools that help. When developing code (systems programming) on the Linux OS: a compilation by Kaiwan N Billimoria :

Tools

Tool Type

Detail

ARM support (on target)?

Comments

USER-SPACE

       
find/grep Source Code browsers Y -busybox Source; reqd on host dev system only
cscope NA
ctags NA
Source Code static analysis. FOSS NA
splint (prev LCLint) NA
Coverity / Klocwork / etc Commercial ?
strace Application trace Y
ltrace Y
[f]printf Application – simple instrumentation Y Code-based
My “MSG” and other macros  Header file  Useful Y
gdb Source-level debuggers Y Usually on host dev system only
ddd ?
Insight ?
ps Process state Y -busybox
pgrep, pkill Y -busybox
pstree ?
top Y
pidstat ?
procfs System state / performance tuning
vmstat generic Y
dstat  Tip:
dstat –time –top-io-adv –top-cpu –top-mem 5
(every 5s)
iotop, iostat, ionice disk IO Y buildroot
sar ? package: sysstat
lsof ?
munin
Valgrind Memory Checkers and analysis Considered the best OSS memory checker suite Y -ver 3.7 on buildroot; only for Cortex A8/A9 && kernel ver < 3.x
MEMWATCH Y
YAMD ?
Electric Fence ?
Dmalloc Y
mtrace Y
iftop Network monitoring, etc ?
iptraf ?
netstat Y -netstat-nat
ethtool Y
tcpdump Y
wireshark Ethernet, USB sniffer N GUI- on host
 Also, BTW, here’s a nice link :

16 commands to check hardware information on Linux

KERNEL

 
printk Kernel – simple instrumentation Y Kernel code-based debugging techniques [note: recommend you use debugfs and not procfs for debug-related stuff].
My “MSG” and other macros  Header file  Useful Y
procfs Kernel Analysis & Tuning w/ sysctl Y
ioctl Y
debugfs Recommended Y
Magic SysRq During development / system lockups Y
gdb with proc/kcore Kernel lookup Y
KGDB Kernel development debugging Y
KDB ?
KProbes, JProbes Non-intrusive kernel hooks  V useful; for learning / debugging Y
SystemTap Kernel scriptable tracing/probing instrumentation tool  (AFAIK, layered on Kprobes) ?
Ftrace Kernel trace framework Y
OProfile Kernel and App profiler ?
LTTng Linux Trace Toolkit next gen – Instrumentation ?
Kdump, Kexec and Crash Crash dump and analysis Y -kexec crash -on host
Perf / Perfmon2 HW-based performance monitoring Y (limited?) Arch-independent
cpufreq Power Management
powerTOP
CGroups Scheduler Y
Proc – sysctl Y
chrt Y buildroot
cpuset, taskset Y buildroot
sparse Kernel-space static code analysis NA -src Reqd on dev host only
QEMU Virtualization, open source Y
VirtualBox ?
KVM N
Tip: Using buildroot,enable the packages/features you want for embedded!
Kaiwan N Billimoria, kaiwanTECH.

A quick-ref pic from Brendan Gregg’s fantastic site on Linux Performance tools (and Linux performance monitoring in general):

linux-tools-BGregg
Click to zoom

 

Exploring Linux procfs via shell scripts

Very often, while working on a Linux project, we’d like information about the system we’re working on: both at a global scope and a local (process) scope.

Have we not wondered: is there a quick way to query which kernel version am using, what interrupts are enabled & hit, what my processor(s) are, details about kernel subsystems, memory usage, file, network, IPC usage, etc etc. Linux’s proc filesystem makes this easy.

So what exactly is the proc filesystem all about?

Essentially, some quick salient points about the proc filesystem:

  • it’s a RAM-based filesystem (think ramdisk; yup, it’s volatile)
  • it’s a kernel feature, not userspace – proc is a filesystem supported by the Linux kernel VFS
  • it serves two primary purposes
    • proc serves as a “view” deep into the kernel internals; we can see details about hardware and software subsystems that userspace otherwise would have no access to (no syscalls)
    • certain “files” under proc, typically anchored under /proc/sys, can be written into: these basically are the “tuning knobs” of the Linux kernel. Sysads, developers, apps, etc exploit this feature
  • proc is mounted on start-up under /proc
  • a quick peek under /proc will show you several “files” and “folders”. These are pseudo-entries in the sense that they exist only in RAM while power is applied. The “folders” that are numbers are in fact the PID of each process that’s alive when you typed ‘ls’! it’s a snapshot of the system at that moment in time..
  • in fact, the name “proc” suggests “process”

At this point, and if you’re not really familiar with this stuff, I’d urge you to peek around /proc on your Linux box, cat-ting stuff as you go. (Also, lest i forget, it’s better to run as root (sudo /bin/bash) so that we don’t get annoying ‘permission denied’ messages). Of course, be careful when you run as root!!!

For example, to get one started off:

Continue reading Exploring Linux procfs via shell scripts

LINUX CHEATSHEET – vmstat, ps, top

LINUX Quick Reference Cheat Sheet

vmstat, ps, top


v 0.1 : Last Updated: March 2009 : <kaiwan at designergraphix dot com>
(c) kaiwan billimoria.

Much of the information below gleaned from various Linux man pages.

 

———————————————————-

vmstat

vmstat fields quick reference

The -a switch displays active/inactive memory, given a 2.5.41 kernel or better.
The -f switch displays the number of forks since boot. This includes the fork, vfork, and clone system calls, and is equivalent to the total number of tasks created. Each process is represented by one or more tasks, depending on thread usage. This display does not repeat.
The -m displays slabinfo.
The -n switch causes the header to be displayed only once rather than periodically.
The -s switch displays a table of various event counters and memory statistics. This display does not repeat.

delay is the delay between updates in seconds. If no delay is specified, only one report is printed with
the average values since boot.
count is the number of updates. If no count is specified and delay is defined, count defaults to infinity.

The -d reports disk statistics (2.5.70 or above required)
The -p followed by some partition name for detailed statistics (2.5.70 or above required)
The -S followed by k or K or m or M switches outputs between 1000, 1024, 1000000, or 1048576 bytes
The -V switch results in displaying version information.

FIELD DESCRIPTION FOR VM MODE

Procs

r: The number of processes waiting for run time << ready-to-run >>.
b: The number of processes in uninterruptible sleep << blocked >>.

Memory << (default) in kilobytes >>
swpd: the amount of virtual memory used.

free: the amount of idle memory.
buff: the amount of memory used as buffers.
cache: the amount of memory used as cache << page cache, not incl. swap cache>> .
inact: the amount of inactive memory. (-a option)
active: the amount of active memory. (-a option)

Swap << in kilobytes/second >>

si: Amount of memory swapped in from disk (/s).
so: Amount of memory swapped to disk (/s).
 Continue reading LINUX CHEATSHEET – vmstat, ps, top