Category: Kernel

Unix/Linux Kernel

oprofile a system wide profiler

oprofile a system wide profiler

By | | Comments 0 Comment

** What is oprofile

In short “oprofile is a system-wide profiler”
need to profile an application and its shared libraries, examine hardware effects such as cache misses and capture the performance behaviour of entire system, then surely you need go with oprofile.

There is to many options and conditions spcefied, where we use only simple and easy one for further details, Please Read Man Pages 😉

Many CPUs provide “performance counters”, hardware registers that can count “events”; for example, cache misses, or CPU cycles. OProfile provides profiles of code based on the number of these occurring events: repeatedly, every time a certain (configurable) number of events has occurred, the PC value is recorded. This information is aggregated into profiles for each binary image.

Some hardware setups do not allow OProfile to use performance counters: in these cases, no events are available, and OProfile operates in timer/RTC mode

** All Docs, Examples and Bugs you will find @ http://oprofile.sourceforge.net

# download tar from http://oprofile.sourceforge.net/download/ or you can install it using yum also

[root@ravi.com ~]# yum install oprofile

Before you can use OProfile, you must set it up. The minimum setup required for this is to tell OProfile where the vmlinux file corresponding to the running kernel is, for example :

[root@ravi.com ~]#  opcontrol –vmlinux=/boot/vmlinux-`uname -r`

If you don’t want to profile the kernel itself, you can tell OProfile you don’t have a vmlinux file :

[root@ravi.com ~]# opcontrol –no-vmlinux

Here we used –no-vmlinux and specified other session directory than default location (/var/lib/oprofile)

[root@ravi.com ~]# opcontrol –no-vmlinux –session-dir=/home/prod/tmpsession

Now we are ready to start the daemon (oprofiled) which collects the profile data :

[root@ravi.com ~]# opcontrol –start –session-dir=/home/prod/tmpsession
Using default event: GLOBAL_POWER_EVENTS:100000:1:1:1
Using 2.6+ OProfile kernel interface.
Using log file /home/prod/tmpsession/samples/oprofiled.log
Daemon started.
Profiler running.

When I want to stop profiling, I can do so with :
(Here I have ran opcontrol only 1 min)
[root@ravi.com ~]# opcontrol –shutdown
Stopping profiling.
Killing daemon.

Lets see what your system & apps tell you 🙂
ophelp : This utility lists the available events and short descriptions.

[root@ravi.com ~]# opreport –session-dir=/home/prod/tmpsession
CPU: P4 / Xeon with 2 hyper-threads, speed 2992.73 MHz (estimated)
Counted GLOBAL_POWER_EVENTS events (time during which processor is not stopped) with a unit mask of 0x01 (mandatory) count 100000
GLOBAL_POWER_E…|
samples|      %|
——————
1109031 51.2560 no-vmlinux
479089 22.1420 libc-2.5.so
207263  9.5791 libperl.so
97973  4.5280 libpython2.4.so.1.0
91993  4.2516 nagios
35979  1.6628 php
35765  1.6529 libz.so.1.2.3
25203  1.1648 ld-2.5.so
21412  0.9896 mysqld
11566  0.5345 libgd.so.2.0.0
10008  0.4625 oprofiled
7661  0.3541 libpthread-2.5.so
6736  0.3113 libnetsnmp.so.10.0.3
5698  0.2633 libpng12.so.0.10.0
4488  0.2074 rateup
3500  0.1618 libcrypto.so.0.9.8e
2138  0.0988 bash
1556  0.0719 libm-2.5.so
1234  0.0570 libmysqlclient.so.15.0.0
606  0.0280 libpcre.so.0.0.1
328  0.0152 mysql.so
316  0.0146 librrd.so.4.1.3
290  0.0134 atop
272  0.0126 sendmail.sendmail
243  0.0112 grep
176  0.0081 nscd
172  0.0079 ping
153  0.0071 nebmodBY0Y2I (deleted)
144  0.0067 libgnutls.so.13.0.6
121  0.0056 init
106  0.0049 gawk

with using symbols you can simply find which apps supporting module takes high load
[root@ravi.com ~]#  opreport –exclude-dependent –symbols –long-filenames –session-dir=/home/prod/tmpsession
CPU: P4 / Xeon with 2 hyper-threads, speed 2992.73 MHz (estimated)
Counted GLOBAL_POWER_EVENTS events (time during which processor is not stopped) with a unit mask of 0x01 (mandatory) count 100000
samples  %        app name                 symbol name
1109031  51.2560  /no-vmlinux              /no-vmlinux
209506    9.6827  /lib64/libc-2.5.so       _int_free
207263    9.5791  /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
97973     4.5280  /usr/lib64/libpython2.4.so.1.0 /usr/lib64/libpython2.4.so.1.0
91993     4.2516  /usr/local/nagios/bin/nagios /usr/local/nagios/bin/nagios
44574     2.0601  /lib64/libc-2.5.so       free
35765     1.6529  /usr/lib64/libz.so.1.2.3 /usr/lib64/libz.so.1.2.3
35032     1.6191  /lib64/libc-2.5.so       vfprintf
22061     1.0196  /lib64/libc-2.5.so       _IO_vfscanf
21412     0.9896  /usr/libexec/mysqld      /usr/libexec/mysqld
16883     0.7803  /lib64/libc-2.5.so       _int_malloc
13631     0.6300  /lib64/libc-2.5.so       memcpy
13165     0.6084  /lib64/libc-2.5.so       _IO_file_xsputn@@GLIBC_2.2.5
1560      0.0721  /lib64/ld-2.5.so         _dl_relocate_object
1462      0.0676  /usr/bin/php             lstep
1385      0.0640  /usr/bin/php             zend_do_fcall_common_helper_SPEC
1258      0.0581  /lib64/libc-2.5.so       rawmemchr
1234      0.0570  /usr/lib64/mysql/libmysqlclient.so.15.0.0 /usr/lib64/mysql/libmysqlclient.so.15.0.0
1226      0.0567  /lib64/libc-2.5.so       realloc
1180      0.0545  /lib64/libc-2.5.so       __tzfile_compute
1156      0.0534  /lib64/ld-2.5.so         check_match.8509
1146      0.0530  /usr/bin/php             _zend_mm_free_int
1084      0.0501  /usr/bin/php             zend_hash_quick_find
1029      0.0476  /lib64/libc-2.5.so       __mpn_mul_1
747       0.0345  /lib64/libc-2.5.so       fgets
738       0.0341  /usr/bin/php             _zval_ptr_dtor

Ref: http://oprofile.sourceforge.net/doc/index.html

-Ravi

How to disable core(s) of CPU

How to disable core(s) of CPU

By | | Comments 2 comments

Introduction ~

The question is why we  need to disable few core of CPU? Sometime it is necessary to run certain applications, which are not compatible with multi core processing.  Disabling core will not affect physically your hardware. Linux OS will simply ignore the core(s) you selected to disable.

Steps 1] How to do it?

Debian/Ubuntu ~

root@laptop:/home/arunsb# cat /boot/grub/menu.lst

title        Ubuntu 9.04, kernel 2.6.28-11-generic
kernel        /vmlinuz-2.6.28-11-generic root=UUID=55d33e45-75c7-54sc-b204-97b44e1d6a39 ro quiet splash maxcpus=1
initrd        /initrd.img-2.6.28-11-generic

Redhat/Fedora based system ~

root@laptop:/home/arunsb# cat /boot/grub/grub.conf

title Red Hat Enterprise Linux ES (2.6.9-78.ELsmp)
root (hd0,4)
kernel /boot/vmlinuz-2.6.9-78.ELsmp ro root=LABEL=/    maxcpus=1
initrd /boot/initrd-2.6.9-78.ELsmp.img

Note ~ after changing grub config file please reboot the system to apply changes!

As shown above “maxcpus=1” indicates that Linux will use only one CPU core. you can change this value as per your requirement and hardware available.

You can  also change this value during  starting of system from GRUB menu but it is temporary setting. To make it permanent you need to modify the  grub.conf (Redhat/Fedora) or menu.lst (debian/Ubuntu) GRUB config file.

Step 2] How to verify ~

I have dual core CPU as shown below and I have disable 1 core so After reboot I should get only one CPU core active

** Before above setting!

root@laptop:/home/arunsb# cat /proc/cpuinfo   | grep processor
processor    : 0
processor    : 1
root@laptop:/home/arunsb#

* Verify after above setting ~

root@laptop:/home/arunsb# cat /proc/cpuinfo   | grep processor
processor       : 0
root@laptop:/home/arunsb#

* How to Disable CPU without Reboot?
root@arunb:~# echo 0 > /sys/devices/system/cpu/cpu1/online

* Confirm ?
root@arunb:~# cat /proc/cpuinfo | grep -i ‘Processor’
processor : 0
root@arunb:~#

Thank you,
Arun Bagul

How to create edit/extract initrd in Ubuntu/Debian and Redhat/Fedora Linux ?

How to create edit/extract initrd in Ubuntu/Debian and Redhat/Fedora Linux ?

By | | Comments 3 comments

Introduction ~

Long back I edited initrd as  old linux (Ubuntu 6.06) box was not able to boot with SCSI hard disk? One of my friend wanted to do the same for other purpose. So got a chance to write article on the same? Let’s start with what is initrd?

What is initrd ?

initrd (Initial Ram Disk) is a temporary file system ( used as /) commonly used in the boot process of the Linux kernel. It is typically used for making preparations before the real root file system can be mounted.

Why someone want to edit/modify initrd ?

I assume that you all are familier with Linux booting process? Once Linux kernel loaded in to memory (RAM) it start init (father/mother of all  process) process. is that true? Let me ask you one question. Before loading actual physical root file system (/) how kernel access /sbin/init script? what is the use by specifying “initrd” file in GRUB ?  hold on!!

Suppose your root partion resides on some SCSI device and driver for this SCSI devices is compiled as a kernel module. Of course this module is required at boot time to have access to the root partion — but it is not in the kernel. Thus the need for an initrd image. Additionally after udev subsystem become common, somebody has to start udev to create device nodes. This is initrd’s duty too.

See the GRUB menu as shown below ~

title        Ubuntu 9.04, kernel 2.6.28-11-generic
kernel        /vmlinuz-2.6.28-11-generic root=/dev/sda3  ro quiet splash
initrd        /initrd.img-2.6.28-11-generic

GRUB loads  kernel and initrd image in to memory(RAM). When kernel boots  it checks for initrd image, and if it exists starts init script that resides on this image. init script is usually written in bash. When init script on initrd image is finished, kernel usually start standard init process ie /sbin/init

Step 1] Copy original initrd image file to temp location  ~

** Create temporary directory and copy initrd file in that temp directory

arunsb@laptop:~$ cp /boot/initrd.img-2.6.28-11-generic  /tmp/

arunsb@laptop:~$ mkdir /tmp/initrd-src

** Now extract “initrd” image –

arunsb@laptop:~$ cd /tmp/initrd-src

arunsb@laptop:/tmp/initrd-src$ gzip -dc  /tmp/initrd.img-2.6.28-11-generic  | cpio -id
38791 blocks
arunsb@laptop:/tmp/initrd-src$ ls -l
total 36
drwxr-xr-x 2 arunsb arunsb 4096 2009-07-12 16:32 bin
drwxr-xr-x 3 arunsb arunsb 4096 2009-07-12 16:32 conf
drwxr-xr-x 6 arunsb arunsb 4096 2009-07-12 16:32 etc
-rwxr-xr-x 1 arunsb arunsb 4825 2009-07-12 16:32 init
drwxr-xr-x 5 arunsb arunsb 4096 2009-07-12 16:32 lib
drwxr-xr-x 2 arunsb arunsb 4096 2009-07-12 16:32 sbin
drwxr-xr-x 8 arunsb arunsb 4096 2009-07-12 16:32 scripts
drwxr-xr-x 3 arunsb arunsb 4096 2009-07-12 16:32 usr
arunsb@laptop:/tmp/initrd-src$

** Check how “init” looks like ~

arunsb@laptop:/tmp/initrd-src$ head init
#!/bin/sh

echo “Loading, please wait…”

[ -d /dev ] || mkdir -m 0755 /dev
[ -d /root ] || mkdir -m 0700 /root
[ -d /sys ] || mkdir /sys
[ -d /proc ] || mkdir /proc
[ -d /tmp ] || mkdir /tmp
mkdir -p /var/lock
arunsb@laptop:/tmp/initrd-src$

Step 2] Edit/Modify as per your requirement

Step 3] How to create initrd image  ~

Create initrd image from scratch –

root@laptop:/home/arunsb# mkinitramfs  -v -o  /tmp/initrd-arun-$(uname -r)

root@laptop:/home/arunsb# ls -l /tmp/initrd-arun-2.6.28-11-generic
-rw-r–r– 1 root root 7536506 2009-07-12 17:11 /tmp/initrd-arun-2.6.28-11-generic

root@laptop:/home/arunsb# du -sh /tmp/initrd-arun-2.6.28-11-generic
7.2M    /tmp/initrd-arun-2.6.28-11-generic
root@laptop:/home/arunsb#

mkinitramfs ~ is the tool used to create initrd image. “initrd” image is a gzipped cpio archive.

** After all modifcation create initrd image as shown below…

arunsb@laptop:/tmp/initrd-src$ find . | cpio –quiet –dereference -o -H newc | gzip -9 > /tmp/initrd.img-2.6.28-11-arun
arunsb@laptop:/tmp/initrd-src$ ls -l /tmp/initrd.img-2.6.28-11-arun
-rw-r–r– 1 arunsb arunsb 7505955 2009-07-12 16:56 /tmp/initrd.img-2.6.28-11-arun
arunsb@laptop:/tmp/initrd-src$

* Enjoy !!

Regards,
Arun Bagul

Authentication modules in Linux/Unix – PAM

Authentication modules in Linux/Unix – PAM

By | | Comments 0 Comment

Pluggable authentication modules or PAM provides a way to develop programs that are independent of authentication scheme. These programs need “authentication modules” to be attatched to them at run-time in order to work. Which authentication module is to be attatched is dependent upon the local system setup and is at the discretion of the local system administrator.

Pluggable authentication modules or PAM are a mechanism to integrate multiple low-level authentication schemes into a high-level API, which allows for programs that rely on authentication to be written independently of the underlying authentication scheme. PAM was first proposed by Sun Microsystems in an Open Software Foundation RFC dated October, 1995. It was adopted as the authentication framework of the Common Desktop Environment. As a stand-alone infrastructure, however, PAM first appeared from an open-source, Linux-PAM, development in Red Hat Linux 3.0.4 in August of 1996. PAM is currently supported in AIX, FreeBSD, HP-UX, Linux, Mac OS X, NetBSD and Solaris. PAM was later standardized as part of the X/Open UNIX standardization process, resulting in the X/Open Single Sign-on (XSSO) standard.

The pluggable nature of PAM is one reason for using dynamic linking of system binaries. However, this necessitates the availability of a recovery mechanism should a problem develop in the linker or shared libraries; for example both NetBSD and FreeBSD supply a /rescue directory containing statically linked versions of important system binaries.

pam-nss