Category: Redhat & Fedora

Redhat & Fedora

Shellshock ‘Deadly serious’ new tech bug found!

Shellshock ‘Deadly serious’ new tech bug found!

This issue affects all products which use the Bash shell and parse values of environment variables. This issue is especially dangerous as there are many possible ways Bash can be called by an application. Quite often if an application executes another binary, Bash is invoked to accomplish this. Because of the pervasive use of the Bash shell, this issue is quite serious and should be treated as such.

All versions prior to those listed as updates for this issue are vulnerable to some degree.

My infected OS version is CentOS-6 and bash version 4.1.2

[root@host75 ~]# lsb_release -a
lsb_release -a
LSB Version: :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
Distributor ID: CentOS
Description: CentOS release 6.4 (Final)
Release: 6.4
Codename: Final

[root@host75 ~]# bash --version
bash --version
GNU bash, version 4.1.2(1)-release (x86_64-redhat-linux-gnu)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Diagnostic Steps:

To test if your version of Bash is vulnerable to this issue, run the following command:

$ env x='() { :;}; echo vulnerable' bash -c "echo this is a test"

If the output of the above command looks as follows:

vulnerable
this is a test

hmm, I got infected!

[root@host75 ~]# env x='() { :;}; echo vulnerable' bash -c "echo this is a test"
vulnerable
this is a test

You are using a vulnerable version of Bash. The patch used to fix this issue ensures that no code is allowed after the end of a Bash function. Thus, if you run the above example with the patched version of Bash, you should get an output similar to:

$ env x='() { :;}; echo vulnerable' bash -c "echo this is a test"
bash: warning: x: ignoring function definition attempt
bash: error importing function definition for `x'
this is a test

If your system is vulnerable, update to the most recent version of the Bash package by running the following command:

$yum update bash

This fix my bash ??????

[root@host75 ~]# yum update bash
Loaded plugins: fastestmirror, security, tmprepo
Loading mirror speeds from cached hostfile
epel/metalink | 15 kB 00:00
* base: centos.eecs.wsu.edu
* epel: mirrors.kernel.org
* extras: centos.chi.host-engine.com
* updates: mirror.raystedman.net
base | 3.7 kB 00:00
epel | 4.4 kB 00:00
epel/primary_db | 6.3 MB 00:05
extras | 3.3 kB 00:00
updates | 3.4 kB 00:00
updates/primary_db | 5.3 MB 00:04
Setting up Update Process
Resolving Dependencies
--> Running transaction check
---> Package bash.x86_64 0:4.1.2-14.el6 will be updated
---> Package bash.x86_64 0:4.1.2-15.el6_5.1 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================================================================================
Package Arch Version Repository Size
================================================================================================================================================
Updating:
bash x86_64 4.1.2-15.el6_5.1 updates 905 k

Transaction Summary
================================================================================================================================================
Upgrade 1 Package(s)

Total download size: 905 k
Is this ok [y/N]: y
Downloading Packages:
bash-4.1.2-15.el6_5.1.x86_64.rpm | 905 kB 00:00
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Updating : bash-4.1.2-15.el6_5.1.x86_64 1/2
Cleanup : bash-4.1.2-14.el6.x86_64 2/2
Verifying : bash-4.1.2-15.el6_5.1.x86_64 1/2
Verifying : bash-4.1.2-14.el6.x86_64 2/2

Updated:
bash.x86_64 0:4.1.2-15.el6_5.1

Complete!

Test if update fixed to patch your bash

[root@host75 ~]# env x='() { :;}; echo vulnerable' bash -c "echo this is a test"
bash: warning: x: ignoring function definition attempt
bash: error importing function definition for `x'
this is a test

How does this impact systems:
This issue affects all products which use the Bash shell and parse values of environment variables. This issue is especially dangerous as there are many possible ways Bash can be called by an application. Quite often if an application executes another binary, Bash is invoked to accomplish this. Because of the pervasive use of the Bash shell, this issue is quite serious and should be treated as such.

All versions prior to those listed as updates for this issue are vulnerable to some degree.

See the appropriate remediation article for specifics.

Functions written in Bash itself do not need to be changed, even if they are exported with “export -f”. Bash will transparently apply the appropriate naming when exporting, and reverse the process when importing function definitions.

Ref:
http://www.bbc.com/news/technology-29361794
https://www.us-cert.gov/ncas/current-activity/2014/09/24/Bourne-Again-Shell-Bash-Remote-Code-Execution-Vulnerability
https://access.redhat.com/articles/1200223
https://securityblog.redhat.com/2014/09/24/bash-specially-crafted-environment-variables-code-injection-attack/
http://lists.gnu.org/archive/html/bug-bash/2014-09/threads.html
https://rhn.redhat.com/errata/RHSA-2014-1306.html

Selecting virtual SCSI Controllers for Disks (VMware VM)

Selecting virtual SCSI Controllers for Disks (VMware VM)

Introduction-
To access virtual disks, a virtual machine uses virtual SCSI controllers. Each virtual disk that a virtual machine can access through one of the virtual SCSI controllers resides in the VMFS datastore, NFS-based datastore, or on a raw disk. The choice of SCSI controller does not affect whether your virtual disk is an IDE or SCSI disk.

Following virtual SCSI controllers commonly used…

A) BusLogic
– This was one of the first emulated vSCSI controllers available in the VMware platform.
– No updates and considered as legacy or for backward compatibility…

B) LSI Logic Parallel
– This was the other emulated vSCSI controller available originally in the VMware platform.
– Most operating systems had a driver that supported a queue depth of 32 and it became a very common choice, if not the default
– Default for Windows 2003/Vista and Linux

C) LSI Logic SAS
– This is an evolution of the parallel driver to support a new future facing standard.
– It began to grown popularity when Microsoft required its use for MCSC within Windows 2008 ore newer.
– Default for Windows 2008 or newer
– Linux guests SCSI disk hotplug works better with LSI Logic SAS
– Personally I use this
D) VMware Paravirtual (aka PVSCSI)
– This vSCSI controller is virtualization aware and was been designed to support very high throughput with minimal processing cost and is therefore the most efficient driver.
– In the past, there were issues if it was used with virtual machines that didn’t do a lot of IOPS, but that was resolved in vSphere 4.1.

* PVSCSI and LSI Logic Parallel/SAS are essentially the same when it comes to overall performance capability.
* Total of 4 vSCSI adapters are supported per virtual machine.  To provide the best performance, one should also distribute virtual disk across as many vSCSI adapters as possible
* Why not IDE? – IDE adapter completes one command at a time while SCSI can queue commands. So SCSI adapter is better optimized for parallel performance. Also Maximum of 4 IDE Devices per VM (includes CDROM) but SCSI allows 60 devices.

Thank You,
Arun

Difference between (Extended) ext2/3 and ext4 File System

Difference between (Extended) ext2/3 and ext4 File System

Difference between (Extended) ext2/3 and ext4 File System

* Ext2
-It was introduced in 1993. Developed by Remy Card.
-ext2 stands for second extended file system.
-This was developed to overcome the limitation of the original ext file system.
-ext2 does not have journaling feature.
-ext2 is recommended for flash drives, usb drives etc
-Maximum individual file size can be from 16GB to 2TB (depends on block size)
-Overall ext2 FS size can be from 2TB to 32TB

* Ext3
-It was introduced in 2001. Developed by Stephen Tweedie.
-ext3 stands for third extended file system.
    -The main benefit of ext3 is that it allows journaling.
-Journaling has a dedicated area in the file system, where all the changes are tracked. When the system crashes,file system
corruption chances are less because of journaling.
-Maximum individual file size can be from 16GB to 2TB
-Overall ext3 FS size can be from 2TB to 32TB
-There are three types of journaling available in ext3 file system.
1) Journal – both Metadata and Content are saved in the journal.
2) Ordered – Only metadata is saved in the journal. Metadata are journaled only after writing the content to disk. This is the default.
3) Writeback – Only metadata is saved in the journal. Metadata might be journaled either before or after the
content is written to the disk.
    -You can convert a ext2 file system to ext3 file system directly (without backup/restore).

* Ext4
-It was introduced in 2008.
-Ext4 stands for fourth extended file system.
-Starting from Linux Kernel 2.6.19 ext4 was available.
-Maximum individual file size can be from 16 GB to 16TB
-Overall maximum ext4 FS size is 1024PB (petabyte), 1PB = 1024TB (terabyte)
-Directory can contain a maximum of 64,000 subdirectories (as opposed to 32,000 in ext3)
-You can also mount an existing ext3 FS as ext4 fs (without having to upgrade it)
    -ext4 default inode size is 256 bytes.(in ext3 inode size is 128 bytes)
-Several other new features are introduced in ext4: multiblock allocation, delayed allocation, journal checksum. fast fsck, etc. All you need to know is that these new features have improved the performance and reliability of the filesystem when compared to ext3
-In ext4, you also have the option of turning the journaling feature “off”.
-Faster file system checking as Unallocated blocks are skipped during FS checking
-Improved timestamps- Up to the nanosecond. Which will defer the year 2038 problem
-Online Defragmentation

What is Extents?
-Ext3 uses a block mapping scheme (block 4Kb), the bigger the file needs huge block mapping will lead to slower handling.
-Ext4 introduces the concept of Extents. An extent is basically a “Bunch of blocks”.
Basically it say “write the data is in the next N blocks ie extent” instead of mapping each individual block separately.
-Ext4 will support up to 128Mb extents,This improve performance and also help in reducing fragmentation.

Multiblock Allocation-
-Ext3 uses a block allocator that decides which free blocks will be used to write the data. But this allocator
can only allocate one block at a time.
-Ext4 will support multi-block allocation, which allocates many blocks in a single call and avoids a lot of overhead.

Thank you,
Arun Bagul

Check_mk Multisite and PNP4nagios

Check_mk Multisite and PNP4nagios

Introduction-

Nagios Check_mk Multisite (plugin) allow user to view/manage distributed nagios using single Web based Interface. However by default
it doesn’t support pnp4nagios graphs (hosts/services from remote nagios) access using (single) Multisite URL.

* To access PNP4nagios graphs of hosts/services from remote nagios using (single) Multisite URL, we need to Add Apache Proxy redirect setting.

1] multisite.mk Conf file-

This is my “check_mk/multisite.mk” conf file. (from Primary multisite Server (prod), SITE1 and SITE2 are two remote nagios)

OMD[prod]:~$ cat etc/check_mk/multisite.mk

….
sites = {
#Primary site
“local” : {
“alias” : “PROD”
},
# Remote site
“SITE1”: {
“alias”: “SITE1”,
“socket”: “tcp:192.168.1.10:6557”,
“url_prefix”: “/SITE1/”,
“nagios_url”: “/SITE1/nagios”,
“nagios_cgi_url”: “/SITE1/nagios/cgi-bin”,
“pnp_url”: “/SITE1/pnp4nagios”,
},
# Remote site
“SITE2”: {
“alias”: “SITE2”,
“socket”: “tcp:192.168.2.10:6557”,
“url_prefix”: “/SITE2/”,
“nagios_url”: “/SITE2/nagios”,
“nagios_cgi_url”: “/SITE2/nagios/cgi-bin”,
“pnp_url”: “/SITE2/pnp4nagios”,
},
}
….
…..
OMD[prod]:~$

2] Apache Proxy Redirect for PNP4nagios –

OMD[prod]:~# cat /etc/httpd/conf.d/multisite_proxy.conf
#SITE1
<Location /SITE1/>
RewriteEngine On
RewriteRule ^/.+/SITE1/(.*) http://192.168.1.10/SITE1/$1 [P]
</Location>

#SITE2
<Location /SITE2/>
RewriteEngine On
RewriteRule ^/.+/SITE2/(.*) http://192.168.2.10/SITE2/$1 [P]
</Location>

OMD[prod]:~#

That’s it! Now you can access pnp4nagios graphs form remote nagios hosts/services using single Multisite URL.

Thank you,
Arun Bagul

How to use RAM (physical) as disk

How to use RAM (physical) as disk

Introduction –
Sometime to gain performance, physical ram can be used as disk space which is very fast compare to normal hard disk.
So, how I can use RAM is disk ? go through this steps.

Step 1) Create directory as shown below where you want to mount RAM as disk

root@arunbagul:~# mkdir -p /tmp/ram
root@arunbagul:~# ls /tmp/ram
root@arunbagul:~#

Step 2) mount RAM ie tmpfs at mount point “/tmp/ram” directory-

root@arunbagul:~#  mount -t tmpfs -o size=10M tmpfs /tmp/ram/

* check mounted on not ?

root@arunbagul:~# df -h
Filesystem            Size  Used Avail Use% Mounted on
tmpfs                  10M     0   10M   0% /tmp/ram

** to mount this permanently add entry in /etc/fstab as shown below.

root@arunbagul:~#  cat /etc/fstab

….

tmpfs        /tmp/ram    tmpfs    defaults,size=10m    0    0

root@arunbagul:~#

– To mount this use mount command (this will mount all mount points mentioned in fstab)

root@arunbagul:~# mount -a

Thank you,
Arun Bagul

hatimerun – provides a time-out mechanism for shell scripts

hatimerun – provides a time-out mechanism for shell scripts

Hello Everyone,
In day to day System Admin activities many times, you stuck to connect to any remote server, due to non-supporting timeout setting, here is the perfect solution for that ‘hatools’ – Thanks ‘MARKUS WINAND’, You can find this tool at http://www.fatalmind.com/software/hatools/

This tool will help you to manage your application and code with the specific timeout and lock, and you will be assure and go for sleep  😀

Download hatools from www.fatalmind.com (http://www.fatalmind.com/software/hatools/hatools-2.14.tar.bz2)

[root@testbed ~]# wget http://www.fatalmind.com/software/hatools/hatools-2.14.tar.bz2

[root@testbed ~]# tar xjvf hatools-2.14.tar.bz2 && cd hatools-2.14

The installation should be very seamless by just doing (Find the doc ‘README’)

[root@testbed hatools-2.14]# ./configure
[root@testbed hatools-2.14]#  make && make install

Now test the hatimerun command

[root@testbed ~]# hatimerun -h
usage: hatimerun [-a] [-e exitcode] [-k signame] -t secs command [args]
hatimerun [-l|-h|-?]
Options:
-a           Async mode. Starts hatimerun in the background
-e exitcode  Changes the exitcode returned by hatimerun on fail
-k signame   Specifies the signal witch will be sent to the process group
if a timeout occures
-t secs      Specifies the timeout in seconds
-l           Print list of available signals on this platform and exit.

Version:
V2.00
Copyright (c) 2001,2003,2005-2007 by Markus Winand <mws@fatalmind.com>
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.

[root@testbed ~]# hatimerun -k TERM -t 15 -t 1 ssh testbed140 uptime
04:36:23 up  3:24,  0 users,  load average: 0.51, 0.40, 0.31

Check with wrong or any ssh port blocked server and get timeout watch. 🙂

Here I am written small script for testing purpose, you can change and use as you want.

#!/bin/bash

# if you want to check and debug , run this script in following way
# bash -x script name <servername> <cmd>

server=$1  # server name or IP
cmd=$2   # command
sec=15      #Timeout seconds change as per your need

test -z $server && echo “server not found ” && echo “Usage: $0 {servername} {command}” && exit 1
test -z $cmd &&  echo “command not found ” && echo “Usage: $0 {servername} {command}” && exit 1
echo “connecting to $server, timewait set for connection is $sec Sec…….”

hatimerun -k TERM -t $sec -t 1 ssh $server $cmd
if [ $? != 0 ] ; then
echo “$server connection timeout …”
fi

 

 

-Ravi

Apache – (28)No space left on device: Cannot create SSLMutex

Apache – (28)No space left on device: Cannot create SSLMutex

Yesterday, I experienced very strange problem, that comes first time (for me), so checked things with apache and modules as diff way, but at the end got it fixed with the help of apache wiki 🙂  Thanks to make it perfect !!

There was something bad going with the apache and I was trying to fix it, as I need to restart it to first fix, but It can’t started, just popup message “[FAILED]”, surprised, ran twice to start it, then checked the error logs for debugging it and found the new error logged into messages.

[Thu Mar 10 00:58:59 2011] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Mar 10 00:58:59 2011] [error] (28)No space left on device: Cannot create SSLMutex
Configuration Failed

I found the page where all the related errors are listed by Apache http://wiki.apache.org/httpd/Logs , this helps me to fix this error.

“This error indicates that the server has run out of available slots for “inter-process communication“. This can occur when a process crashes before cleaning up after itself.The ipcs command will list semaphores and shared memory slots in use. If there is a large number of entries in use by the apache user use ipcrm to remove them. Then, check your server’s error log (possibly with LogLevel debug set) to figure out why it’s crashing, or see: http://httpd.apache.org/dev/debugging.html

[root@testbed ~]# ipcs -s |grep apacheuser
—— Semaphore Arrays ——–
key        semid      owner      perms      nsems
0x00000000 26836992   apacheuser 600        1
0x00000000 26902529   apacheuser 600        1
0x00000000 26935298   apacheuser 600        1
0x00000000 27361283   apacheuser 600        1
0x00000000 27426820   apacheuser 600        1
0x00000000 27459589   apacheuser 600        1
0x00000000 22937608   apacheuser 600        1
0x00000000 23003145   apacheuser 600        1
0x00000000 23035914   apacheuser 600        1
0x00000000 33456140   apacheuser 600        1
0x00000000 34308109   apacheuser 600        1
0x00000000 34340878   apacheuser 600        1
0x00000000 48791567   apacheuser 600        1
0x00000000 49053712   apacheuser 600        1
0x00000000 49086481   apacheuser 600        1

[root@testbed ~]# ipcs -s |grep apacheuser | awk ‘{print $2}’
26836992
26902529
26935298
27361283
27426820
27459589
22937608
23003145
23035914
33456140
34308109
34340878
48791567
49053712
49086481

[root@testbed ~]# ipcs -s |grep apacheuser | awk ‘{print $2}’| xargs ipcrm sem
resource(s) deleted

[root@testbed ~]# /etc/init.d/httpd start
Starting httpd:                                            [  OK  ]

Ref:
http://wiki.apache.org/httpd/Logs
http://wiki.apache.org/httpd/IPCExhaustion

-Ravi

Benchmarking for System and High Memory-CPU utilization – Part 1

Benchmarking for System and High Memory-CPU utilization – Part 1

Hello Guy’s,

Every System Admin and technical person who is working or doing any kind of High Availability stuff, He really needs bench marking to check and verify system health under development, here I am very beginner in this kind of system benchmarking to know how server/CPU/memory and network reacts when doing this and given load (purposely) on system to use or run the service or process. I will explain how I have done this in the part -2 😉

Memory Util

 

CPU Util

 

-Ravi

Yum rollback and repackage

Yum rollback and repackage

Introduction –
Fedora’s package management tools — including yum, pup, and pirut  — are all based on the RPM package format and management system. One little-known secret about RPM is that it can be configured to repackage  files from an RPM package during package uninstallation, saving the (possibly modified) files into a new RPM package. The repackaged RPM incorporates any changes that you have made to the configuration files, scripts, and data files that were originally included with the software. This means that it’s possible to rollback the uninstallation of software, which will restore the package to the state it was in before it was removed.

The rollback mechanism can also undo package installations by uninstalling the newly-installed packages, and since a software update is a performed by installing a new package version and then removing the old one, the rollback mechanism can also undo package updates.

Open /etc/yum.conf file and put below two line

#yum rollback select enable
tsflags=repackage

or append it a simple way 🙂

[root@testbed ~]# echo “tsflags=repackage” >> /etc/yum.conf

Now create or edit /etc/rpm/macros and put “%_repackage_all_erasures 1” entry in it.

[root@testbed ~]# echo “%_repackage_all_erasures 1” >> /etc/rpm/macros

[root@testbed ~]# cat /etc/rpm/macros
%_repackage_all_erasures 1

Repackaged files are stored in /var/spool/repackage

[root@testbed ~]# ll -ld /var/spool/repackage
drwxr-xr-x 2 root root 4096 Sep  4  2009 /var/spool/repackage

Now we are test it with python packages, (I have used centos 5.5 for it)

[root@testbed ~]# yum -y install python*

Installed:
python-dmidecode.i386 0:3.10.13-1.el5_5.1           python-docs.noarch 0:2.4.3-1.1                    python-exo.i386 0:0.3.4-1.el5.centos
python-imaging.i386 0:1.1.5-5.el5                   python-imaging-devel.i386 0:1.1.5-5.el5           python-lcms.i386 0:1.18-0.1.beta1.el5_3.2
python-pyblock.i386 0:0.26-4.el5                    python-setuptools.noarch 0:0.6c5-2.el5            python-tools.i386 0:2.4.3-27.el5_5.3

Dependency Installed:
exo.i386 0:0.3.4-1.el5.centos                     libbdevid-python.i386 0:5.1.19.6-61.el5_5.2            libxfce4mcs.i386 0:4.4.2-1.el5.centos
libxfce4util.i386 0:4.4.2-1.el5.centos            libxfcegui4.i386 0:4.4.2-1.el5.centos                  tix.i386 1:8.4.0-11.fc6
tkinter.i386 0:2.4.3-27.el5_5.3

Updated:
python.i386 0:2.4.3-27.el5_5.3                python-devel.i386 0:2.4.3-27.el5_5.3                python-virtinst.noarch 0:0.400.3-9.el5_5.1

Dependency Updated:
mkinitrd.i386 0:5.1.19.6-61.el5_5.2                                             nash.i386 0:5.1.19.6-61.el5_5.2

Complete!
[root@testbed ~]# ls /var/spool/repackage/
mkinitrd-5.1.19.6-61.i386.rpm  python-2.4.3-27.el5.i386.rpm        python-virtinst-0.400.3-9.el5.noarch.rpm
nash-5.1.19.6-61.i386.rpm      python-devel-2.4.3-27.el5.i386.rpm

Yum Roll Back options available:

* rpm -Uhv –rollback ‘9:00 am’
* rpm -Uhv –rollback ‘4 hours ago’
* rpm -Uhv –rollback ‘december 25’

[root@testbed ~]# date
Mon Jan 24 12:10:19 IST 2011

Now we reverting package rollback to a previous state.

[root@testbed ~]# rpm -Uhv –rollback ‘9:00 am’
Rollback packages (+5/-21) to Mon Jan 24 11:58:17 2011 (0x4d3d1c01):
Preparing…                ########################################### [100%]
1:nash                   ########################################### [ 10%]
2:python                 ########################################### [ 20%]
3:mkinitrd               ########################################### [ 30%]
4:python-devel           ########################################### [ 40%]
5:python-virtinst        ########################################### [ 50%]
Cleaning up repackaged packages:
Removing /var/spool/repackage/mkinitrd-5.1.19.6-61.i386.rpm:
Removing /var/spool/repackage/nash-5.1.19.6-61.i386.rpm:
Removing /var/spool/repackage/python-2.4.3-27.el5.i386.rpm:
Removing /var/spool/repackage/python-devel-2.4.3-27.el5.i386.rpm:
Removing /var/spool/repackage/python-virtinst-0.400.3-9.el5.noarch.rpm:

Now you can check your previous versions of python & mkinitrd, nash packages.

[root@testbed ~]# rpm -qa|grep -E ‘python|mkinitrd|nash’

The repackage/rollback approach is far from perfect — for example, data files created and used with a package (but not in files provided as part of the package) are not saved during repackaging, and some RPM scripts assume that packages are only upgraded and never downgraded. Nonetheless, package rollback can be a very useful feature, especially when an update breaks something that used to work.
Repackaging can take a lot of space, so it’s disabled by default, and there is no way to enable it or to perform a rollback from the command line. Here, in a nutshell, are instructions for using this feature:

Ref: http://dailypackage.fedorabook.com/index.php?/archives/17-Wednesday-Why-Repackaging-and-Rollbacks.html

-Ravi

oprofile a system wide profiler

oprofile a system wide profiler

** What is oprofile

In short “oprofile is a system-wide profiler”
need to profile an application and its shared libraries, examine hardware effects such as cache misses and capture the performance behaviour of entire system, then surely you need go with oprofile.

There is to many options and conditions spcefied, where we use only simple and easy one for further details, Please Read Man Pages 😉

Many CPUs provide “performance counters”, hardware registers that can count “events”; for example, cache misses, or CPU cycles. OProfile provides profiles of code based on the number of these occurring events: repeatedly, every time a certain (configurable) number of events has occurred, the PC value is recorded. This information is aggregated into profiles for each binary image.

Some hardware setups do not allow OProfile to use performance counters: in these cases, no events are available, and OProfile operates in timer/RTC mode

** All Docs, Examples and Bugs you will find @ http://oprofile.sourceforge.net

# download tar from http://oprofile.sourceforge.net/download/ or you can install it using yum also

[root@ravi.com ~]# yum install oprofile

Before you can use OProfile, you must set it up. The minimum setup required for this is to tell OProfile where the vmlinux file corresponding to the running kernel is, for example :

[root@ravi.com ~]#  opcontrol –vmlinux=/boot/vmlinux-`uname -r`

If you don’t want to profile the kernel itself, you can tell OProfile you don’t have a vmlinux file :

[root@ravi.com ~]# opcontrol –no-vmlinux

Here we used –no-vmlinux and specified other session directory than default location (/var/lib/oprofile)

[root@ravi.com ~]# opcontrol –no-vmlinux –session-dir=/home/prod/tmpsession

Now we are ready to start the daemon (oprofiled) which collects the profile data :

[root@ravi.com ~]# opcontrol –start –session-dir=/home/prod/tmpsession
Using default event: GLOBAL_POWER_EVENTS:100000:1:1:1
Using 2.6+ OProfile kernel interface.
Using log file /home/prod/tmpsession/samples/oprofiled.log
Daemon started.
Profiler running.

When I want to stop profiling, I can do so with :
(Here I have ran opcontrol only 1 min)
[root@ravi.com ~]# opcontrol –shutdown
Stopping profiling.
Killing daemon.

Lets see what your system & apps tell you 🙂
ophelp : This utility lists the available events and short descriptions.

[root@ravi.com ~]# opreport –session-dir=/home/prod/tmpsession
CPU: P4 / Xeon with 2 hyper-threads, speed 2992.73 MHz (estimated)
Counted GLOBAL_POWER_EVENTS events (time during which processor is not stopped) with a unit mask of 0x01 (mandatory) count 100000
GLOBAL_POWER_E…|
samples|      %|
——————
1109031 51.2560 no-vmlinux
479089 22.1420 libc-2.5.so
207263  9.5791 libperl.so
97973  4.5280 libpython2.4.so.1.0
91993  4.2516 nagios
35979  1.6628 php
35765  1.6529 libz.so.1.2.3
25203  1.1648 ld-2.5.so
21412  0.9896 mysqld
11566  0.5345 libgd.so.2.0.0
10008  0.4625 oprofiled
7661  0.3541 libpthread-2.5.so
6736  0.3113 libnetsnmp.so.10.0.3
5698  0.2633 libpng12.so.0.10.0
4488  0.2074 rateup
3500  0.1618 libcrypto.so.0.9.8e
2138  0.0988 bash
1556  0.0719 libm-2.5.so
1234  0.0570 libmysqlclient.so.15.0.0
606  0.0280 libpcre.so.0.0.1
328  0.0152 mysql.so
316  0.0146 librrd.so.4.1.3
290  0.0134 atop
272  0.0126 sendmail.sendmail
243  0.0112 grep
176  0.0081 nscd
172  0.0079 ping
153  0.0071 nebmodBY0Y2I (deleted)
144  0.0067 libgnutls.so.13.0.6
121  0.0056 init
106  0.0049 gawk

with using symbols you can simply find which apps supporting module takes high load
[root@ravi.com ~]#  opreport –exclude-dependent –symbols –long-filenames –session-dir=/home/prod/tmpsession
CPU: P4 / Xeon with 2 hyper-threads, speed 2992.73 MHz (estimated)
Counted GLOBAL_POWER_EVENTS events (time during which processor is not stopped) with a unit mask of 0x01 (mandatory) count 100000
samples  %        app name                 symbol name
1109031  51.2560  /no-vmlinux              /no-vmlinux
209506    9.6827  /lib64/libc-2.5.so       _int_free
207263    9.5791  /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
97973     4.5280  /usr/lib64/libpython2.4.so.1.0 /usr/lib64/libpython2.4.so.1.0
91993     4.2516  /usr/local/nagios/bin/nagios /usr/local/nagios/bin/nagios
44574     2.0601  /lib64/libc-2.5.so       free
35765     1.6529  /usr/lib64/libz.so.1.2.3 /usr/lib64/libz.so.1.2.3
35032     1.6191  /lib64/libc-2.5.so       vfprintf
22061     1.0196  /lib64/libc-2.5.so       _IO_vfscanf
21412     0.9896  /usr/libexec/mysqld      /usr/libexec/mysqld
16883     0.7803  /lib64/libc-2.5.so       _int_malloc
13631     0.6300  /lib64/libc-2.5.so       memcpy
13165     0.6084  /lib64/libc-2.5.so       _IO_file_xsputn@@GLIBC_2.2.5
1560      0.0721  /lib64/ld-2.5.so         _dl_relocate_object
1462      0.0676  /usr/bin/php             lstep
1385      0.0640  /usr/bin/php             zend_do_fcall_common_helper_SPEC
1258      0.0581  /lib64/libc-2.5.so       rawmemchr
1234      0.0570  /usr/lib64/mysql/libmysqlclient.so.15.0.0 /usr/lib64/mysql/libmysqlclient.so.15.0.0
1226      0.0567  /lib64/libc-2.5.so       realloc
1180      0.0545  /lib64/libc-2.5.so       __tzfile_compute
1156      0.0534  /lib64/ld-2.5.so         check_match.8509
1146      0.0530  /usr/bin/php             _zend_mm_free_int
1084      0.0501  /usr/bin/php             zend_hash_quick_find
1029      0.0476  /lib64/libc-2.5.so       __mpn_mul_1
747       0.0345  /lib64/libc-2.5.so       fgets
738       0.0341  /usr/bin/php             _zval_ptr_dtor

Ref: http://oprofile.sourceforge.net/doc/index.html

-Ravi