Author: Ravi Bhure

Shellshock ‘Deadly serious’ new tech bug found!

Shellshock ‘Deadly serious’ new tech bug found!

This issue affects all products which use the Bash shell and parse values of environment variables. This issue is especially dangerous as there are many possible ways Bash can be called by an application. Quite often if an application executes another binary, Bash is invoked to accomplish this. Because of the pervasive use of the Bash shell, this issue is quite serious and should be treated as such.

All versions prior to those listed as updates for this issue are vulnerable to some degree.

My infected OS version is CentOS-6 and bash version 4.1.2

[root@host75 ~]# lsb_release -a
lsb_release -a
LSB Version: :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-noarch
Distributor ID: CentOS
Description: CentOS release 6.4 (Final)
Release: 6.4
Codename: Final

[root@host75 ~]# bash --version
bash --version
GNU bash, version 4.1.2(1)-release (x86_64-redhat-linux-gnu)
Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Diagnostic Steps:

To test if your version of Bash is vulnerable to this issue, run the following command:

$ env x='() { :;}; echo vulnerable' bash -c "echo this is a test"

If the output of the above command looks as follows:

vulnerable
this is a test

hmm, I got infected!

[root@host75 ~]# env x='() { :;}; echo vulnerable' bash -c "echo this is a test"
vulnerable
this is a test

You are using a vulnerable version of Bash. The patch used to fix this issue ensures that no code is allowed after the end of a Bash function. Thus, if you run the above example with the patched version of Bash, you should get an output similar to:

$ env x='() { :;}; echo vulnerable' bash -c "echo this is a test"
bash: warning: x: ignoring function definition attempt
bash: error importing function definition for `x'
this is a test

If your system is vulnerable, update to the most recent version of the Bash package by running the following command:

$yum update bash

This fix my bash ??????

[root@host75 ~]# yum update bash
Loaded plugins: fastestmirror, security, tmprepo
Loading mirror speeds from cached hostfile
epel/metalink | 15 kB 00:00
* base: centos.eecs.wsu.edu
* epel: mirrors.kernel.org
* extras: centos.chi.host-engine.com
* updates: mirror.raystedman.net
base | 3.7 kB 00:00
epel | 4.4 kB 00:00
epel/primary_db | 6.3 MB 00:05
extras | 3.3 kB 00:00
updates | 3.4 kB 00:00
updates/primary_db | 5.3 MB 00:04
Setting up Update Process
Resolving Dependencies
--> Running transaction check
---> Package bash.x86_64 0:4.1.2-14.el6 will be updated
---> Package bash.x86_64 0:4.1.2-15.el6_5.1 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================================================================================
Package Arch Version Repository Size
================================================================================================================================================
Updating:
bash x86_64 4.1.2-15.el6_5.1 updates 905 k

Transaction Summary
================================================================================================================================================
Upgrade 1 Package(s)

Total download size: 905 k
Is this ok [y/N]: y
Downloading Packages:
bash-4.1.2-15.el6_5.1.x86_64.rpm | 905 kB 00:00
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Updating : bash-4.1.2-15.el6_5.1.x86_64 1/2
Cleanup : bash-4.1.2-14.el6.x86_64 2/2
Verifying : bash-4.1.2-15.el6_5.1.x86_64 1/2
Verifying : bash-4.1.2-14.el6.x86_64 2/2

Updated:
bash.x86_64 0:4.1.2-15.el6_5.1

Complete!

Test if update fixed to patch your bash

[root@host75 ~]# env x='() { :;}; echo vulnerable' bash -c "echo this is a test"
bash: warning: x: ignoring function definition attempt
bash: error importing function definition for `x'
this is a test

How does this impact systems:
This issue affects all products which use the Bash shell and parse values of environment variables. This issue is especially dangerous as there are many possible ways Bash can be called by an application. Quite often if an application executes another binary, Bash is invoked to accomplish this. Because of the pervasive use of the Bash shell, this issue is quite serious and should be treated as such.

All versions prior to those listed as updates for this issue are vulnerable to some degree.

See the appropriate remediation article for specifics.

Functions written in Bash itself do not need to be changed, even if they are exported with “export -f”. Bash will transparently apply the appropriate naming when exporting, and reverse the process when importing function definitions.

Ref:
http://www.bbc.com/news/technology-29361794
https://www.us-cert.gov/ncas/current-activity/2014/09/24/Bourne-Again-Shell-Bash-Remote-Code-Execution-Vulnerability
https://access.redhat.com/articles/1200223
https://securityblog.redhat.com/2014/09/24/bash-specially-crafted-environment-variables-code-injection-attack/
http://lists.gnu.org/archive/html/bug-bash/2014-09/threads.html
https://rhn.redhat.com/errata/RHSA-2014-1306.html

hatimerun – provides a time-out mechanism for shell scripts

hatimerun – provides a time-out mechanism for shell scripts

Hello Everyone,
In day to day System Admin activities many times, you stuck to connect to any remote server, due to non-supporting timeout setting, here is the perfect solution for that ‘hatools’ – Thanks ‘MARKUS WINAND’, You can find this tool at http://www.fatalmind.com/software/hatools/

This tool will help you to manage your application and code with the specific timeout and lock, and you will be assure and go for sleep  😀

Download hatools from www.fatalmind.com (http://www.fatalmind.com/software/hatools/hatools-2.14.tar.bz2)

[root@testbed ~]# wget http://www.fatalmind.com/software/hatools/hatools-2.14.tar.bz2

[root@testbed ~]# tar xjvf hatools-2.14.tar.bz2 && cd hatools-2.14

The installation should be very seamless by just doing (Find the doc ‘README’)

[root@testbed hatools-2.14]# ./configure
[root@testbed hatools-2.14]#  make && make install

Now test the hatimerun command

[root@testbed ~]# hatimerun -h
usage: hatimerun [-a] [-e exitcode] [-k signame] -t secs command [args]
hatimerun [-l|-h|-?]
Options:
-a           Async mode. Starts hatimerun in the background
-e exitcode  Changes the exitcode returned by hatimerun on fail
-k signame   Specifies the signal witch will be sent to the process group
if a timeout occures
-t secs      Specifies the timeout in seconds
-l           Print list of available signals on this platform and exit.

Version:
V2.00
Copyright (c) 2001,2003,2005-2007 by Markus Winand <mws@fatalmind.com>
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.

[root@testbed ~]# hatimerun -k TERM -t 15 -t 1 ssh testbed140 uptime
04:36:23 up  3:24,  0 users,  load average: 0.51, 0.40, 0.31

Check with wrong or any ssh port blocked server and get timeout watch. 🙂

Here I am written small script for testing purpose, you can change and use as you want.

#!/bin/bash

# if you want to check and debug , run this script in following way
# bash -x script name <servername> <cmd>

server=$1  # server name or IP
cmd=$2   # command
sec=15      #Timeout seconds change as per your need

test -z $server && echo “server not found ” && echo “Usage: $0 {servername} {command}” && exit 1
test -z $cmd &&  echo “command not found ” && echo “Usage: $0 {servername} {command}” && exit 1
echo “connecting to $server, timewait set for connection is $sec Sec…….”

hatimerun -k TERM -t $sec -t 1 ssh $server $cmd
if [ $? != 0 ] ; then
echo “$server connection timeout …”
fi

 

 

-Ravi

Apache – (28)No space left on device: Cannot create SSLMutex

Apache – (28)No space left on device: Cannot create SSLMutex

Yesterday, I experienced very strange problem, that comes first time (for me), so checked things with apache and modules as diff way, but at the end got it fixed with the help of apache wiki 🙂  Thanks to make it perfect !!

There was something bad going with the apache and I was trying to fix it, as I need to restart it to first fix, but It can’t started, just popup message “[FAILED]”, surprised, ran twice to start it, then checked the error logs for debugging it and found the new error logged into messages.

[Thu Mar 10 00:58:59 2011] [notice] suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu Mar 10 00:58:59 2011] [error] (28)No space left on device: Cannot create SSLMutex
Configuration Failed

I found the page where all the related errors are listed by Apache http://wiki.apache.org/httpd/Logs , this helps me to fix this error.

“This error indicates that the server has run out of available slots for “inter-process communication“. This can occur when a process crashes before cleaning up after itself.The ipcs command will list semaphores and shared memory slots in use. If there is a large number of entries in use by the apache user use ipcrm to remove them. Then, check your server’s error log (possibly with LogLevel debug set) to figure out why it’s crashing, or see: http://httpd.apache.org/dev/debugging.html

[root@testbed ~]# ipcs -s |grep apacheuser
—— Semaphore Arrays ——–
key        semid      owner      perms      nsems
0x00000000 26836992   apacheuser 600        1
0x00000000 26902529   apacheuser 600        1
0x00000000 26935298   apacheuser 600        1
0x00000000 27361283   apacheuser 600        1
0x00000000 27426820   apacheuser 600        1
0x00000000 27459589   apacheuser 600        1
0x00000000 22937608   apacheuser 600        1
0x00000000 23003145   apacheuser 600        1
0x00000000 23035914   apacheuser 600        1
0x00000000 33456140   apacheuser 600        1
0x00000000 34308109   apacheuser 600        1
0x00000000 34340878   apacheuser 600        1
0x00000000 48791567   apacheuser 600        1
0x00000000 49053712   apacheuser 600        1
0x00000000 49086481   apacheuser 600        1

[root@testbed ~]# ipcs -s |grep apacheuser | awk ‘{print $2}’
26836992
26902529
26935298
27361283
27426820
27459589
22937608
23003145
23035914
33456140
34308109
34340878
48791567
49053712
49086481

[root@testbed ~]# ipcs -s |grep apacheuser | awk ‘{print $2}’| xargs ipcrm sem
resource(s) deleted

[root@testbed ~]# /etc/init.d/httpd start
Starting httpd:                                            [  OK  ]

Ref:
http://wiki.apache.org/httpd/Logs
http://wiki.apache.org/httpd/IPCExhaustion

-Ravi

Benchmarking for System and High Memory-CPU utilization – Part 1

Benchmarking for System and High Memory-CPU utilization – Part 1

Hello Guy’s,

Every System Admin and technical person who is working or doing any kind of High Availability stuff, He really needs bench marking to check and verify system health under development, here I am very beginner in this kind of system benchmarking to know how server/CPU/memory and network reacts when doing this and given load (purposely) on system to use or run the service or process. I will explain how I have done this in the part -2 😉

Memory Util

 

CPU Util

 

-Ravi

Yum rollback and repackage

Yum rollback and repackage

Introduction –
Fedora’s package management tools — including yum, pup, and pirut  — are all based on the RPM package format and management system. One little-known secret about RPM is that it can be configured to repackage  files from an RPM package during package uninstallation, saving the (possibly modified) files into a new RPM package. The repackaged RPM incorporates any changes that you have made to the configuration files, scripts, and data files that were originally included with the software. This means that it’s possible to rollback the uninstallation of software, which will restore the package to the state it was in before it was removed.

The rollback mechanism can also undo package installations by uninstalling the newly-installed packages, and since a software update is a performed by installing a new package version and then removing the old one, the rollback mechanism can also undo package updates.

Open /etc/yum.conf file and put below two line

#yum rollback select enable
tsflags=repackage

or append it a simple way 🙂

[root@testbed ~]# echo “tsflags=repackage” >> /etc/yum.conf

Now create or edit /etc/rpm/macros and put “%_repackage_all_erasures 1” entry in it.

[root@testbed ~]# echo “%_repackage_all_erasures 1” >> /etc/rpm/macros

[root@testbed ~]# cat /etc/rpm/macros
%_repackage_all_erasures 1

Repackaged files are stored in /var/spool/repackage

[root@testbed ~]# ll -ld /var/spool/repackage
drwxr-xr-x 2 root root 4096 Sep  4  2009 /var/spool/repackage

Now we are test it with python packages, (I have used centos 5.5 for it)

[root@testbed ~]# yum -y install python*

Installed:
python-dmidecode.i386 0:3.10.13-1.el5_5.1           python-docs.noarch 0:2.4.3-1.1                    python-exo.i386 0:0.3.4-1.el5.centos
python-imaging.i386 0:1.1.5-5.el5                   python-imaging-devel.i386 0:1.1.5-5.el5           python-lcms.i386 0:1.18-0.1.beta1.el5_3.2
python-pyblock.i386 0:0.26-4.el5                    python-setuptools.noarch 0:0.6c5-2.el5            python-tools.i386 0:2.4.3-27.el5_5.3

Dependency Installed:
exo.i386 0:0.3.4-1.el5.centos                     libbdevid-python.i386 0:5.1.19.6-61.el5_5.2            libxfce4mcs.i386 0:4.4.2-1.el5.centos
libxfce4util.i386 0:4.4.2-1.el5.centos            libxfcegui4.i386 0:4.4.2-1.el5.centos                  tix.i386 1:8.4.0-11.fc6
tkinter.i386 0:2.4.3-27.el5_5.3

Updated:
python.i386 0:2.4.3-27.el5_5.3                python-devel.i386 0:2.4.3-27.el5_5.3                python-virtinst.noarch 0:0.400.3-9.el5_5.1

Dependency Updated:
mkinitrd.i386 0:5.1.19.6-61.el5_5.2                                             nash.i386 0:5.1.19.6-61.el5_5.2

Complete!
[root@testbed ~]# ls /var/spool/repackage/
mkinitrd-5.1.19.6-61.i386.rpm  python-2.4.3-27.el5.i386.rpm        python-virtinst-0.400.3-9.el5.noarch.rpm
nash-5.1.19.6-61.i386.rpm      python-devel-2.4.3-27.el5.i386.rpm

Yum Roll Back options available:

* rpm -Uhv –rollback ‘9:00 am’
* rpm -Uhv –rollback ‘4 hours ago’
* rpm -Uhv –rollback ‘december 25’

[root@testbed ~]# date
Mon Jan 24 12:10:19 IST 2011

Now we reverting package rollback to a previous state.

[root@testbed ~]# rpm -Uhv –rollback ‘9:00 am’
Rollback packages (+5/-21) to Mon Jan 24 11:58:17 2011 (0x4d3d1c01):
Preparing…                ########################################### [100%]
1:nash                   ########################################### [ 10%]
2:python                 ########################################### [ 20%]
3:mkinitrd               ########################################### [ 30%]
4:python-devel           ########################################### [ 40%]
5:python-virtinst        ########################################### [ 50%]
Cleaning up repackaged packages:
Removing /var/spool/repackage/mkinitrd-5.1.19.6-61.i386.rpm:
Removing /var/spool/repackage/nash-5.1.19.6-61.i386.rpm:
Removing /var/spool/repackage/python-2.4.3-27.el5.i386.rpm:
Removing /var/spool/repackage/python-devel-2.4.3-27.el5.i386.rpm:
Removing /var/spool/repackage/python-virtinst-0.400.3-9.el5.noarch.rpm:

Now you can check your previous versions of python & mkinitrd, nash packages.

[root@testbed ~]# rpm -qa|grep -E ‘python|mkinitrd|nash’

The repackage/rollback approach is far from perfect — for example, data files created and used with a package (but not in files provided as part of the package) are not saved during repackaging, and some RPM scripts assume that packages are only upgraded and never downgraded. Nonetheless, package rollback can be a very useful feature, especially when an update breaks something that used to work.
Repackaging can take a lot of space, so it’s disabled by default, and there is no way to enable it or to perform a rollback from the command line. Here, in a nutshell, are instructions for using this feature:

Ref: http://dailypackage.fedorabook.com/index.php?/archives/17-Wednesday-Why-Repackaging-and-Rollbacks.html

-Ravi

oprofile a system wide profiler

oprofile a system wide profiler

** What is oprofile

In short “oprofile is a system-wide profiler”
need to profile an application and its shared libraries, examine hardware effects such as cache misses and capture the performance behaviour of entire system, then surely you need go with oprofile.

There is to many options and conditions spcefied, where we use only simple and easy one for further details, Please Read Man Pages 😉

Many CPUs provide “performance counters”, hardware registers that can count “events”; for example, cache misses, or CPU cycles. OProfile provides profiles of code based on the number of these occurring events: repeatedly, every time a certain (configurable) number of events has occurred, the PC value is recorded. This information is aggregated into profiles for each binary image.

Some hardware setups do not allow OProfile to use performance counters: in these cases, no events are available, and OProfile operates in timer/RTC mode

** All Docs, Examples and Bugs you will find @ http://oprofile.sourceforge.net

# download tar from http://oprofile.sourceforge.net/download/ or you can install it using yum also

[root@ravi.com ~]# yum install oprofile

Before you can use OProfile, you must set it up. The minimum setup required for this is to tell OProfile where the vmlinux file corresponding to the running kernel is, for example :

[root@ravi.com ~]#  opcontrol –vmlinux=/boot/vmlinux-`uname -r`

If you don’t want to profile the kernel itself, you can tell OProfile you don’t have a vmlinux file :

[root@ravi.com ~]# opcontrol –no-vmlinux

Here we used –no-vmlinux and specified other session directory than default location (/var/lib/oprofile)

[root@ravi.com ~]# opcontrol –no-vmlinux –session-dir=/home/prod/tmpsession

Now we are ready to start the daemon (oprofiled) which collects the profile data :

[root@ravi.com ~]# opcontrol –start –session-dir=/home/prod/tmpsession
Using default event: GLOBAL_POWER_EVENTS:100000:1:1:1
Using 2.6+ OProfile kernel interface.
Using log file /home/prod/tmpsession/samples/oprofiled.log
Daemon started.
Profiler running.

When I want to stop profiling, I can do so with :
(Here I have ran opcontrol only 1 min)
[root@ravi.com ~]# opcontrol –shutdown
Stopping profiling.
Killing daemon.

Lets see what your system & apps tell you 🙂
ophelp : This utility lists the available events and short descriptions.

[root@ravi.com ~]# opreport –session-dir=/home/prod/tmpsession
CPU: P4 / Xeon with 2 hyper-threads, speed 2992.73 MHz (estimated)
Counted GLOBAL_POWER_EVENTS events (time during which processor is not stopped) with a unit mask of 0x01 (mandatory) count 100000
GLOBAL_POWER_E…|
samples|      %|
——————
1109031 51.2560 no-vmlinux
479089 22.1420 libc-2.5.so
207263  9.5791 libperl.so
97973  4.5280 libpython2.4.so.1.0
91993  4.2516 nagios
35979  1.6628 php
35765  1.6529 libz.so.1.2.3
25203  1.1648 ld-2.5.so
21412  0.9896 mysqld
11566  0.5345 libgd.so.2.0.0
10008  0.4625 oprofiled
7661  0.3541 libpthread-2.5.so
6736  0.3113 libnetsnmp.so.10.0.3
5698  0.2633 libpng12.so.0.10.0
4488  0.2074 rateup
3500  0.1618 libcrypto.so.0.9.8e
2138  0.0988 bash
1556  0.0719 libm-2.5.so
1234  0.0570 libmysqlclient.so.15.0.0
606  0.0280 libpcre.so.0.0.1
328  0.0152 mysql.so
316  0.0146 librrd.so.4.1.3
290  0.0134 atop
272  0.0126 sendmail.sendmail
243  0.0112 grep
176  0.0081 nscd
172  0.0079 ping
153  0.0071 nebmodBY0Y2I (deleted)
144  0.0067 libgnutls.so.13.0.6
121  0.0056 init
106  0.0049 gawk

with using symbols you can simply find which apps supporting module takes high load
[root@ravi.com ~]#  opreport –exclude-dependent –symbols –long-filenames –session-dir=/home/prod/tmpsession
CPU: P4 / Xeon with 2 hyper-threads, speed 2992.73 MHz (estimated)
Counted GLOBAL_POWER_EVENTS events (time during which processor is not stopped) with a unit mask of 0x01 (mandatory) count 100000
samples  %        app name                 symbol name
1109031  51.2560  /no-vmlinux              /no-vmlinux
209506    9.6827  /lib64/libc-2.5.so       _int_free
207263    9.5791  /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so /usr/lib64/perl5/5.8.8/x86_64-linux-thread-multi/CORE/libperl.so
97973     4.5280  /usr/lib64/libpython2.4.so.1.0 /usr/lib64/libpython2.4.so.1.0
91993     4.2516  /usr/local/nagios/bin/nagios /usr/local/nagios/bin/nagios
44574     2.0601  /lib64/libc-2.5.so       free
35765     1.6529  /usr/lib64/libz.so.1.2.3 /usr/lib64/libz.so.1.2.3
35032     1.6191  /lib64/libc-2.5.so       vfprintf
22061     1.0196  /lib64/libc-2.5.so       _IO_vfscanf
21412     0.9896  /usr/libexec/mysqld      /usr/libexec/mysqld
16883     0.7803  /lib64/libc-2.5.so       _int_malloc
13631     0.6300  /lib64/libc-2.5.so       memcpy
13165     0.6084  /lib64/libc-2.5.so       _IO_file_xsputn@@GLIBC_2.2.5
1560      0.0721  /lib64/ld-2.5.so         _dl_relocate_object
1462      0.0676  /usr/bin/php             lstep
1385      0.0640  /usr/bin/php             zend_do_fcall_common_helper_SPEC
1258      0.0581  /lib64/libc-2.5.so       rawmemchr
1234      0.0570  /usr/lib64/mysql/libmysqlclient.so.15.0.0 /usr/lib64/mysql/libmysqlclient.so.15.0.0
1226      0.0567  /lib64/libc-2.5.so       realloc
1180      0.0545  /lib64/libc-2.5.so       __tzfile_compute
1156      0.0534  /lib64/ld-2.5.so         check_match.8509
1146      0.0530  /usr/bin/php             _zend_mm_free_int
1084      0.0501  /usr/bin/php             zend_hash_quick_find
1029      0.0476  /lib64/libc-2.5.so       __mpn_mul_1
747       0.0345  /lib64/libc-2.5.so       fgets
738       0.0341  /usr/bin/php             _zval_ptr_dtor

Ref: http://oprofile.sourceforge.net/doc/index.html

-Ravi

Adding and Removing swap on Linux

Adding and Removing swap on Linux

Introduction –

Here we are going to create 4G swap file using cmd ‘dd’, later we will activate it.
we are named and create ‘/extraswap’ swap file for additional swap

dd command options
if = input file (read from FILE instead of stdin)
of = output file name (write to FILE instead of stdout)
bs = BYTES rate
count = BLOCKS (copy only number of block)

dd if=/dev/zero of=/extraswap bs=1024 count=4096000

***Adding extraswap Swap file

[root@ravi ~]# dd if=/dev/zero of=/extraswap bs=1024 count=4096000
4096000+0 records in
4096000+0 records out
4194304000 bytes (4.2 GB) copied, 162.277 seconds, 25.8 MB/s

Set up a Linux swap area using ‘mkswap’ command

[root@ravi ~]# mkswap /extraswap
Setting up swapspace version 1, size = 4194299 kB

Activate swap using ‘swapon’ command

[root@ravi ~]# swapon /extraswap

[root@ravi ~]# free -m
total       used       free     shared    buffers     cached
Mem:         16053       6148       9905          0        293       5488
-/+ buffers/cache:        367      15686
Swap:         3999          0       3999

Edit /etc/fstab and put below entry into it to swap on automatic after reboot server

/extraswap              none                    swap    defaults        0 0

*** Remove /extraswap file

[root@ravi ~]# swapoff /extraswap

Remove /extraswap swap file entry from /etc/fstab and run ‘mount -a

Delete /extraswap

[root@ravi ~]# rm /extraswap


-Ravi

How to use socat with haproxy stat

How to use socat with haproxy stat

*** Introduction –

All you know about the haproxy, that its the one of the good opensource load balancing software and to check the fun stats of haproxy here we using ‘socat’ – Multipurpose relay (SOcket CAT)


* What is socat?

Socat  is  a  command  line based utility that establishes two bidirectional byte streams and transfers data between them. Because the streams can be constructed from a large set of different types of data sinks and sources (see address  types),  and  because  lots  of address options may be applied to the streams, socat can be used for many different purposes. (see more info at ‘man socat’ 🙂 or at http://www.dest-unreach.org/socat/)
* How to use ‘socat’ with haproxy stat

Step 1) Download ‘socat’ from http://www.dest-unreach.org/socat/download/  latest version ~ “socat-2.0.0-b3.tar.gz”

ravi@arun:~$ wget http://www.dest-unreach.org/socat/download/socat-1.7.1.2.tar.gz

ravi@arun:~$ tar xvzf socat-1.7.1.2.tar.gz

ravi@arun:~$ cd socat-1.7.1.2

NOTE ~ No need to install the ‘fipsld’ package if you got the below msg after running the ‘make’ just following steps for

compiling socat….

FIPSLD_CC=gcc fipsld -O -D_GNU_SOURCE -Wall -Wno-parentheses  -DHAVE_CONFIG_H -I.  -I.   -c -o socat.o socat.c
/bin/sh: fipsld: command not found
make: *** [socat.o] Error 127

ravi@arun:~$ ./configure –disable-fips
ravi@arun:~$ make

To install it login as root
ravi@arun:~$ su –

ravi@arun:~# make install

Step 2) Now you need to add stats socket PATH in Haproxy configuration and restart haproxy as per shown in following example,

where I have added it under in ‘global’ setting –

ravi@arun:~# more /etc/haproxy/myhaproxy.cfg

#———–Start of haproxy Config file————–
global
log 127.0.0.1   local0
log 127.0.0.1   local1 notice
#log loghost    local0 info
maxconn 25000
#debug
#quiet
user ravi
group ravi
stats socket    /tmp/haproxy
defaults
option          contstats
timeout         connect 5s
timeout         client 25s
timeout         server 25s
maxconn         100

listen ravitestbed      0.0.0.0:80 ##ravi.com IP
mode            tcp
balance         roundrobin
server          web1 192.168.19.117
server          web2 192.168.19.122

listen stats
bind            0.0.0.0:8081
mode            http
#stats          uri /stat  #Comment this if you need to specify diff stat path for viewing stat page
stats enable
stats auth admin:admin ##Auth user pass

#———–End of haproxy Config file————–

Step 3) Used /tmp/haproxy. Now you can send the commands to get stats from HAProxy –

Now time to use socat

ravi@arun:~# echo “”  | socat unix-connect:/tmp/haproxy stdio
Unknown command. Please enter one of the following commands only :
show info   : report information about the running process
show stat   : report counters for each proxy and server
show errors : report last request and response errors for each proxy
show sess   : report the list of current sessions

This will dump (possibly huge) info about all know sessions.

ravi@arun:~$ echo “show sess” | socat unix-connect:/tmp/haproxy stdio
0x9ee3520: proto=tcpv4 src=192.168.19.117:4721 fe=ravitestbed be=ravitestbed srv=arun as=0 ts=08 age=4s calls=3
rq[f=009202h,l=0,an=00h,rx=20s,wx=,ax=] rp[f=009202h,l=0,an=00h,rx=20s,wx=,ax=] s0=[7,8h,fd=1,ex=] s1=[7,8h,fd=2,ex=] exp=20s
0x9eeb8e8: proto=tcpv4 src=192.168.19.117:4723 fe=ravitestbed be=ravitestbed srv=arun as=0 ts=08 age=4s calls=3
rq[f=009000h,l=0,an=00h,rx=20s,wx=,ax=] rp[f=009202h,l=0,an=00h,rx=20s,wx=,ax=] s0=[7,8h,fd=8,ex=] s1=[7,8h,fd=9,ex=] exp=20s
0x9ef3d08: proto=tcpv4 src=192.168.19.117:4725 fe=ravitestbed be=ravitestbed srv=arun as=0 ts=08 age=4s calls=3
rq[f=009000h,l=0,an=00h,rx=20s,wx=,ax=] rp[f=009202h,l=0,an=00h,rx=20s,wx=,ax=] s0=[7,8h,fd=12,ex=] s1=[7,8h,fd=13,ex=]
exp=20s
0x9f04548: proto=unix_stream as=2 ts=09 age=0s calls=2 rq[f=00e042h,l=10,an=20h,rx=10s,wx=,ax=]

rp[f=048060h,l=716,an=00h,rx=,wx=10s,ax=] s0=[7,0h,fd=3,ex=] s1=[0,0h,fd=-1,ex=] exp=9s

This will give you information about the running HAProxy process such as pid, uptime and etc.

ravi@arun:~$ echo “show info” | socat unix-connect:/tmp/haproxy stdio
Name: HAProxy
Version: 1.3.23
Release_date: 2010/01/28
Nbproc: 1
Process_num: 1
Pid: 11829
Uptime: 0d 0h42m53s
Uptime_sec: 2573
Memmax_MB: 0
Ulimit-n: 50013
Maxsock: 50013
Maxconn: 25000
Maxpipes: 0
CurrConns: 1
PipesUsed: 0
PipesFree: 0
Tasks: 1
Run_queue: 1
node: ravi.world
description:

This will give you stats on all of your backends and frontends, some of the same stuff you see on the stats page enabled by the stats uri configuration. As an added bonus it’s all in CSV.

ravi@arun:~$ echo “show stat” | socat unix-connect:/tmp/haproxy stdio
#
pxname,svname,qcur,qmax,scur,smax,slim,stot,bin,bout,dreq,dresp,ereq,econ,eresp,wretr,wredis,status,weight,act,bck,chkfail,ch
kdown,lastchg,downtime,qlimit,pid,iid,sid,throttle,lbtot,tracked,type,rate,rate_lim,rate_max,
ravitestbed,FRONTEND,,,0,5,100,30,32582,50616,0,0,0,,,,,OPEN,,,,,,,,,1,1,0,,,,0,0,0,5,
ravitestbed,trupti,0,0,0,2,,15,7020,22722,,0,,0,0,0,0,no check,1,1,0,,,,,,1,1,1,,15,,2,0,,2,
ravitestbed,arun,0,0,0,5,,15,25562,27894,,0,,0,0,0,0,no check,1,1,0,,,,,,1,1,2,,15,,2,0,,3,
ravitestbed,BACKEND,0,0,0,5,100,30,32582,50616,0,0,,0,0,0,0,UP,2,2,0,,0,2710,0,,1,1,0,,30,,1,0,,5,
stats,FRONTEND,,,0,1,100,21,9605,152357,0,0,0,,,,,OPEN,,,,,,,,,1,2,0,,,,0,0,0,9,
stats,BACKEND,0,0,0,1,100,5,9605,152357,0,0,,5,0,0,0,UP,0,0,0,,0,2710,0,,1,2,0,,0,,1,0,,4,

show errors will give you a capture of last error on each backend/frontend.

ravi@arun:~$ echo “show errors” | socat unix-connect:/tmp/haproxy stdio

Reffer:
http://www.dest-unreach.org/socat/
http://haproxy.1wt.eu/download/1.3/doc/configuration.txt

Thanks to Joe (http://www.joeandmotorboat.com)

Thank you,
Ravi

Haproxy + apache dropping the connections

Haproxy + apache dropping the connections

Many times  haproxy and apache does not reliable to serve the connections without tune or we say we need to set system as well some kernel parameters to work it better.

Here haproxy gives an errors to connect to apache, at that time it logs the errors into ‘dmesg | tail’ or in ‘/var/log/messages’ “kernel: ip_conntrack: table full, dropping packet” that is related to ip_conntrack kernel module.

Conntrack table is hash table (hash map) of fixed size (8192 entries by default), which is used for primary lookup. When the slot in the table is found it points to list of conntrack structures, so secondary lookup is done using list traversal. 65536/8192 gives 8 – the average list length. You may want to experiment with this value on heavily loaded systems.

If this error founds into /var/log/messages or dmesg you have to apply following steps to resolve.

Here I have done this to changes and added few settings in kernel also we will do it to set apache MPM and Haproxy tunning with sysctl.conf

Note: I have tried all this workaround and apply on CentOS-5.2, but don’t worry ip_conntrack module is default in kernel 2.6 +

1) To check ip_contrack is compiled with your kernel

[root@ravi.com ~]# modinfo ip_conntrack
filename:       /lib/modules/2.6.18-128.el5/kernel/net/ipv4/netfilter/ip_conntrack.ko
license:        GPL
srcversion:     F1390E605BBFB05078B78E8
depends:        nfnetlink
vermagic:       2.6.18-128.el5 SMP mod_unload gcc-4.1
module_sig:     883f350497747c575ed35fe9471dce112565509f4b58f4f3e440c6bcc05c2fba9bbdd224bdeb8209e293da385133a876e44a7b449ba59a882a8282b

2) Probe ip_conntrack kernel module or add it in /etc/modprobe.conf

[root@ravi.com ~]# modprobe ip_conntrack hashsize=131072

or

open /etc/modprobe.conf and add below lines at the end of file

options ip_conntrack hashsize=131072

3) before go to apply the 4th step, just check the ip_conntrack setting is into /etc/sysctl.conf

grep “ip_conntrack” /etc/sysctl.conf

if its found then apply 4th step or edit the /etc/sysctl.conf and add the given two lines at the end of file and save it then go for 4th step
(the value is compare to your RAM and set it to below)

net.ipv4.ip_conntrack_max = 16777216
net.ipv4.netfilter.ip_conntrack_max = 16777216

4) To apply the sysctl parameters run ‘sysctl -p’

[root@ravi.com ~]# sysctl -p

5) Now check the ip_conntrack is logging the connections and check not dropping any more

[root@ravi.com ~]# cat /proc/slabinfo | grep conn
ip_conntrack_expect      0      0    136   28    1 : tunables  120   60    8 : slabdata      0      0      0
ip_conntrack      216053 231335    304   13    1 : tunables   54   27    8 : slabdata  17795  17795    216

6) Also you can check how much memory utilized by ip_conntrack module per connection.

[root@ravi.com ~]# grep ip_conntrack /var/log/messages

/var/log/messages.2:Jan 14 21:46:04 ravi kernel: ip_conntrack version 2.4 (8192 buckets, 65536 max) – 304 bytes per conntrack

1M connections would require 304MB of kernel memory.

Thanks
Ravi

How to configure-add java extension for php on CentOS 5

How to configure-add java extension for php on CentOS 5

Here I have used php version 5.2.11 and to jdk-6u18-linux-x64.bin

( from given JAVA download link)

[root@ravi.com ~]# yum  -y install httpd php php-devel php-gd php-cli php-xml php-ldap php-common php-pear php-pdo

Install java jdk and set env in ~/.bashrc also run this on CLI to verify before next step.

export JAVA_HOME=//usr/java/jdk1.6.0_18
export PATH=/usr/java/jdk1.6.0_18/bin:$PATH

[root@ravi.com ~]# echo $JAVA_HOME

Now set the dynamic linker library path into /etc/ld.so.conf

/usr/java/jdk1.6.0_18/jre/lib/amd64
/usr/java/jdk1.6.0_18/jre/lib/amd64/server

[root@ravi.com ~]# ldconfig

Now download php-java-bridge source rpm

[root@ravi.com ~]# wget ftp://195.220.108.108/linux/sourceforge/p/project/ph/php-java-bridge/OldFiles/php-java-bridge-4.1.8-1.src.rpm

unpack the php-java-bridge rpm

[root@ravi.com ~]# rpm2cpio php-java-bridge-4.1.8-1.src.rpm | cpio -ivd
php-java-bridge.spec
php-java-bridge_4.1.8.tar.gz
16155 blocks

untar the php-java-bridge and configure the module

[root@ravi.com ~]# tar xzf php-java-bridge_4.1.8.tar.gz && cd php-java-bridge-4.1.8

[root@ravi.com ~]# phpize

[root@ravi.com ~]# ./configure –with-java=$JAVA_HOME  &&  make &&  make install

[root@ravi.com ~]# echo “extension=java.so” > /etc/php.d/java.ini

open the php.ini and edit the at the end of file

[java]
java.class.path=/usr/lib64/php/modules/
java.library=/usr/java/jdk1.6.0_18/jre/lib/amd64/server/libjvm.so

Now restart apache
[root@ravi.com ~]# /etc/init.d/httpd restart

[root@ravi.com ~]# php -i | grep java
/etc/php.d/java.ini,
java
java support => Enabled
java bridge => 4.1.8
java.java_home =>
java.java => java
java.log_file => <stderr>
java.log_level => no value (use back-end’s default level)
java.security_policy => Off
java command =>  LD_LIBRARY_PATH=/usr/lib64/php/modules:/usr/lib/10.2.0.3/client64/lib java -Djava.library.path=/usr/lib64/php/modules -Djava.class.path=/usr/lib64/php/modules/JavaBridge.jar -Djava.awt.headless=true -Dphp.java.bridge.base=/usr/lib64/php/modules php.java.bridge.Standalone LOCAL:9267 1
java status => running
java server => 9267
PATH => /usr/java/jdk1.6.0_18/bin:/usr/java/jdk1.6.0_18/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
JAVA_HOME => /usr/java/jdk1.6.0_18
_SERVER[“PATH”] => /usr/java/jdk1.6.0_18/bin:/usr/java/jdk1.6.0_18/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
_SERVER[“JAVA_HOME”] => /usr/java/jdk1.6.0_18
_ENV[“PATH”] => /usr/java/jdk1.6.0_18/bin:/usr/java/jdk1.6.0_18/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
_ENV[“JAVA_HOME”] => /usr/java/jdk1.6.0_18

Test your code in test.php and run to check
<?php
// get instance of Java class java.lang.System in PHP
$system = new Java(‘java.lang.System’); // demonstrate property access
print ‘Java version=’.$system->getProperty(‘java.version’).’ ‘;
print ‘Java vendor=’ .$system->getProperty(‘java.vendor’).’ ‘;
print ‘OS=’.$system->getProperty(‘os.name’).’ ‘.
$system->getProperty(‘os.version’).’ on ‘.
$system->getProperty(‘os.arch’).’ ‘; // java.util.Date example
$formatter = new Java(‘java.text.SimpleDateFormat’,
“EEEE, MMMM dd, yyyy ‘at’ h:mm:ss a zzzz”); print $formatter->format(new Java(‘java.util.Date’));
?>

[root@ravi.com ~]# php test.php
Java version=1.6.0_18 Java vendor=Sun Microsystems Inc. OS=Linux 2.6.18-53.el5xen on amd64 Friday, January 22, 2010 at 4:44:48 AM Pacific Standard Time
Thanks

Ravi