Day: January 24, 2010

Haproxy + apache dropping the connections

Haproxy + apache dropping the connections

Many times  haproxy and apache does not reliable to serve the connections without tune or we say we need to set system as well some kernel parameters to work it better.

Here haproxy gives an errors to connect to apache, at that time it logs the errors into ‘dmesg | tail’ or in ‘/var/log/messages’ “kernel: ip_conntrack: table full, dropping packet” that is related to ip_conntrack kernel module.

Conntrack table is hash table (hash map) of fixed size (8192 entries by default), which is used for primary lookup. When the slot in the table is found it points to list of conntrack structures, so secondary lookup is done using list traversal. 65536/8192 gives 8 – the average list length. You may want to experiment with this value on heavily loaded systems.

If this error founds into /var/log/messages or dmesg you have to apply following steps to resolve.

Here I have done this to changes and added few settings in kernel also we will do it to set apache MPM and Haproxy tunning with sysctl.conf

Note: I have tried all this workaround and apply on CentOS-5.2, but don’t worry ip_conntrack module is default in kernel 2.6 +

1) To check ip_contrack is compiled with your kernel

[root@ravi.com ~]# modinfo ip_conntrack
filename:       /lib/modules/2.6.18-128.el5/kernel/net/ipv4/netfilter/ip_conntrack.ko
license:        GPL
srcversion:     F1390E605BBFB05078B78E8
depends:        nfnetlink
vermagic:       2.6.18-128.el5 SMP mod_unload gcc-4.1
module_sig:     883f350497747c575ed35fe9471dce112565509f4b58f4f3e440c6bcc05c2fba9bbdd224bdeb8209e293da385133a876e44a7b449ba59a882a8282b

2) Probe ip_conntrack kernel module or add it in /etc/modprobe.conf

[root@ravi.com ~]# modprobe ip_conntrack hashsize=131072

or

open /etc/modprobe.conf and add below lines at the end of file

options ip_conntrack hashsize=131072

3) before go to apply the 4th step, just check the ip_conntrack setting is into /etc/sysctl.conf

grep “ip_conntrack” /etc/sysctl.conf

if its found then apply 4th step or edit the /etc/sysctl.conf and add the given two lines at the end of file and save it then go for 4th step
(the value is compare to your RAM and set it to below)

net.ipv4.ip_conntrack_max = 16777216
net.ipv4.netfilter.ip_conntrack_max = 16777216

4) To apply the sysctl parameters run ‘sysctl -p’

[root@ravi.com ~]# sysctl -p

5) Now check the ip_conntrack is logging the connections and check not dropping any more

[root@ravi.com ~]# cat /proc/slabinfo | grep conn
ip_conntrack_expect      0      0    136   28    1 : tunables  120   60    8 : slabdata      0      0      0
ip_conntrack      216053 231335    304   13    1 : tunables   54   27    8 : slabdata  17795  17795    216

6) Also you can check how much memory utilized by ip_conntrack module per connection.

[root@ravi.com ~]# grep ip_conntrack /var/log/messages

/var/log/messages.2:Jan 14 21:46:04 ravi kernel: ip_conntrack version 2.4 (8192 buckets, 65536 max) – 304 bytes per conntrack

1M connections would require 304MB of kernel memory.

Thanks
Ravi