Haproxy + apache dropping the connections
Many times haproxy and apache does not reliable to serve the connections without tune or we say we need to set system as well some kernel parameters to work it better.
Here haproxy gives an errors to connect to apache, at that time it logs the errors into ‘dmesg | tail’ or in ‘/var/log/messages’ “kernel: ip_conntrack: table full, dropping packet” that is related to ip_conntrack kernel module.
Conntrack table is hash table (hash map) of fixed size (8192 entries by default), which is used for primary lookup. When the slot in the table is found it points to list of conntrack structures, so secondary lookup is done using list traversal. 65536/8192 gives 8 – the average list length. You may want to experiment with this value on heavily loaded systems.
If this error founds into /var/log/messages or dmesg you have to apply following steps to resolve.
Here I have done this to changes and added few settings in kernel also we will do it to set apache MPM and Haproxy tunning with sysctl.conf
Note: I have tried all this workaround and apply on CentOS-5.2, but don’t worry ip_conntrack module is default in kernel 2.6 +
1) To check ip_contrack is compiled with your kernel
[root@ravi.com ~]# modinfo ip_conntrack
filename: /lib/modules/2.6.18-128.el5/kernel/net/ipv4/netfilter/ip_conntrack.ko
license: GPL
srcversion: F1390E605BBFB05078B78E8
depends: nfnetlink
vermagic: 2.6.18-128.el5 SMP mod_unload gcc-4.1
module_sig: 883f350497747c575ed35fe9471dce112565509f4b58f4f3e440c6bcc05c2fba9bbdd224bdeb8209e293da385133a876e44a7b449ba59a882a8282b
2) Probe ip_conntrack kernel module or add it in /etc/modprobe.conf
[root@ravi.com ~]# modprobe ip_conntrack hashsize=131072
or
open /etc/modprobe.conf and add below lines at the end of file
options ip_conntrack hashsize=131072
3) before go to apply the 4th step, just check the ip_conntrack setting is into /etc/sysctl.conf
grep “ip_conntrack” /etc/sysctl.conf
if its found then apply 4th step or edit the /etc/sysctl.conf and add the given two lines at the end of file and save it then go for 4th step
(the value is compare to your RAM and set it to below)
net.ipv4.ip_conntrack_max = 16777216
net.ipv4.netfilter.ip_conntrack_max = 16777216
4) To apply the sysctl parameters run ‘sysctl -p’
[root@ravi.com ~]# sysctl -p
5) Now check the ip_conntrack is logging the connections and check not dropping any more
[root@ravi.com ~]# cat /proc/slabinfo | grep conn
ip_conntrack_expect 0 0 136 28 1 : tunables 120 60 8 : slabdata 0 0 0
ip_conntrack 216053 231335 304 13 1 : tunables 54 27 8 : slabdata 17795 17795 216
6) Also you can check how much memory utilized by ip_conntrack module per connection.
[root@ravi.com ~]# grep ip_conntrack /var/log/messages
/var/log/messages.2:Jan 14 21:46:04 ravi kernel: ip_conntrack version 2.4 (8192 buckets, 65536 max) – 304 bytes per conntrack
1M connections would require 304MB of kernel memory.
Thanks
Ravi
Enabling mod-geoip