Jan
15
Posted on 15-01-2012
Filed Under (Nagios Monitoring) by Arun Bagul

Introduction –
Check_MK is one of the best Nagios monitoring system plugin that allows creating rule-based configuration using Python and offloading work from the Nagios core to make it scale better, allowing more systems to be monitored from a single Nagios server.  It comes with a set of system checks and web based UI and a module that allows fast access to the multiple Nagios servers
To see Top 5 monitoring tools comparison here – http://www.indiangnu.org/2011/top-5-open-source-network-and-performance-monitoring-alerting-system/

** What is Check_MK ?
Check_MK is Nagios plugin to monitor remote host,services and Nagios Multisite GUI to monitor multiple nagios from single web based UI.

** What are features of Check_MK?

Check_MK is the best nagios plugin for monitoring remote host. Beauty of this plugin is that most of the commons services are checked in only one connection to remote hosts and results are submitted as passive results to nagios.  This improves performance, less network traffic and of course no load or burden on remote host (which we are monitoring).
For Check_MK, we have to installed check_mk agent ie check_mk_agent simple shell script and bind this script to  6556 tcp port using xinetd daemon.

* Check_MK also provides following features.
– Check_MK support dynamic nagios configuration generation, inventory of checks.
Livestatus – a beautiful plugin to communicate with nagios (unix socket)
Nagios Multisite GUI – monitor multiple nagios from single web based interface
– SNMP supports and automatic service detections for many cisco switches, routers and NetApp filers as well.
– whatever Nagios supports

* Nagios Multisite GUI ~ Another beauty of Check_MK nagios plugin is monitoring multiple nagios using only one web based interface.
This web based interface uses Check_MK Livestatus plugin to get nagios data.
– Livestatus nagios plugin to communicate with nagios using SQL like language (unix socket)

** How to configure Check_MK Agent on Linux/Unix ?

Go to site “http://mathias-kettner.de” and download check_mk_agent rpm or deb package and install agent.

* xinetd setting –
[root@arunb ~]# cat /etc/xinetd.d/check_mk
service check_mk
{
type           = UNLISTED
port           = 6556
socket_type    = stream
protocol       = tcp
wait           = no
user           = root
server         = /usr/bin/check_mk_agent
#only_from      = 127.0.0.1 10.0.20.1 10.0.20.2
disable        = no
}
[root@arunb ~]#

* How to start check_mk_agent

[root@arunb ~]#  /etc/init.d/xinetd start

* How to add custom checks in Check_MK agent –
Check_MK agent support two type of custom checks …
  a) MRPE – Check_MK Remote Plugin Executor (same as NRPE) – Create “/etc/check_mk/mrpe.cfg” file if doesn’t exist.
[root@arunb ~]# cat /etc/check_mk/mrpe.cfg
#check_mk Remote Plugin Executor config file
# Service_Name         CommandLine
MySQL_Server    /home/check_mk/nagios_scripts/check_mysql
MySQL_Slave     /home/check_mk/nagios_scripts/check_mysqlslavesql.sh
[root@arunb ~]#
* plugins script as same as nrpe

  b) Local Checks – go to local directory as shown below and put your script which will be executed by check_mk_agent shell script.

[root@arunb ~]# ls /usr/lib/check_mk_agent/
local  plugins
[root@arunb ~]# cat /usr/lib/check_mk_agent/local/nagios_mailq.sh
#!/bin/bash
#nagips plugin dir path
nagios_plugin=”/usr/lib64/nagios/plugins”
str=”/var/spool/mqueue is empty Total requests: 0″
count=$( mailq | grep ‘Total requests:’ | awk -F’Total requests: ‘ ‘{print $2}’)
#echo “‘$count'”
if [ $count  -lt 5 ];then
status=0
echo “$status  mailq count=$count  OK – Total mails in queue: $count”
elif [ $count -ge 5 ] &&  [ $count -lt 20 ];then
status=1
echo “$status  mailq count=$count  WARNING – Total mails in queue: $count”
elif [ $count -ge 20 ];then
status=2
echo “$status  mailq  count=$count  CRITICAL – Total mails in queue: $count”
fi
#end
[root@arunb ~]#

** Check_MK Nagios configuration –

Go to site “http://mathias-kettner.de” and download check_mk rpm or deb package or source with livestatus install check_mk nagios plugin

** main.mk config file –

[root@arunb ~]# cat /etc/check_mk/main.mk
# Put your host names here
# all_hosts = [ ‘localhost’ ]
all_hosts = [
‘localhost|LOC’,
‘arun-pc|LOC|Windows’,
‘host1’|LOC|http_port’,
‘host2’|LOC|http_port’,
]

– here LOC and http_port and WINDOWS are called as “Tag” which will be used in check_mk config file for writing rules.

* I have added extra config in following file –

[root@arunb ~]# cat /etc/check_mk/conf.d/check_mk-config.mk

define_hostgroups     = True
define_servicegroups  = True
# tcp timeout
tcp_connect_timeout = 25.0

define_hostgroups = {
“LOC”       : “My Network”,
“WINDOWS”   : “Windows Servers”,
}

define_servicegroups = {
“MySQL”      : “MySQL DB”,
“ORACLE”     : “Oracle DB”,
“HTTPD”      : “Web Server”,
“WINDOWS”    : “Windows Checks”,
}
host_groups = [

( ‘LOC’,  [‘LOC’], ALL_HOSTS ),
( ‘WINDOWS’, [‘Windows’], ALL_HOSTS ),
]

service_groups = [
( “MySQL”, ALL_HOSTS, [“MySQL_S*”]),
( “MAIL”, ALL_HOSTS, [“proc_SMTP*”]),
( “NAGIOS”, ALL_HOSTS, [“proc_nagios*”]),
( “HTTPD”, ALL_HOSTS, [“proc_Apache2”]),
]

host_contactgroups = [
( ‘admin,admin-sms’, [‘LOC’], ALL_HOSTS ),
( ‘winadmin,winadmin-sms’, [‘Windows’], ALL_HOSTS ),
]

service_contactgroups = [
# Notification for all services to admin group
( ‘admin’, ALL_HOSTS,[“.*”]),
( ‘dbadmin,db-admin-sms’, ALL_HOSTS, [“MySQL_S*”] ),
]

# Inventory

filesystem_default_levels[“levels”] = ( 80, 90 )
ntp_default_levels = (20, 1000.0, 5000.0)
if_default_error_levels = (0.2, 0.5)

ignored_services = [
( [“Windows”], ALL_HOSTS, [ “LOG Application”,”LOG Security”, “LOG System”,”LOG DNS Server”,”DHCP Stats”] ),
]

inventory_processes = [
( “SSH”, “/usr/sbin/sshd”, ANY_USER, 1, 1, 10, 15),
( “Apache2”, “/usr/sbin/httpd”, “apache”, 1, 1, 90, 100 ),
( “FTP_Server”, “~/usr/sbin/vsftpd.*”, “root”, 1, 1, 40, 50),
( “DNS”, “~/usr/sbin/named.*”, “named”, 1, 1, 5, 10),
( “nagios”, “~/omd/sites/.*/bin/nagios.*-ud.*/omd/sites/.*/tmp/nagios/nagios.cfg*”, ANY_USER, 1, 1, 100, 200),
]

#Classical Nagios Services
legacy_checks = [
((“check_http_arg!/”, “http_port”, False), [“http_port”], ALL_HOSTS ),
]
extra_nagios_conf += r”””
# check_http command
define command{
command_name    check_http_arg
command_line    $USER1$/check_http -H $HOSTADDRESS$ -w 30 -c 40 -t 60 -u $ARG1$
}
“””
[root@arunb ~]#

** How to Take Host inventory –  

Taking host inventory is first step after configuring or adding hosts in main.mk file as shown above.
Defining extra rules like “check_mk-config.mk” file can be  done as per your requirement.

– for all host
[root@arunb ~]# check_mk -v -u -I

– for one host
[root@arunb ~]# check_mk -v -u -I tcp host1

– verify inventory
[root@arunb ~]# check_mk -v -D host1   OR
[root@arunb ~]# check_mk -v -d host1

** Generate Nagios Config –
Generating nagios config after taking hosts inventory is final check_mk step.

[root@arunb ~]# check_mk -v -U

* Verify Nagios setting using nagios -v command

[root@arunb ~]#  /usr/sbin/nagios3 -v /etc/nagios3/nagios.cfg

* Start/Restart Nagios

[root@arunb ~]# /etc/init.d/nagios start  OR

[root@arunb ~]# check_mk -v -R

Check_MK web interface – http://localhost/check_mk/ or http://your_nagios_server_ipaddr/check_mk/

** How to set Nagios/ Check_MK Multisite –

– First step is to configure livestatus on remote nagios and add following configuration on Multisite nagios check_mk “multisite.mk” config file

* xinetd config for livestatus –

[root@remote ~]#  cat /etc/xinetd.d/livestatus
# Nagios – check_mk Livestatus multisites
service livestatus
{
type        = UNLISTED
port        = 6557
socket_type    = stream
protocol    = tcp
wait        = no
cps             = 100 3
flags           = NODELAY
user   =  nagios
server        = /usr/bin/unixcat
server_args  =  /opt/nagios/var/rw/live
disable        = no
}
[root@remote ~]#

– restart xinetd to start livestatus over tcp port 6557

[root@remote ~]#  /etc/init.d/xinetd start

** Add remote nagios site in “multisite.mk” config  file as shown below –

[root@arunb ~]# cat  /etc/check_mk/multisite.mk
admin_users = [ “omdadmin”, “admin” ]
sites = {
# local nagios
“local” : {
“alias” : “LOC”
},
# Remote site
“REMOTE”: {
“alias”:          “Remote Nagios Site”,
“socket”:         “tcp:192.168.0.120:6557”,
“nagios_url”:     “/nagios”,
“nagios_cgi_url”: “/nagios/cgi-bin”,
“pnp_url”:        “/pnp4nagios/”,
},
}

[root@arunb ~]#

That’s it! Enjoy Nagios and Check_MK!

Thank you,
Arun Bagul

(1) Comment    Read More   
Aug
12
Posted on 12-08-2011
Filed Under (Nagios Monitoring, UNIX/Linux) by Arun Bagul

Introduction – We wrote perl script for Nagios daily and weekly Reporting. This script is parsing nagios.log for all alerts notification and generating nagios report in CSV format…

** How to use?

arunb@arunb:~$ perl /home/arunb/nagios-log-parsing.pl
* Usage: /home/arunb/nagios-log-parsing.pl { <nagios_log file>  [ –summary ] }
arunb@arunb:~$

** To Generate Daily Nagios Report –
arunb@arunb:~$ perl /home/arunb/nagios-log-parsing.pl  /usr/local/nagios/var/nagios.log
arunb@arunb:~$ perl /home/arunb/nagios-log-parsing.pl  /usr/local/nagios/var/nagios.log  –summary

arunb@arunb:~$ perl /home/arunb/nagios-log-parsing.pl  /usr/local/nagios/var/nagios.log  –summary  > /tmp/NAGIOS-daily.csv

** To Generate Weekly Nagios Report –

Add all weeks nagios.log files absolute path (per line) in file say “/tmp/nagios-logfile”

arunb@arunb:~$  for i in `cat /tmp/nagios-logfile`; do  perl /home/arunb/nagios-log-parsing.pl  $i; done | grep -v ^$ > /tmp/NAGIOS-weekly.csv

* Want to see the script ?

arunb@arunb:~$ cat /home/arunb/nagios-log-parsing.pl

#!/usr/bin/perl

my $file = $ARGV[0];
my $stime = 0;
my $htime = 0;
my ($s_alert,$h_alert ) = (undef,undef);

my $STATE_OK;
my $STATE_WARNING;
my $STATE_CRITICAL;
my $STATE_UNKNOWN;
my %STATE_COUNTER;
my %ALERT;

############################

if ($ARGV[0]) {
#my $file = "/usr/local/nagios/var/nagios.log";

open (MYFILE, "<$file" ) or die "Can;t open nagios file";
while(<MYFILE>) {
chomp;
my $line = $_;
##print "\n".$line;
#if ($line =~ m/[(.*)].*;(.*);(.*);(.*);.*/) {
#if ($line =~ m/\[(.*)\]\s+(HOST|SERVICE)\sNOTIFICATION:\s\w+;(.*);.*/) {
if ($line =~ m/\[(.*)\]\s+SERVICE\sNOTIFICATION:\s\w+;(.*);.*/) {
my $time = $1;
my $mydate = scalar(localtime($time));
my ($host,$service,$alert)  = split(';',$2);
#print "\nARUN=>$1 = $host,$service,$alert";
if ($s_alert eq $service) {
my $time_diff = $time - $stime;
if ($time_diff > 600 ) {
if ($alert ne "OK") {
if (exists $ALERT{"$service"}) { $ALERT{"$service"} = $ALERT{"$service"} + 1;}
else { $ALERT{"$service"} = 1 ; }
$STATE_COUNTER{"$alert"} = $STATE_COUNTER{"$alert"} + 1;
print "\n$mydate [$time],$host,$service,$alert";
}
}
} else {
if ($alert ne "OK") {
if (exists $ALERT{"$service"}) { $ALERT{"$service"} = $ALERT{"$service"} + 1;}
else { $ALERT{"$service"} = 1 ; }
$STATE_COUNTER{"$alert"} = $STATE_COUNTER{"$alert"} + 1;
print "\n$mydate [$time],$host,$service,$alert";
}
}
$stime = $time;
$s_alert = $service;

} elsif ($line =~ m/\[(.*)\]\s+HOST\sNOTIFICATION:\s\w+;(.*);.*/) {
my $time = $1;
my $mydate = scalar(localtime($time));
my ($host,$hdown,$alert)  = split(';',$2);
#print "\nARUN=>$1 = $host,$hdown";
if ($hdown eq "DOWN") {
if ($h_alert eq $host) {
my $time_diff = $time - $htime;
if ($time_diff > 600 ) {
$STATE_COUNTER{"$hdown"} = $STATE_COUNTER{"$hdown"} + 1;
print "\n$mydate [$time],$host,$hdown";
}
} else {
$STATE_COUNTER{"$hdown"} = $STATE_COUNTER{"$hdown"} + 1;
print "\n$mydate [$time],$host,$hdown";
}
}
$htime = $time;
$h_alert = $host;
}
}
## summary
my $total_counter = 0;
my $total_service = keys(%ALERT);
if ($ARGV[1] eq "--summary") {
print "\n\n"."-" x 20;
print "\nSummary Report\n";
print "-" x 20 . "\n";
print "\nAlert_Type,Total_Count";
while (my ($key,$value)=  each(%STATE_COUNTER)) {  print "\n$key,$value"; }
print "\n\nService_Name,Total_Count";
while (my ($key,$value)=  each(%ALERT)) { print "\n$key,$value"; $total_counter = $total_counter + $value; }
print "\n\nTotal Alerts = $total_counter";
print "\nTotal Service Failed = $total_service";
}

} else { print " * Usage: $0 { <nagios_log file>  [ --summary ] }";}

#end
print "\n";

** To download the script “http://www.indiangnu.org/wp-content/uploads/2011/nagios-log-parsing-pl.txt

Thank You,
Arun Bagul

(1) Comment    Read More   
Dec
30
Posted on 30-12-2010
Filed Under (Nagios Monitoring, UNIX/Linux) by Arun Bagul

Introduction –

Nagios log file print the time in epoch time format. To convert that nagios log with normal time/date format you can use the following perl script.

Please download the script from here- http://www.indiangnu.org/wp-content/uploads/2010/nagios-log-pl.txt

* How to use it –

root@me:~# cat /etc/nagios/var/nagios.log | grep -i  NOTIFICATION | /root/nagmon/nagios-log.pl

* Pasting the code here…

root@me:~#  /root/nagmon/nagios-log.pl
#!/usr/bin/perl

# Author - IndianGNU.org
# Read nagios.log file and show log with
# converted date field in human-readable format
# 

$numArgs = $#ARGV + 1;

if ( $ARGV[0] eq "-h")
{
	print " * Usage: cat   like '/usr/local/nagios/var/nagios.log' | $0 \n"; exit 1;
}

sub epochtime
{
  my $epoch_time = shift;
  ($sec,$min,$hour,$day,$month,$year) = localtime($epoch_time);

  # correct the date and month for humans
  $year = 1900 + $year;
  $month++;

  return sprintf("%02d/%02d/%02d %02d:%02d:%02d", $year, $month, $day, $hour, $min, $sec);
}

while (<>)
{
  my $epoch = substr $_, 1, 10;
  my $remainder = substr $_, 13;
  my $human_date = &epochtime($epoch);
  printf("[%s] %s", $human_date, $remainder);
}
exit;

Thanks you,
Arun

(0) Comments    Read More   

www.flickr.com
arunbagul's photos More of arunbagul's photos
Get Adobe Flash player
-->