This document address how to disable nagios notification’s with Ops Center
When using Ops Center for patching the Solaris servers
The challenge is how to disable nagios notifications remotely (within Ops Center) before patching, then re-enable notifications when done.
Nagios notification overview
There are two ways to send Nagios external commends.
- Using the web interface.
- Sending commends to a Nagios pipe called nagios.cmd (location by us /usr/local/nagios/var2.12/rw/nagios.cmd).
Nagios notifications
As noted above, there is two ways to change Nagios configuration / notifications.
The challenges becomes on how this could be accomplish remotely?
Methods to disable Nagios notifications remotely
There is at-least three ways to disable Nagios notifications remotely.
- Use curl to send commends to Nagios with cmd.cgi (Needs to be enabled).
- Use perl, php, etc.. to send commends to nagios.cmd.
- Send an email to a local Nagios user and use procmail to take action.
Note: Option 3 is already used in our environment for remote acknowledgment
Ops Center and Nagios
With Ops Center patching the Soalris servers I had to find a way to disable notifications remotely with one of the methods described above
With ops Center you have the option to run a pre-action and a post-action, this could run a script to disable nagios notifications,
the challenge with using a script is
- Run a pre-action script on The Ops Center server, the problem with that is that the Ops Center server has no idea which servers are now bing patched
- The workaround/solution for that is a small script that Sun Support provided (the script is in available Here, the script could only run on the Ops Center server, the script will connect to the Ops Center Data Base and lookup the current active jobs, and with some logic getting the server names being patched, we could then run curl to disable/enable nagios notifications
- Run a script on the server being patched, The problem with that is, the server being patched could not connect to Ops Center with HTTP making it imposable to use this method
- The workaround/solution for this is opening the firewall which i don’t think will happen
This brings me to the last method which was used.
- Modify and extend the current Nagios perl script, to accept remote commends send to Nagios like disable/enable notifications.
- Solution, I Installed a small php script that will disable/enable Nagios host and or services, I modified the /usr/local/nagios/procmail/process_mail.pl to process additional user input.
- You could now send an email to nagios@domain.com to enable or disable a host/service
- The email must come from a valid user, I used usera@domain.com
- The subject line should contain 3 values separated by :
For example this will turn notifications off:
-
-
- usera:hostname:off
-
For example this will turn notifications on:
-
-
- usera:hostname:on
-
Ops Center configuration
Note: email needs to work on the client this should work.
I uploaded to Ops Center two scripts called disable-nagios-notifaction.sh and enable-nagios-notifaction.sh
To upload a pre-actions/post-actions (as well as probes/macros).
Login to Ops Center
Click on Libraries > Local Content > Upload Local Action (on the left side)
On the popup
- Select SPARC
- Give the action a name
- Select pre or post action
- Browse to the action script
Click upload, a job will be created.
Once done you will need to create a profile from this.
Click on Libraries > Update profiles > New profile (on the left side)
- Select SPARC
- Click on local > pre or post actions (select your script name
- Click required
Give the Profile a name and description
Now last schedule a patching job
Click on Libraries > Update profiles >New Update OS Job (on the left side)
- Click on the green + sin and add 3 more jobs
- Select your pre-actions script to disable nagios as the first
- Optional select and add any patch back-out scripts
- Select Check Bugs Fix for the next script
- Select your post-actions script to enable nagios as the next
- Change the policy to Yes for ALL
- Select the target system(s)
Leave “Use the same Targets for all tasks in the job” and click next, select your schedule time and off you go.
Scripts used to disable/enable Nagios
- Disable nagios notification
- Enable nagios notification
- Nagios Process mail
- Nagios Notifications enable/disable script
- Ops Center Get list of target hosts jobs
Nagios Disable / Enable scripts – add to process_mail.pl
# Add by Eli Kleimman on 3/15/10, this will disable/enable notification used by usera for patching
if( $body =~ /(?i)usera:/ ) {
#example mail action = "usera:hostname:on"
@notify_fileds = split(/:/, $body);
if ($notify_fileds[0] eq "usera") {
if ($notify_fileds[2] eq "off") {
system ("/usr/local/nagios/procmail/notifications.pl $notify_fileds[1] off");
system ("/usr/local/nagios/procmail/notifications.pl $notify_fileds[1] all off");
}
if ($notify_fileds[2] eq "on") {
system ("/usr/bin/perl /usr/local/nagios/procmail/notifications.pl $notify_fileds[1] on");
system ("/usr/bin/perl /usr/local/nagios/procmail/notifications.pl $notify_fileds[1] all on");
}
}
}
disable-nagios-notifaction.sh
#!/bin/sh set -x /usr/bin/mailx -r usera@domain.com -s usera:`hostname`:off nagios@domain.com <enable-nagios-notifaction.sh
#!/bin/sh set -x /usr/bin/mailx -r usera@domain.com -s usera:`hostname`:on nagios@domain.com <Example to enable/disable nagios with curl
#Disable host check curl --anyauth -u usera:password -d "cmd_typ=25&cmd_mod=2&host=hostname&btnSubmit=Commit" "http://nagios.domain.com/nagios/cgi-bin/cmd.cgi" #Disable host & servicess curl --anyauth -u usera:password -d "cmd_typ=29&cmd_mod=2&host=hostname&btnSubmit=Commit" "http://nagios.domain.com/nagios/cgi-bin/cmd.cgi" # ## # #Enable host check curl --anyauth -u usera:password -d "cmd_typ=24&cmd_mod=2&host=hostname&btnSubmit=Commit" "http://nagios.domain.com/nagios/cgi-bin/cmd.cgi" #Enable host & servicess curl --anyauth -u usera:1password -d "cmd_typ=28&cmd_mod=2&host=hostname&btnSubmit=Commit" "http://nagios.domain.com/nagios/cgi-bin/cmd.cgi" disable only a service by name curl -d "cmd_typ=23&cmd_mod=2&host=[THE HOST]&service=[THE SERVICE]&btnSubmit=Commit" "http://[[NAGIOS SERVER]/nagios/cgi-bin/cmd.cgi" enable only a service by name curl -d "cmd_typ=22&cmd_mod=2&host=[THE HOST]&service=[THE SERVICE]&btnSubmit=Commit" "http://[[NAGIOS SERVER]/nagios/cgi-bin/cmd.cgi"You might also like this. Oracle Ops Center ASR Blacklist Bug Workaround
You might also like this. Oracle Ops Center ZFS, Device fail-over Scripts