Integrating Oracle Ops Center Alert Notifications With Nagios

1 Star2 Stars3 Stars4 Stars5 Stars (No Ratings Yet)
Loading...

This document address how to disable nagios notification’s with Ops Center

When using Ops Center for patching the Solaris servers
The challenge is how to disable nagios notifications remotely (within Ops Center) before patching, then re-enable notifications when done.

Nagios notification overview

There are two ways to send Nagios external commends.

  • Using the web interface.
  • Sending commends to a Nagios pipe called nagios.cmd (location by us /usr/local/nagios/var2.12/rw/nagios.cmd).

Nagios notifications

As noted above, there is two ways to change Nagios configuration / notifications.
The challenges becomes on how this could be accomplish remotely?

Methods to disable Nagios notifications remotely

There is at-least three ways to disable Nagios notifications remotely.

  • Use curl to send commends to Nagios with cmd.cgi (Needs to be enabled).
  • Use perl, php, etc.. to send commends to nagios.cmd.
  • Send an email to a local Nagios user and use procmail to take action.

Note: Option 3 is already used in our environment for remote acknowledgment

Ops Center and Nagios

With Ops Center patching the Soalris servers I had to find a way to disable notifications remotely with one of the methods described above
With ops Center you have the option to run a pre-action and a post-action, this could run a script to disable nagios notifications,
the challenge with using a script is

  • Run a pre-action script on The Ops Center server, the problem with that is that the Ops Center server has no idea which servers are now bing patched
    • The workaround/solution for that is a small script that Sun Support provided (the script is in available Here, the script could only run on the Ops Center server, the script will connect to the Ops Center Data Base and lookup the current active jobs, and with some logic getting the server names being patched, we could then run curl to disable/enable nagios notifications
  • Run a script on the server being patched, The problem with that is, the server being patched could not connect to Ops Center with HTTP making it imposable to use this method
    • The workaround/solution for this is opening the firewall which i don’t think will happen

This brings me to the last method which was used.

  • Modify and extend the current Nagios perl script, to accept remote commends send to Nagios like disable/enable notifications.
    • Solution, I Installed a small php script that will disable/enable Nagios host and or services, I modified the /usr/local/nagios/procmail/process_mail.pl to process additional user input.
  • You could now send an email to nagios@domain.com to enable or disable a host/service
    • The email must come from a valid user, I used usera@domain.com
    • The subject line should contain 3 values separated by :

For example this will turn notifications off:

      • usera:hostname:off

For example this will turn notifications on:

      • usera:hostname:on

Ops Center configuration

Note: email needs to work on the client this should work.
I uploaded to Ops Center two scripts called disable-nagios-notifaction.sh and enable-nagios-notifaction.sh
To upload a pre-actions/post-actions (as well as probes/macros).
Login to Ops Center
Click on Libraries > Local Content > Upload Local Action (on the left side)
On the popup

  • Select SPARC
  • Give the action a name
  • Select pre or post action
  • Browse to the action script

Click upload, a job will be created.
Once done you will need to create a profile from this.
Click on Libraries > Update profiles > New profile (on the left side)

  • Select SPARC
  • Click on local > pre or post actions (select your script name
  • Click required

Give the Profile a name and description
Now last schedule a patching job
Click on Libraries > Update profiles >New Update OS Job (on the left side)

  • Click on the green + sin and add 3 more jobs
  • Select your pre-actions script to disable nagios as the first
  • Optional select and add any patch back-out scripts
  • Select Check Bugs Fix for the next script
  • Select your post-actions script to enable nagios as the next
  • Change the policy to Yes for ALL
  • Select the target system(s)

Leave “Use the same Targets for all tasks in the job” and click next, select your schedule time and off you go.

Scripts used to disable/enable Nagios

Nagios Disable / Enable scripts – add to process_mail.pl

disable-nagios-notifaction.sh

enable-nagios-notifaction.sh

Example to enable/disable nagios with curl

You might also like this. Oracle Ops Center ASR Blacklist Bug Workaround
You might also like this. Oracle Ops Center ZFS, Device fail-over Scripts

Leave a Reply

avatar
  Subscribe  
Notify of