The following is the GNU All-permissive License as recommended in https://www.gnu.org/licenses/license-recommendations.en.html

Copying and distribution of this file, with or without modification, are permitted in any medium without royalty provided the copyright notice and this notice are preserved. This file is offered as-is, without any warranty.

Contributions are welcome. See https://savannah.gnu.org/maintenance/fsf/.

[[!toc levels=4]]

Nagios

Nagios is one of our monitoring solutions.

Useful commands for Nagios and check_mk

Login to Klaxon

ssh root@klaxon.fsf.org

Refresh DNS Cache

check_mk --update-dns-cache
check_mk -U && service nagios3 reload

Update Nagios' expectations of the presence and value of check_mk results:

# Inventory
check_mk -II
# Generate nagios config
check_mk -U
# Apply config
/etc/init.d/nagios3 reload

Update inventory for only one host

check_mk -II vcs1.savannah.gnu.org

View a machine's mk

cd /etc/check_mk/conf.d/machines/
less vcs1.savannah.gnu.org

Update email queue thresholds and other variables

vim /etc/check_mk/conf.d/custom.mk
check_mk -U
service nagios3 reload

Adding a new URL Content service check

There is a custom script to generate URL checks /usr/local/bin/nagios-website-add.sh

Examples:

/usr/local/bin/nagios-website-add.sh -s -6 -C '/irc/!The Lounge' -w irc.libreplanet.org -h irc1p.libreplanet.org
/usr/local/bin/nagios-website-add.sh -s -6 -A -C '/!We prioritize your privacy' -w jitsi.member.fsf.org -h jitsi1p.fsf.org

See /etc/nagios3/conf.d/web_services.cfg for some more examples.

Adding a custom service check script

Add a script to /usr/lib/check_mk_agent/local/. See ansible for examples. For it to go into effect:

For a single host:

check_mk -II FQDN; service nagios3 reload

Or to fully reload everything:

check_mk -II;check_mk -U; service nagios3 reload

Add a new remote check for all hosts

https://docs.checkmk.com/latest/en/cmc_migration.html

vim /etc/check_mk/main.mk

custom_checks += [
  ({
      'command_name':        'check-down-firewall',
      'service_description': 'Check for down firewalls',
      'command_line':        '/usr/local/bin/check-down-firewall $HOSTNAME$ 1234',
      'has_perfdata':        False,
  },
  [],
  ALL_HOSTS )]

Useful web interface tips

Disabling a notification

Visit https://klaxon.fsf.org/nagios3/ and login.
On the left column, click on Host Groups.
Click on a specific VM. In my example, I wanted to silence docker alerts from emba-runner.gnu.org which gave 1,482 unhelpful alerts in two months.
Click on the service that is giving unhelpful alerts. In my example, I wanted to silence Interfaces 3, 4, and 5.
In the right column, click on Disable notifications for this service To reverse the change, click on Enable notifications for this service.
No more alert notifications for that!

Disabling an alert via Nagstamon

Right-click on an active alert, and choose "acknowledge". If you want the ack to persist after it's okay then comes back / flaps, then check the "sticky" option in the dialog. Click OK.

Disabling alerts types for a host

Alerts can be disabled by type, per host, by editing the ignored_services section at /etc/check_mk/conf.d/custom.mk on klaxon

On-call scripts

See klaxon.