The following is the GNU All-permissive License as recommended in https://www.gnu.org/licenses/license-recommendations.en.html

Copying and distribution of this file, with or without modification, are permitted in any medium without royalty provided the copyright notice and this notice are preserved. This file is offered as-is, without any warranty.

Contributions are welcome. See https://savannah.gnu.org/maintenance/fsf/.

Prometheus

Use cases:

Alerting (replace Nagios & Checkmk)
Explore data, eg to debug problems
Create Munin-like graphs, maybe eventually replace Munin

Prometheus UI is at https://prom.fsf.org/graph and https://prom.office.fsf.org/graph. When not in office or on the VPN https://valis.gnu.org/graph is an alternative for https://prom.office.fsf.org/graph, if it does not work.

The password is in spd. Alertmanager has the same pass.

Alertmanager UI is at https://prom.fsf.org:9095/#/alerts and https://prom.office.fsf.org:9095/#/alerts. When not in office or on the VPN, https://valis.gnu.org:9095/#/alerts is an alternative for https://prom.office.fsf.org:9095/#/alerts, if it does not work.

Blackbox exporter UI is at https://prom.fsf.org:9116 and https://prom.office.fsf.org:9116

Don't silence an alert forever, we will forget about it.

Alertmanager UI is mostly useful for temporarily silencing alerts. amtool cli program is more advanced and the main documented way of interacting with alertmanager.

To silence all alerts temporarily, add a matcher like x!="1"

Host based differentiation of alert priority: This is configured via ansible inventory [priority_high:children], which is a list of groups. One of which is [priority_high_hosts], in order to directly add hosts.

Initial brain dump meeting notes pad: https://etherpad.fsf.org/p/2pRPC_jizUwnEVg5kam4

amtool configuration

The good basic example usage: https://github.com/prometheus/alertmanager

~/.config/amtool/config.yml :

alertmanager.url: "https://admin:PUT_PASSWORD_FROM_SPD_HERE@prom.fsf.org:9095"
output: extended
comment_required: false

amall() {
  echo "$(tput setaf 5 2>/dev/null ||:)█ coresite █$(tput sgr0 2>/dev/null||:)"
  amfsf "$@"
  echo "$(tput setaf 5 2>/dev/null ||:)█ office █$(tput sgr0 2>/dev/null||:)"
  amoffice "$@"
}
amallq() { # amall quiet
  amfsf "$@"
  amoffice "$@"
}
amfsf() {
  sedi -r '/alertmanager.url/s/@prom.office/@prom/' ~/.config/amtool/config.yml
  amtool "$@"
}
amoffice() {
  sedi -r '/alertmanager.url/s/@prom.fsf/@prom.office.fsf/' ~/.config/amtool/config.yml
  amtool "$@"
}
amls() {
  amall silence query "$@"
}
amrmall() {
  # note: not sure if quoting of this arg is correct
  amfsf silence expire "$(amfsf silence query -q)"
  amoffice silence expire "$(amoffice silence query -q)"
}

Good Documentation

Official docs
https://www.robustperception.io/blog/
https://utcc.utoronto.ca/~cks/space/blog/sysadmin/

Useful for understanding Blackbox relabeling:

Links:

GitHub

Debugging blackbox grep failure

Curl can almost always get the same response as blackbox exporter probe did. Append the url to the following:

curl -H "User-Agent: Blackbox Exporter/0.24.0" -4
# or for ipv6
curl -H "User-Agent: Blackbox Exporter/0.24.0" -6

You can view debug logs via https://prom.fsf.org:9116/

You can also construct a probe url and manually set it off and get the logs

https://prom.fsf.org:9116/probe?target=https://h-node.org&module=grep4_h-node.org&debug=true

# one liner, append target and module name as args to the following:
bburl() { echo "https://prom.fsf.org:9116/probe?target=$1&module=$2&debug=true"; }; bburl

Background, future improvements

In 2023, Ian estimated we would need about 120gb of storage to store data on CoreSite hosts for 200 days.

todo:

see if we can make critical alerts be black in nagstamon
Monitor the office network being down from the data center.
Update valis & office WiFi to new LibreCMC version and add node exporter
Setup alerts that are at least as comprehensive as Nagios
monitor the office network being down from the data center.
migrate roles/prom/files/prometheus0p.fsf.org/etc/prometheus/file_sd/blackbox_https/ansible.yml to be grep tests then remove
Make a disk filling alert to be 3 days out
update valis & office wifi to new librecmc version and add node exporter
Setup other exporters like MySQL
Get more disk space for the VM, extend storage to 200 days.
Setup Munin-like graphs

Notes on manual deployment steps for machines not ansible managed

To check what is listening on 9100:

ss -lptn | grep 9100

vim /etc/default/iptables-rules or depending on machine vim /etc/default/iptables ```

Prometheus node exporter. whole subnet to allow for ip changes

-A INPUT -m tcp -p tcp --src 209.51.188.0/24 --dport 9100 -j ACCEPT -A state-check -m tcp -p tcp --src 209.51.188.0/24 --dport 9100 -j ACCEPT

Prometheus from office public ip

-A INPUT -m tcp -p tcp --src 74.94.156.210 --dport 9100 -j ACCEPT ```

If the machine has an IPv6 address, node exporter will only listen on that by default. The chain name may vary, examples below.

vim /etc/default/ip6tables-rules -A state-check -p tcp -m tcp --src 2001:470:142:5::115 --dport 9100 -j ACCEPT -A input_block -p tcp -m tcp --src 2001:470:142:5::115 --dport 9100 -j ACCEPT

For office network: ```

prometheus node exporter

-A INPUT -m tcp -p tcp --src 192.168.0.56 --dport 9100 -j ACCEPT ```

On Savannah machines

/etc/shorewall6/rules

ACCEPT  net:2001:470:142:5::115     fw     tcp     9100  # FSF Prometheus

cd /etc
cat shorewall6/rules
sed -i '/Nagios/a ACCEPT net:2001:470:142:5::115     fw     tcp     9100  # FSF Prometheus' /etc/shorewall6/rules
i diff shorewall*
shorewall6 safe-restart
git add shorewall*
git commit -m 'adding FSF Prometheus'

IPv4 stuff if needed

sed -i '/Nagios/a ACCEPT net:209.51.188.115/24   fw      tcp     9100  # FSF prometheus' /etc/shorewall/rules
i diff shorewall*
shorewall safe-restart

/etc/shorewall/rules

ACCEPT  net:209.51.188.115/24   fw      tcp     9100  # FSF Prometheus

Exporter security background

The only real risk of exposing node-exporter to the internet is that it could expose that we are using outdated software.

The other known issue with exposing node-exporter is potential for DOS by sending lots of GETs, but we expose lots of we pages that cause much more processing than node-exporter, so that is not a serious issue.

Other exporters could have sensitive data, evaluate each one as we adopt it. There was a Kubernetes exporter that had sensitive info and was used nefariously. The Prometheus upstream strangely just says "you can restrict access to it if you want", but doesn't recommend you do, and then just implicitly recommends that you don't by not doing much to document anything about it. The main way we restrict it is via tls. Secondary is the firewall. Apache/nginx tls + basic auth may be the best option for exporters other than node-exporter.

How to curl a node-exporter from the Prometheus server. alter host= to curl a different host

host=localhost; curl --cert /etc/prometheus/ssl/prometheus_cert.pem --key /etc/prometheus/ssl/prometheus_key.pem --cacert /etc/prometheus/ssl/prom_node_cert.pem --resolve prom_node:9100:$host -v https://prom_node:9100/metrics

How certs were initially generated:

openssl req -x509 -newkey rsa:2048 -keyout ${keydir}prom_node_key.pem -out prom_node_cert.pem -days 29220 -nodes -subj /commonName=prom_node/ -addext "subjectAltName=DNS:prom_node"

openssl req -x509 -newkey rsa:2048 -keyout ${keydir}prometheus_key.pem -out prometheus_cert.pem -days 29220 -nodes -subj /commonName=prometheus/ -addext "subjectAltName=DNS:prometheus"

Background on notification methods

Highly desirable alerting properties:

Ability to silence all notifications on a phone except for alerts.
Sent as directly as possible from prometheus server to phone & computer so failures in other systems don't disable alerts.
Not intermixed with security sensitive communication stream.

Evaluation of options:

Currently used options:

xmpp server running on prometheus server: https://github.com/trazfr/prometheus-xmpp-alerting there are 3 xmpp apps in fdroid conversations, cheogram, and monocles chat, latter 2 are forks of cheogram. Even though none of them can be used for general xmpp communication and silence all notifications except for alerts, you can dedicate one app to only receiving alerts.

Nagstamon: this works fine but only for a computer.

Potentially good options but would need work to implement:

Imap from the prometheus server: Needs investigation about notification if you want to receive other mail. Possible we could build something like k9mail but renamed to get a separate app with its own notifications.

Liberachat IRC Bot: Not totally direct. Useful for volunteer transparency. Revolution IRC allows silencing notifications except for custom rules.

Bot send to self-hosted irc server on prometheus server: Would have to learn how to run an irc server.

Not great options but they could work:

gotify: The gotify android app can only connect to one gotify server. This would mean we would have to send multiple data centers alerts to one gotify server, making that a single point of failure. There is a github issue for this years old, the creator said he won't do it because it isn't useful to him personally. Before I knew this, I got it running, it doesn't have a good message template by default. Also, only has an android client.

SMS alerts via email bridge: We started with this. These work ok, but they rely on having non-voip sms. And the most minimal builtin template for them is far too verbose.

SMTP: Not sent directly. This requires sysadmins to have some outside mail server and account setup that will only receive alerts. I have this, but other sysadmins don't. I could provide it to them, but that also makes it somewhat prone to temporary failures and downtime.

Bad rejected options:

xmpp server on prometheus host using https://github.com/trazfr/prometheus-xmpp-alerting This has only like 75 lines of code. Run the version in t11 with debug flag, its logs just say connecting, but tcpdump shows nothing. I looked long and hard to figure out what was going wrong, to no avail.

Background on why we are avoiding Grafana for graphs

Grafana:

Pros:

More popular

Cons:

Graphs are setup by clicking in a web GUI and stored in undocumented JSON format.
GUI has bugs on GNU/Linux abrowser
Uses more resources than Prometheus
Open core business model

Prometheus

Pros:

Graph definitions uses PromQL, which is needed for setting up alerts. helps you learn and remember
Graph definitions are text files

Cons:

Graphs often take longer to create than Grafana

Nagstamon

I wrote documentation for configuring our Prometheus instances with Nagstamon, but it got lost in a gluestick malfunction.

Filter pending Attempts

Prometheus announces when it is actively checking something which is noise for Nagstamon.

Click on the Filters button.
Click on the Regular expression for attempt checkbox.
Write pending in the text box.
Click on the OK button.