Have you ever thought of way to monitor new hosts without having to spend much time adding the NRPE plugins, command check definitions and other custom configurations manually on each of them?
No problem, I have just faced that very same situation. Also, got tired of it pretty quickly. So how should we solve it?
The solution we are providing here is pretty simple: apply an Icinga2 monitoring template to a brand new, fresh installed machine thanks to Ansible.
NOTICE: for the examples provided we will be using Debian-like distros, so if yours is different you may have to adapt those affected parts, such as package manager related commands, specific Ansible plugins and so on.
Installing dependencies
The only things we need to configure on our machine are the SSH keys (so we can apply our playbooks normally), and to install the sudo package.
For the SSH keys you can copy your public key with the following command:
ssh-copy-id -i path/to/your/key ${YOUR_USERNAME}@${YOUR_NEW_MACHINE}
In case you don’t have a key set up, you can create one as follows:
ssh-keygen -t rsa
Then fill in the information the shell is going to prompt for. After that, from inside your new machine, run the following as root (or using sudo):
apt-get update
apt-get install sudo -y
When the package is installed, be sure to run visudo and configure the user you will be using properly, otherwise the Ansible steps may fail. If you are using root user directly (which I don’t recommend, insert security disclaimer here) these last steps are not needed at all.
Due to time constraints we’re not going to cover the Icinga2 installation on this article.
We’re going to assume you’ve already set it up and it’s running properly.
Setting up Ansible
From the machine you’re going to be using for the Ansible deployments, you will need to have a directory structure such as this one:
|-- inventories
| `-- my_machines
| `-- hosts
|-- playbooks
|-- icinga_add_host.yml
|-- install_nrpe_client.yml
|-- files
|-- nrpe
|-- nrpe.cfg.template
`-- nrpe_local.cfg.template
In this configuration, two playbooks are set up, the first one install_nrpe_client.yml has the following content:
---
- hosts: "{{ host }}"
tasks:
- name: "Install NRPE client and monitoring plugins"
apt:
pkg: ["nagios-nrpe-server", "monitoring-plugins", "nagios-plugins-contrib"]
force_apt_get: yes
update_cache: yes
state: present
tags: install
- name: Copy NRPE service core files
copy: src={{ item.src }} dest={{ item.dest }}
with_items:
- { src: 'nrpe/nrpe_local.cfg.template', dest: '/etc/nagios/nrpe_local.cfg' }
- { src: 'nrpe/nrpe.cfg.template', dest: '/etc/nagios/nrpe.cfg' }
tags: copy
- name: Restart nagios-nrpe-server service
service: name=nagios-nrpe-server state=restarted
tags: restart
The playbook just goes on the machine and performs the needed package set up to run the monitoring services for that machine.
The second one, icinga_add_host.yml, has the following content:
---
- hosts: ${YOUR_ICINGA2_SERVER}
tasks:
- name: "Add host to Icinga"
copy:
dest: /etc/icinga2/conf.d/homelab/{{ host }}.conf
content: |
object Host "{{ host }}" {
import "generic-host"
address = "{{ host }}"
vars.os = "Linux"
vars.disks["disk /"] = {
disk_partitions = "/"
}
vars.notification["mail"] = {
groups = [ "icingaadmins" ]
}
}
tags: add-host-template
- name: "Restart Icinga2 service"
service: name=icinga2 state=restarted
tags: restart
Note that we’re using the default way of adding a host in Icinga2, of course this can be further extended by adding multiple and new commands and services.
Finally, there is the inventories/my_machines/hosts file, which should only have one line for now:
new_machine_hostname
Keep this in mind for later.
Setting up Icinga2
As you may have already seen, there are two other files in this setup, both templates are for the Icinga2 service configuration itself and command check definitions.
The file nrpe.cfg.template, is almost a clone of the default nrpe.cfg, as the only meaningful change to get things working is the allowed_hosts variable.
Where you must declare the address or FQDN of your Icinga2 server, so you can leave it intact except for that one bit (seriously don’t forget this).
The rest is just matter of custom preferences.
Also, the nrpe_local.cfg.template is the default file I chose to host all my custom command checks. However you can get things working too just by copy-pasting the ones declared in the default nrpe.cfg file or directly uncommenting them right there.
If you choose to copy them, it would end up looking something like this:
command[check_users]=/usr/lib/nagios/plugins/check_users -w 5 -c 10
command[check_load]=/usr/lib/nagios/plugins/check_load -r -w .15,.10,.05 -c .30,.25,.20
command[check_hda1]=/usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /dev/hda1
command[check_zombie_procs]=/usr/lib/nagios/plugins/check_procs -w 5 -c 10 -s Z
command[check_total_procs]=/usr/lib/nagios/plugins/check_procs -w 150 -c 200
### MISC SYSTEM METRICS ###
command[check_users]=/usr/lib/nagios/plugins/check_users $ARG1$
command[check_load]=/usr/lib/nagios/plugins/check_load $ARG1$
command[check_disk]=/usr/lib/nagios/plugins/check_disk $ARG1$
command[check_swap]=/usr/lib/nagios/plugins/check_swap $ARG1$
command[check_cpu_stats]=/usr/lib/nagios/plugins/check_cpu_stats.sh $ARG1$
command[check_mem]=/usr/lib/nagios/plugins/custom_check_mem -n $ARG1$
### GENERIC SERVICES ###
command[check_init_service]=sudo /usr/lib/nagios/plugins/check_init_service $ARG1$
command[check_services]=/usr/lib/nagios/plugins/check_services -p $ARG1$
### SYSTEM UPDATES ###
command[check_yum]=/usr/lib/nagios/plugins/check_yum
command[check_apt]=/usr/lib/nagios/plugins/check_apt
### PROCESSES ###
command[check_all_procs]=/usr/lib/nagios/plugins/custom_check_procs
command[check_procs]=/usr/lib/nagios/plugins/check_procs $ARG1$
### OPEN FILES ###
command[check_open_files]=/usr/lib/nagios/plugins/check_open_files.pl $ARG1$
### NETWORK CONNECTIONS ###
command[check_netstat]=/usr/lib/nagios/plugins/check_netstat.pl -p $ARG1$ $ARG2$
Running the playbooks
At this point everything should be ready to start monitoring your new machine, so the way you would do this is by running the following commands from your super amazing laptop:
ansible-playbook -i ansible/inventories/my_hosts/hosts ansible/playbooks/nrpe_client.yml --extra-vars "host=${new_machine_hostname}"
ansible-playbook -i ansible/inventories/my_hosts/hosts ansible/playbooks/icinga_add_host.yml --extra-vars "host=${new_machine_hostname}"
IMPORTANT: The “new_machine_hostname” value must coincide with the one set in the hosts inventory file (I told you to keep that in mind for a reason!), else it will return an error message.
Conclusion
As you can see, it can be pretty easy to monitor new hosts with Icinga2 when it comes to get things done by using automation software, and this is just a slight example.
Hope you find it useful!