Featured image of post Using Icinga2 and Ansible: one playbook to monitor them all!

Using Icinga2 and Ansible: one playbook to monitor them all!

Have you ever thought of way to monitor new hosts without having to spend much time adding the NRPE plugins

Have you ever thought of way to monitor new hosts without having to spend much time adding the NRPE plugins, command check definitions and other custom configurations manually on each of them?

No problem, I have just faced that very same situation. Also, got tired of it pretty quickly. So how should we solve it?

The solution we are providing here is pretty simple: apply an Icinga2 monitoring template to a brand new, fresh installed machine thanks to Ansible.

NOTICE: for the examples provided we will be using Debian-like distros, so if yours is different you may have to adapt those affected parts, such as package manager related commands, specific Ansible plugins and so on.

Installing dependencies

The only things we need to configure on our machine are the SSH keys (so we can apply our playbooks normally), and to install the sudo package.

For the SSH keys you can copy your public key with the following command:

ssh-copy-id -i path/to/your/key ${YOUR_USERNAME}@${YOUR_NEW_MACHINE}

In case you don’t have a key set up, you can create one as follows:

ssh-keygen -t rsa

Then fill in the information the shell is going to prompt for. After that, from inside your new machine, run the following as root (or using sudo):

apt-get update
apt-get install sudo -y

When the package is installed, be sure to run visudo and configure the user you will be using properly, otherwise the Ansible steps may fail. If you are using root user directly (which I don’t recommend, insert security disclaimer here) these last steps are not needed at all.

Due to time constraints we’re not going to cover the Icinga2 installation on this article.

We’re going to assume you’ve already set it up and it’s running properly.

Setting up Ansible

From the machine you’re going to be using for the Ansible deployments, you will need to have a directory structure such as this one:

|-- inventories
|   `-- my_machines
|       `-- hosts
|-- playbooks
    |-- icinga_add_host.yml
    |-- install_nrpe_client.yml
    |-- files
        |-- nrpe
            |-- nrpe.cfg.template
            `-- nrpe_local.cfg.template

In this configuration, two playbooks are set up, the first one install_nrpe_client.yml has the following content:

- hosts: "{{ host }}"


    - name: "Install NRPE client and monitoring plugins"
        pkg: ["nagios-nrpe-server", "monitoring-plugins", "nagios-plugins-contrib"]
        force_apt_get: yes
        update_cache: yes
        state: present
      tags: install

    - name: Copy NRPE service core files
      copy: src={{ item.src }} dest={{ item.dest }}
        - { src: 'nrpe/nrpe_local.cfg.template', dest: '/etc/nagios/nrpe_local.cfg' }
        - { src: 'nrpe/nrpe.cfg.template', dest: '/etc/nagios/nrpe.cfg' }
      tags: copy

    - name: Restart nagios-nrpe-server service
      service: name=nagios-nrpe-server state=restarted
      tags: restart

The playbook just goes on the machine and performs the needed package set up to run the monitoring services for that machine.

The second one, icinga_add_host.yml, has the following content:



    - name: "Add host to Icinga"
        dest: /etc/icinga2/conf.d/homelab/{{ host }}.conf
        content: |
          object Host "{{ host }}" {
            import "generic-host"
            address = "{{ host }}"
            vars.os = "Linux"
            vars.disks["disk /"] = {
              disk_partitions = "/"
            vars.notification["mail"] = {
              groups = [ "icingaadmins" ]
      tags: add-host-template

    - name: "Restart Icinga2 service"
      service: name=icinga2 state=restarted
      tags: restart

Note that we’re using the default way of adding a host in Icinga2, of course this can be further extended by adding multiple and new commands and services.

Finally, there is the inventories/my_machines/hosts file, which should only have one line for now:


Keep this in mind for later.

Setting up Icinga2

As you may have already seen, there are two other files in this setup, both templates are for the Icinga2 service configuration itself and command check definitions.

The file nrpe.cfg.template, is almost a clone of the default nrpe.cfg, as the only meaningful change to get things working is the allowed_hosts variable.

Where you must declare the address or FQDN of your Icinga2 server, so you can leave it intact except for that one bit (seriously don’t forget this).

The rest is just matter of custom preferences.

Also, the nrpe_local.cfg.template is the default file I chose to host all my custom command checks. However you can get things working too just by copy-pasting the ones declared in the default nrpe.cfg file or directly uncommenting them right there.

If you choose to copy them, it would end up looking something like this:

command[check_users]=/usr/lib/nagios/plugins/check_users -w 5 -c 10
command[check_load]=/usr/lib/nagios/plugins/check_load -r -w .15,.10,.05 -c .30,.25,.20
command[check_hda1]=/usr/lib/nagios/plugins/check_disk -w 20% -c 10% -p /dev/hda1
command[check_zombie_procs]=/usr/lib/nagios/plugins/check_procs -w 5 -c 10 -s Z
command[check_total_procs]=/usr/lib/nagios/plugins/check_procs -w 150 -c 200

command[check_users]=/usr/lib/nagios/plugins/check_users $ARG1$
command[check_load]=/usr/lib/nagios/plugins/check_load $ARG1$
command[check_disk]=/usr/lib/nagios/plugins/check_disk $ARG1$
command[check_swap]=/usr/lib/nagios/plugins/check_swap $ARG1$
command[check_cpu_stats]=/usr/lib/nagios/plugins/check_cpu_stats.sh $ARG1$
command[check_mem]=/usr/lib/nagios/plugins/custom_check_mem -n $ARG1$

command[check_init_service]=sudo /usr/lib/nagios/plugins/check_init_service $ARG1$
command[check_services]=/usr/lib/nagios/plugins/check_services -p $ARG1$


command[check_procs]=/usr/lib/nagios/plugins/check_procs $ARG1$

### OPEN FILES ###
command[check_open_files]=/usr/lib/nagios/plugins/check_open_files.pl $ARG1$

command[check_netstat]=/usr/lib/nagios/plugins/check_netstat.pl -p $ARG1$ $ARG2$

Running the playbooks

At this point everything should be ready to start monitoring your new machine, so the way you would do this is by running the following commands from your super amazing laptop:

ansible-playbook -i ansible/inventories/my_hosts/hosts ansible/playbooks/nrpe_client.yml --extra-vars "host=${new_machine_hostname}"
ansible-playbook -i ansible/inventories/my_hosts/hosts ansible/playbooks/icinga_add_host.yml --extra-vars "host=${new_machine_hostname}"

IMPORTANT: The “new_machine_hostname” value must coincide with the one set in the hosts inventory file (I told you to keep that in mind for a reason!), else it will return an error message.


As you can see, it can be pretty easy to monitor new hosts with Icinga2 when it comes to get things done by using automation software, and this is just a slight example.

Hope you find it useful!

Built with Hugo