<?xml version="1.0" encoding="UTF-8"?>
<rss xmlns:atom="http://www.w3.org/2005/Atom" version="2.0">
    <channel>
      <title>moritzvd.eu - Nagios</title>
      <link>https://moritzvd.eu</link>
      <description></description>
      <generator>Zola</generator>
      <language>en</language>
      <atom:link href="https://moritzvd.eu/tags/nagios/rss.xml" rel="self" type="application/rss+xml"/>
      <lastBuildDate>Wed, 06 May 2020 12:47:00 +0000</lastBuildDate>
      <item>
          <title>Monitoring Traefik 2 with Icinga</title>
          <pubDate>Wed, 06 May 2020 12:47:00 +0000</pubDate>
          <author>Unknown</author>
          <link>https://moritzvd.eu/posts/monitor-traefik-icinga/</link>
          <guid>https://moritzvd.eu/posts/monitor-traefik-icinga/</guid>
          <description xml:base="https://moritzvd.eu/posts/monitor-traefik-icinga/">&lt;p&gt;&lt;img src=&quot;&#x2F;images&#x2F;traefik-icinga-banner.png&quot; alt=&quot;Monitoring Traefik 2 with Icinga&quot; &#x2F;&gt;&lt;&#x2F;p&gt;
&lt;p&gt;Checking if a specific website or web application is alive and well is a classic task of monitoring software. I monitor personal web services with good old Icinga2. Last year I wrote a small Python script that runs on a Icinga2 master or satellite node that queries the Traefik API to automate the generation of Icinga2 rules. For me this was a nice and useful exercise to learn how to use REST APIs together with Python for system administration. In the process of &lt;a href=&quot;https:&#x2F;&#x2F;moritzvd.eu&#x2F;upgrade-traefik-2&#x2F;&quot;&gt;upgrading Traefik 1.7 to 2.0&lt;&#x2F;a&gt; I also had to modify this script.&lt;&#x2F;p&gt;
&lt;p&gt;So without further ado lets jump right in. The way my script works:&lt;&#x2F;p&gt;
&lt;p&gt;It fetches the exposed Traefik Routers with their coresponding hostnames and services and creates Icinga2 &lt;a href=&quot;https:&#x2F;&#x2F;icinga.com&#x2F;docs&#x2F;icinga2&#x2F;latest&#x2F;doc&#x2F;03-monitoring-basics&#x2F;#apply-rules&quot;&gt;apply rules&lt;&#x2F;a&gt; that use the &lt;a href=&quot;https:&#x2F;&#x2F;icinga.com&#x2F;docs&#x2F;icinga2&#x2F;latest&#x2F;doc&#x2F;10-icinga-template-library&#x2F;#http&quot;&gt;HTTP check&lt;&#x2F;a&gt;. The script is designed to run as a nightly cronjob so that new services that should be monitored are added automatically.&lt;&#x2F;p&gt;
&lt;h2 id=&quot;requirements&quot;&gt;Requirements&lt;a class=&quot;anchor&quot; aria-hidden=&quot;true&quot; href=&quot;#requirements&quot; hidden=&quot;&quot;&gt;#&lt;&#x2F;a&gt;
&lt;&#x2F;h2&gt;
&lt;ul&gt;
&lt;li&gt;A working Traefik v2.x reverse proxy with it&#x27;s API exposed&lt;&#x2F;li&gt;
&lt;li&gt;Icinga2 monitoring server&lt;&#x2F;li&gt;
&lt;&#x2F;ul&gt;
&lt;p&gt;Before proceeding please make shure you can access Traefik&#x27;s API. For example in this way:&lt;&#x2F;p&gt;
&lt;pre data-lang=&quot;sh&quot; style=&quot;background-color:#2b303b;color:#c0c5ce;&quot; class=&quot;language-sh &quot;&gt;&lt;code class=&quot;language-sh&quot; data-lang=&quot;sh&quot;&gt;&lt;span style=&quot;color:#bf616a;&quot;&gt;curl -s --user&lt;&#x2F;span&gt;&lt;span&gt; admin:passw0rd http:&#x2F;&#x2F;example.lan:8080&#x2F;api&#x2F;http&#x2F;routers | &lt;&#x2F;span&gt;&lt;span style=&quot;color:#bf616a;&quot;&gt;jq
&lt;&#x2F;span&gt;&lt;span style=&quot;color:#96b5b4;&quot;&gt;[
&lt;&#x2F;span&gt;&lt;span&gt;  {
&lt;&#x2F;span&gt;&lt;span&gt;    &amp;quot;&lt;&#x2F;span&gt;&lt;span style=&quot;color:#a3be8c;&quot;&gt;entryPoints&lt;&#x2F;span&gt;&lt;span&gt;&amp;quot;&lt;&#x2F;span&gt;&lt;span style=&quot;color:#bf616a;&quot;&gt;:&lt;&#x2F;span&gt;&lt;span&gt; [
&lt;&#x2F;span&gt;&lt;span&gt;      &amp;quot;&lt;&#x2F;span&gt;&lt;span style=&quot;color:#a3be8c;&quot;&gt;dashboard&lt;&#x2F;span&gt;&lt;span&gt;&amp;quot;
&lt;&#x2F;span&gt;&lt;span&gt;    &lt;&#x2F;span&gt;&lt;span style=&quot;color:#bf616a;&quot;&gt;],
&lt;&#x2F;span&gt;&lt;span&gt;    &amp;quot;&lt;&#x2F;span&gt;&lt;span style=&quot;color:#a3be8c;&quot;&gt;middlewares&lt;&#x2F;span&gt;&lt;span&gt;&amp;quot;&lt;&#x2F;span&gt;&lt;span style=&quot;color:#bf616a;&quot;&gt;:&lt;&#x2F;span&gt;&lt;span&gt; [
&lt;&#x2F;span&gt;&lt;span&gt;      &amp;quot;&lt;&#x2F;span&gt;&lt;span style=&quot;color:#a3be8c;&quot;&gt;dashboard-auth@file&lt;&#x2F;span&gt;&lt;span&gt;&amp;quot;
&lt;&#x2F;span&gt;&lt;span&gt;    &lt;&#x2F;span&gt;&lt;span style=&quot;color:#bf616a;&quot;&gt;],
&lt;&#x2F;span&gt;&lt;span&gt;    &lt;&#x2F;span&gt;&lt;span style=&quot;color:#bf616a;&quot;&gt;...
&lt;&#x2F;span&gt;&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;h2 id=&quot;installation&quot;&gt;Installation&lt;a class=&quot;anchor&quot; aria-hidden=&quot;true&quot; href=&quot;#installation&quot; hidden=&quot;&quot;&gt;#&lt;&#x2F;a&gt;
&lt;&#x2F;h2&gt;
&lt;p&gt;Run all this steps on a Icinga 2 master or satellite node.&lt;&#x2F;p&gt;
&lt;h3 id=&quot;download-script-and-install-script&quot;&gt;Download script and install script&lt;a class=&quot;anchor&quot; aria-hidden=&quot;true&quot; href=&quot;#download-script-and-install-script&quot; hidden=&quot;&quot;&gt;#&lt;&#x2F;a&gt;
&lt;&#x2F;h3&gt;
&lt;pre data-lang=&quot;sh&quot; style=&quot;background-color:#2b303b;color:#c0c5ce;&quot; class=&quot;language-sh &quot;&gt;&lt;code class=&quot;language-sh&quot; data-lang=&quot;sh&quot;&gt;&lt;span style=&quot;color:#bf616a;&quot;&gt;git&lt;&#x2F;span&gt;&lt;span&gt; clone https:&#x2F;&#x2F;github.com&#x2F;movd&#x2F;traefik_icinga_check
&lt;&#x2F;span&gt;&lt;span style=&quot;color:#bf616a;&quot;&gt;sudo -u&lt;&#x2F;span&gt;&lt;span&gt; nagios mkdir &#x2F;etc&#x2F;icinga2&#x2F;cronjobs
&lt;&#x2F;span&gt;&lt;span style=&quot;color:#bf616a;&quot;&gt;chmod&lt;&#x2F;span&gt;&lt;span&gt; +x traefik_icinga_check&#x2F;traefik2_to_icinga.py
&lt;&#x2F;span&gt;&lt;span style=&quot;color:#bf616a;&quot;&gt;sudo&lt;&#x2F;span&gt;&lt;span&gt; cp traefik_icinga_check&#x2F;traefik2_to_icinga.py  &#x2F;etc&#x2F;icinga2&#x2F;cronjobs&#x2F;
&lt;&#x2F;span&gt;&lt;span style=&quot;color:#bf616a;&quot;&gt;sudo&lt;&#x2F;span&gt;&lt;span&gt; chown nagios:nagios &#x2F;etc&#x2F;icinga2&#x2F;cronjobs&#x2F;traefik2_to_icinga.py
&lt;&#x2F;span&gt;&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;h3 id=&quot;set-up-parameters-via-env&quot;&gt;Set up parameters via .env&lt;a class=&quot;anchor&quot; aria-hidden=&quot;true&quot; href=&quot;#set-up-parameters-via-env&quot; hidden=&quot;&quot;&gt;#&lt;&#x2F;a&gt;
&lt;&#x2F;h3&gt;
&lt;pre data-lang=&quot;sh&quot; style=&quot;background-color:#2b303b;color:#c0c5ce;&quot; class=&quot;language-sh &quot;&gt;&lt;code class=&quot;language-sh&quot; data-lang=&quot;sh&quot;&gt;&lt;span style=&quot;color:#bf616a;&quot;&gt;$&lt;&#x2F;span&gt;&lt;span&gt; sudo&lt;&#x2F;span&gt;&lt;span style=&quot;color:#bf616a;&quot;&gt; -u&lt;&#x2F;span&gt;&lt;span&gt; nagios touch &#x2F;etc&#x2F;icinga2&#x2F;cronjobs&#x2F;.env
&lt;&#x2F;span&gt;&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;p&gt;Open and insert parameters &lt;code&gt;&#x2F;etc&#x2F;icinga2&#x2F;cronjobs&#x2F;.env&lt;&#x2F;code&gt; (same as Traefik dashboard)&lt;&#x2F;p&gt;
&lt;pre data-lang=&quot;sh&quot; style=&quot;background-color:#2b303b;color:#c0c5ce;&quot; class=&quot;language-sh &quot;&gt;&lt;code class=&quot;language-sh&quot; data-lang=&quot;sh&quot;&gt;&lt;span style=&quot;color:#bf616a;&quot;&gt;TRAEFIK_API_HOSTNAME&lt;&#x2F;span&gt;&lt;span&gt;=&amp;#39;&lt;&#x2F;span&gt;&lt;span style=&quot;color:#a3be8c;&quot;&gt;example.lan:8000&lt;&#x2F;span&gt;&lt;span&gt;&amp;#39;
&lt;&#x2F;span&gt;&lt;span style=&quot;color:#bf616a;&quot;&gt;TRAEFIK_USERNAME&lt;&#x2F;span&gt;&lt;span&gt;=&amp;#39;&lt;&#x2F;span&gt;&lt;span style=&quot;color:#a3be8c;&quot;&gt;admin&lt;&#x2F;span&gt;&lt;span&gt;&amp;#39;
&lt;&#x2F;span&gt;&lt;span style=&quot;color:#bf616a;&quot;&gt;TRAEFIK_PASSWORD&lt;&#x2F;span&gt;&lt;span&gt;=&amp;#39;&lt;&#x2F;span&gt;&lt;span style=&quot;color:#a3be8c;&quot;&gt;passw0rd&lt;&#x2F;span&gt;&lt;span&gt;&amp;#39;
&lt;&#x2F;span&gt;&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;h3 id=&quot;test-the-script-manually-by-printing-to-stdout&quot;&gt;Test the script manually by printing to STDOUT&lt;a class=&quot;anchor&quot; aria-hidden=&quot;true&quot; href=&quot;#test-the-script-manually-by-printing-to-stdout&quot; hidden=&quot;&quot;&gt;#&lt;&#x2F;a&gt;
&lt;&#x2F;h3&gt;
&lt;pre data-lang=&quot;sh&quot; style=&quot;background-color:#2b303b;color:#c0c5ce;&quot; class=&quot;language-sh &quot;&gt;&lt;code class=&quot;language-sh&quot; data-lang=&quot;sh&quot;&gt;&lt;span style=&quot;color:#bf616a;&quot;&gt;$&lt;&#x2F;span&gt;&lt;span&gt; sudo&lt;&#x2F;span&gt;&lt;span style=&quot;color:#bf616a;&quot;&gt; -u&lt;&#x2F;span&gt;&lt;span&gt; nagios &#x2F;etc&#x2F;icinga2&#x2F;cronjobs&#x2F;traefik2_to_icinga.py
&lt;&#x2F;span&gt;&lt;span style=&quot;color:#bf616a;&quot;&gt;apply&lt;&#x2F;span&gt;&lt;span&gt; Service &amp;quot;&lt;&#x2F;span&gt;&lt;span style=&quot;color:#a3be8c;&quot;&gt;example.com https apache-apache&lt;&#x2F;span&gt;&lt;span&gt;&amp;quot; {
&lt;&#x2F;span&gt;&lt;span&gt;                import &amp;quot;&lt;&#x2F;span&gt;&lt;span style=&quot;color:#a3be8c;&quot;&gt;generic-service&lt;&#x2F;span&gt;&lt;span&gt;&amp;quot;
&lt;&#x2F;span&gt;&lt;span&gt;                check_command = &amp;quot;&lt;&#x2F;span&gt;&lt;span style=&quot;color:#a3be8c;&quot;&gt;http&lt;&#x2F;span&gt;&lt;span&gt;&amp;quot;
&lt;&#x2F;span&gt;&lt;span&gt;                vars.http_address = &amp;quot;&lt;&#x2F;span&gt;&lt;span style=&quot;color:#a3be8c;&quot;&gt;example.com&lt;&#x2F;span&gt;&lt;span&gt;&amp;quot;
&lt;&#x2F;span&gt;&lt;span&gt;                vars.http_vhost = &amp;quot;&lt;&#x2F;span&gt;&lt;span style=&quot;color:#a3be8c;&quot;&gt;example.com&lt;&#x2F;span&gt;&lt;span&gt;&amp;quot;
&lt;&#x2F;span&gt;&lt;span&gt;
&lt;&#x2F;span&gt;&lt;span&gt;                vars.http_ssl = &amp;quot;&lt;&#x2F;span&gt;&lt;span style=&quot;color:#a3be8c;&quot;&gt;1&lt;&#x2F;span&gt;&lt;span&gt;&amp;quot;
&lt;&#x2F;span&gt;&lt;span&gt;                vars.http_sni = &amp;quot;&lt;&#x2F;span&gt;&lt;span style=&quot;color:#a3be8c;&quot;&gt;true&lt;&#x2F;span&gt;&lt;span&gt;&amp;quot;
&lt;&#x2F;span&gt;&lt;span&gt;
&lt;&#x2F;span&gt;&lt;span&gt;                vars.notification&lt;&#x2F;span&gt;&lt;span style=&quot;color:#b48ead;&quot;&gt;[&lt;&#x2F;span&gt;&lt;span&gt;&amp;quot;&lt;&#x2F;span&gt;&lt;span style=&quot;color:#a3be8c;&quot;&gt;mail&lt;&#x2F;span&gt;&lt;span&gt;&amp;quot;&lt;&#x2F;span&gt;&lt;span style=&quot;color:#b48ead;&quot;&gt;]&lt;&#x2F;span&gt;&lt;span&gt; = {
&lt;&#x2F;span&gt;&lt;span&gt;                    groups = &lt;&#x2F;span&gt;&lt;span style=&quot;color:#b48ead;&quot;&gt;[ &lt;&#x2F;span&gt;&lt;span&gt;&amp;quot;&lt;&#x2F;span&gt;&lt;span style=&quot;color:#a3be8c;&quot;&gt;icingaadmins&lt;&#x2F;span&gt;&lt;span&gt;&amp;quot; &lt;&#x2F;span&gt;&lt;span style=&quot;color:#b48ead;&quot;&gt;]
&lt;&#x2F;span&gt;&lt;span&gt;                }
&lt;&#x2F;span&gt;&lt;span&gt;...
&lt;&#x2F;span&gt;&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;h3 id=&quot;configure-the-host-that-runs-traefik&quot;&gt;Configure the Host that runs Traefik&lt;a class=&quot;anchor&quot; aria-hidden=&quot;true&quot; href=&quot;#configure-the-host-that-runs-traefik&quot; hidden=&quot;&quot;&gt;#&lt;&#x2F;a&gt;
&lt;&#x2F;h3&gt;
&lt;p&gt;Edit your Traefik host and add the &lt;code&gt;vars.services&lt;&#x2F;code&gt; array, so that it matches the rules set in &lt;code&gt;traefik2_to_icinga.py&lt;&#x2F;code&gt;&lt;&#x2F;p&gt;
&lt;pre data-lang=&quot;sh&quot; style=&quot;background-color:#2b303b;color:#c0c5ce;&quot; class=&quot;language-sh &quot;&gt;&lt;code class=&quot;language-sh&quot; data-lang=&quot;sh&quot;&gt;&lt;span style=&quot;color:#bf616a;&quot;&gt;object&lt;&#x2F;span&gt;&lt;span&gt; Host &amp;quot;&lt;&#x2F;span&gt;&lt;span style=&quot;color:#a3be8c;&quot;&gt;Reverse Proxy&lt;&#x2F;span&gt;&lt;span&gt;&amp;quot; {
&lt;&#x2F;span&gt;&lt;span&gt;  import &amp;quot;&lt;&#x2F;span&gt;&lt;span style=&quot;color:#a3be8c;&quot;&gt;generic-host&lt;&#x2F;span&gt;&lt;span&gt;&amp;quot;
&lt;&#x2F;span&gt;&lt;span&gt;  address = &amp;quot;&lt;&#x2F;span&gt;&lt;span style=&quot;color:#a3be8c;&quot;&gt;10.0.0.254&lt;&#x2F;span&gt;&lt;span&gt;&amp;quot;
&lt;&#x2F;span&gt;&lt;span&gt;  vars.os = &amp;quot;&lt;&#x2F;span&gt;&lt;span style=&quot;color:#a3be8c;&quot;&gt;Linux&lt;&#x2F;span&gt;&lt;span&gt;&amp;quot;
&lt;&#x2F;span&gt;&lt;span&gt;  vars.services = &lt;&#x2F;span&gt;&lt;span style=&quot;color:#b48ead;&quot;&gt;[&lt;&#x2F;span&gt;&lt;span&gt;&amp;quot;&lt;&#x2F;span&gt;&lt;span style=&quot;color:#a3be8c;&quot;&gt;traefik&lt;&#x2F;span&gt;&lt;span&gt;&amp;quot;&lt;&#x2F;span&gt;&lt;span style=&quot;color:#b48ead;&quot;&gt;]
&lt;&#x2F;span&gt;&lt;span&gt;}
&lt;&#x2F;span&gt;&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;h3 id=&quot;create-cronjob-for-nagios-user&quot;&gt;Create cronjob for &lt;code&gt;nagios&lt;&#x2F;code&gt; user&lt;a class=&quot;anchor&quot; aria-hidden=&quot;true&quot; href=&quot;#create-cronjob-for-nagios-user&quot; hidden=&quot;&quot;&gt;#&lt;&#x2F;a&gt;
&lt;&#x2F;h3&gt;
&lt;pre data-lang=&quot;sh&quot; style=&quot;background-color:#2b303b;color:#c0c5ce;&quot; class=&quot;language-sh &quot;&gt;&lt;code class=&quot;language-sh&quot; data-lang=&quot;sh&quot;&gt;&lt;span style=&quot;color:#bf616a;&quot;&gt;sudo -u&lt;&#x2F;span&gt;&lt;span&gt; nagios crontab&lt;&#x2F;span&gt;&lt;span style=&quot;color:#bf616a;&quot;&gt; -e
&lt;&#x2F;span&gt;&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;p&gt;Edit and insert nightly cronjob. Something like that should suffice.&lt;&#x2F;p&gt;
&lt;pre data-lang=&quot;sh&quot; style=&quot;background-color:#2b303b;color:#c0c5ce;&quot; class=&quot;language-sh &quot;&gt;&lt;code class=&quot;language-sh&quot; data-lang=&quot;sh&quot;&gt;&lt;span style=&quot;color:#65737e;&quot;&gt;# Daily at 2am: Generate Icinga checks from Traefik API and restart Icinga2 service
&lt;&#x2F;span&gt;&lt;span style=&quot;color:#bf616a;&quot;&gt;0&lt;&#x2F;span&gt;&lt;span&gt; 2 * * * &#x2F;etc&#x2F;icinga2&#x2F;cronjobs&#x2F;traefik2_to_icinga.py &amp;gt; &#x2F;etc&#x2F;icinga2&#x2F;zones.d&#x2F;YOUR-ZONE&#x2F;traefik_services.conf &amp;amp;&amp;amp; &lt;&#x2F;span&gt;&lt;span style=&quot;color:#bf616a;&quot;&gt;systemctl&lt;&#x2F;span&gt;&lt;span&gt; restart icinga2 &amp;gt;&#x2F;dev&#x2F;null &lt;&#x2F;span&gt;&lt;span style=&quot;color:#d08770;&quot;&gt;2&lt;&#x2F;span&gt;&lt;span&gt;&amp;gt;&amp;amp;&lt;&#x2F;span&gt;&lt;span style=&quot;color:#d08770;&quot;&gt;1
&lt;&#x2F;span&gt;&lt;&#x2F;code&gt;&lt;&#x2F;pre&gt;
&lt;p&gt;&lt;em&gt;Note:&lt;&#x2F;em&gt; Of course you need to print the output of &lt;code&gt;traefik2_to_icinga.py&lt;&#x2F;code&gt; to a directory that is either inside your zone or in conf.d&lt;&#x2F;p&gt;
&lt;p&gt;The end result should look like this:&lt;&#x2F;p&gt;
&lt;p&gt;&lt;img src=&quot;&#x2F;images&#x2F;screenshot-monitoring-traefik-icinga.png&quot; alt=&quot;Screenshot of Icinga2 with Traefik Service&quot; &#x2F;&gt;&lt;&#x2F;p&gt;
&lt;p&gt;I am looking forward to any feedback or even &lt;a href=&quot;https:&#x2F;&#x2F;github.com&#x2F;movd&#x2F;traefik_icinga_check&quot;&gt;pull-requests&lt;&#x2F;a&gt; for the small script.&lt;&#x2F;p&gt;
&lt;p&gt;&lt;small&gt;&lt;em&gt;Source for the graphics used in the header: &lt;a href=&quot;https:&#x2F;&#x2F;www.freepik.com&#x2F;free-photos-vectors&#x2F;background&quot;&gt;Background vector created by vectorpouch - www.freepik.com&lt;&#x2F;a&gt;&lt;&#x2F;em&gt;&lt;&#x2F;small&gt;&lt;&#x2F;p&gt;
</description>
      </item>
    </channel>
</rss>
