Icinga 2 – Monitoring automatisiert mit Puppet Teil 9: Profile

This entry is part 9 of 9 in the series Icinga 2 Monitoring automatisiert mit Puppet

Es ist nun nahzu schon ein Jahr her, dass in dieser Blogserie ein Artikel erschien. Zeit diese Serie fortzusetzen, auch weil wir für diese Jahr noch den Release 2.0.0 des Moduls puppet-icinga2 planen. In den kommenden Artikeln möchte ich sukezessive eine Profil-Klasse entwickeln, die einen Icinga-2-Server inklusive Plugins, IDO, Api und Icinga Web 2 installiert und konfiguriert. Als weitere Anforderung soll dies erfolgreich auf RedHat- wie auch Debian-Systemen durchgeführt werden können. Getestet wurde der Code auf Debian-9 (stretch) und CentOS-7.

Als erstes Teilziel für diesen Artikel soll Icinga 2 nebst Plugins enthalten sein. Bei den Plugins soll auch berücksichtig sein, dass es Plugins wie check_icmp oder check_dhcp gibt, die erweiterte Berechtigungen benötigen. So dürfen ICMP-Pakete oder auch DHCP-Request auf Unix nur im Ring-0 erzeugt werden. Auf RedHat wird dies über das Setzen des setuid-Bits erreicht, unter Debian mittels Posix Capabilities. Damit stellt uns Debian nicht vor größere Herausforderungen, die Plugins für RedHat-Systeme erfordern jedoch, dass der Aufrufende Benutzer Mitglied der Gruppe nagios sein muss. Um dies Anforderung zu realisieren, muss der Benutzer icinga der Gruppe nagios hinzugefügt werden, dass bei Puppet heißt, er ist via User-Resource zu verwalten. Am besten überlässt man das Anlegen vom Benutzer icinga weiterhin dem Paket icinga2, damit werden solche Eigenschaften wie UID oder das Home-Verzeichnis dem Paketverantwortlichen überlassen, d.h. aber auch die Paketinstallation muss vor der User-Resource und die wiederum vor der Klasse icinga2 abgearbeitet werden. Der Service, der von der Klasse verwaltet wird, darf erst gestartet werden, wenn der Benutzer schon korrekt konfiguriert ist. Andernfalls würde Icinga als Prozess weiterhin unter einem Benutzer laufen, der zum Startzeitpunkt von seiner Zugehörigkeit zur Gruppe nagios noch nichts wusste.
Zusätzlich muss auch die Gruppe nagios vor der User-Resource vorhanden sein, was sichergestellt ist, wenn vorher das Paket nagios-plugins-all installiert ist.

class profile::icinga2::server {

  case $::osfamily {
    'redhat': {
      require ::profile::repo::epel
      require ::profile::repo::icinga

      $manage_package = false
      $manage_repo    = false

      package { [ 'nagios-plugins-all', 'icinga2' ]:
        ensure => installed,
        before => User['icinga'],

      user { 'icinga':
        groups => [ 'nagios' ],
        before => Class['icinga2']
    } # RedHat
    'debian': {
      $manage_package = true
      $manage_repo    = true

      package { 'monitoring-plugins':
        ensure => installed,
    } # Debian
    default: {
      fail("'Your operatingsystem ${::operatingsystem} is not supported.'")
  } # case

  class { '::icinga2':
    manage_package => $manage_package,
    manage_repo    => $manage_repo,

Die Plugins befinden sich bei RedHat in einem zusätzlichen Repository, dem EPEL-Repository, das in der dedizierten Profilklasse profile::repo::epel verwaltet wird. Gleiches gilt für das Icinga-Repo mit der Klasse profile::repo::icinga, was auf RedHat für die gesonderte Paketinstalltion vorhanden sein muss und damit nicht, wie unter Debian, der Klasse icinga2 überlassen werden kann.

class profile::repo::epel {
  yumrepo { 'epel':
    descr => "Extra Packages for Enterprise Linux ${::operatingsystemmajrelease} - \$basearch",
    mirrorlist => "https://mirrors.fedoraproject.org/metalink?repo=epel-${::operatingsystemmajrelease}&arch=\$basearch",
    failovermethod => 'priority',
    enabled => '1',
    gpgcheck => '1',
    gpgkey => "http://dl.fedoraproject.org/pub/epel/RPM-GPG-KEY-EPEL-${::operatingsystemmajrelease}",

class profile::repo::icinga {
  yumrepo { 'ICINGA-release':
    descr => 'ICINGA (stable release for epel)',
    baseurl => 'http://packages.icinga.org/epel/$releasever/release/',
    failovermethod => 'priority',
    enabled => '1',
    gpgcheck => '1',
    gpgkey => 'http://packages.icinga.org/icinga.key',
Lennart Betz

Autor: Lennart Betz

Der diplomierte Mathematiker arbeitet bei NETWAYS im Bereich Consulting und bereichert seine Kunden mit seinem Wissen zu Icinga, Nagios und anderen Open Source Administrationstools. Im Büro erleuchtet Lennart seine Kollegen mit fundierten geschichtlichen Vorträgen die seinesgleichen suchen.

OSMC 2018 – Day 2

The evening event was again great food, drinks and conversation and while it ended in the early morning for some people, rooms were full of attendees again for the first talk. It was a hard choice between probably great talks but in the end I had chosen Rodrigue Chakode with “Make IT monitoring ready for cloud-native systems“. Being a long-term contributor to several Open Source Monitoring he used his experience to develop Realopinsight as a tool bringing existing monitoring tools together and extending them for monitoring cloud-native application platforms. In his live demo he showed the webinterface and Icinga 2, Zabbix and Kubernetes integration including aggregation of the severity for a specific service across the different solutions.

OSMC 2018

Scoring a Technical Cyber Defense Exercise with Nagios and Selenium” by Mauno Pihelgas was a quite uncommon case study. Locked Shields is the biggest Cyber Defense exercise involving 22 teams defending systems provided by vendors against hundreds of attacks. Mauno is responsible for the availability scoring system which gives the defending teams bonus points for availability of the systems, but of course it makes also available for attacks which if successful will cause loss of points. The data collected by Nagios and Selenium are then forwarded to Kafka and Elasticsearch to provide abuse control and overall scoring. To give you some numbers over the 2 days of the exercise about 34 million checks are executed and logged.

Susanne Greiner’s talk “Mit KI zu mehr Automatisierung bei der Fehleranalyse” was on using Artificial Intelligence for automatic failure analyses. Her talk started from anomaly detection and forecasting, went through user experience and ended with machine learning and deep learning. It is always great to see what experts can do with data, so running anomaly detection and forecasting on the data, adding labels for user experience and feeding them to the AI can increase troubleshooting capabilities. And better troubleshooting will result in better availability and user experience of course what perhaps is the main goal of all IT.

At the evening event there was again some gambling and after lunch the guys how managed to win the most chips won some real prices.

OSMC 2018 Gambling Winners

While some still enjoyed the event massage, Carsten Köbke started the afternoon sessions with the best talk title “Katzeninhalt mit ein wenig Einhornmagie” (Cat content with a little bit of unicorn magic). Being the author of the Icinga Web 2 module for Grafana and several themes for Icinga Web 2 he demonstrated and explained his work to the audience. It is very nice to see performance data with annotations extracted from the Icinga database nicely presented in Grafana. The themes part of the talk was based on the idea of every one can do this and monitoring can be fun.

Thomas and Daniel teamed up to focus on log management and help people on choosing their tool wisely in their talk “Fokus Log-Management: Wähle dein Werkzeug weise“. They compared the Elastic stack and Graylog with each other in multiple categories, showing up advantages and disadvantages and which tool fits best for which user group.

Eliminating Alerts or ‘Operation Forest’” by Rihards Olups was a great talk on how he tried reducing alerts to get a better acceptance and handling of the remaining alerts, getting problems solved instead of ignored. The ‘Operation forest’ mentioned in the talk’s title is his synonym for there infrastructure and alerts are trash he does not like in his forest, because trash attracts trash, like alerts attract alerts because if the numbers grows they tend to be ignored and more problems will get critical causing more alerts. It is not a problem of the tool used for monitoring and alerting but he had not only nice hints on changing culture but also technical ones like focusing on one monitoring solution, knowing and using all features or making problems more recognizable like putting them into the message of the day. For those having the same problems in their environment he wrote a shitlist you can check the problems you have and the number of checked items will indicate how shitty your environment is, I recommend having a look at this list.

Last but not least Nicolai Buchwitz talked about the “Visualization of your distributed infrastructure” and with his Map module for Icinga Web 2 he is providing a very powerful tool to visualize it. All the new features you get from the latest 1.1.0 release make it even more useful and the outlook on future extensions looks promising. Nicolai concluded with a nice live demo showing all this functionality.

So it was again a great conference, thanks to all speakers, attendees and sponsors for making this possible. I wish everyone not staying for the hackathon or Open Source camp “Save travels”. Slides, videos and pictures will be online in the near future. I hope to see you on next year’s OSMC on November, 4th – 7th!

Dirk Götz

Autor: Dirk Götz

Dirk ist Red Hat Spezialist und arbeitet bei NETWAYS im Bereich Consulting für Icinga, Puppet, Ansible, Foreman und andere Systems-Management-Lösungen. Früher war er bei einem Träger der gesetzlichen Rentenversicherung als Senior Administrator beschäftigt und auch für die Ausbildung der Azubis verantwortlich wie nun bei NETWAYS.

OSMC 2018 – or: The Thirteen-Star Conference

“How many stars do you have?” – Just take a look at the name tag, count the stars and you’re talking. At a lot of participants badges you’ll see a considerable number of them. Once you’ve been to OSMC, you usually come back. Why is that so? Here’s what I found out:


My first conversation partner is a big catch: 13 stars – 13 years of OSMC! Stefan Kublik arrived from Neckarsulm and is taking part for Fujitsu TDS GmbH. To the first conferences, at that time still Nagios, he went for another employer. He changed companies, OSMC remained. What Kublik particularly appreciates: “Many people here are working on the same problems. The OSMC is great to get an overview.” Where is the community standing? Where is the journey going? Thruk and Prometheus interest him, the products he uses himself. But Kublik also picks up on new topics and continues researching at home. What has changed over the years? OSMC has become more international, he says. Will he come again? “I have to! The series of stars can’t stop! As long as I work in this field, I will come back!”


Two stars: For the second time Kevin Honka is at the OSMC. For his first he has actually been here as a speaker. “I was relatively active in the monitoring portal,” he says. And so he was asked if he would like to report about his activities at the OSMC. “Now I come to every Icinga camp and of course to the OSMC.” He is interested in Icinga, Graylog, Git, virtualization techniques… What he appreciates the most? “The people. And the food,” he says with a grin. And he adds: “I have seldom had better technical discussions.



Ulrike Klusik has only been working in monitoring for three quarters of a year – but OSMC already has her! Thanks to her boss at ConSol Consulting & Solutions Software GmbH. Klusik thinks the conference is above all a good opportunity to get to know Open Source tools. Besides the ones she knows already – OMD, which they develop themselves, and OpenShift and Prometheus. “I found the lecture on Sensu very interesting,” she says, “or: Oberservability in einer Microservicewelt”. As a newcomer to monitoring, she is particularly interested in lectures on general procedures. She thinks it’s worth coming back: “It’s good to look beyond one’s own nose. Unfortunately, one often has too little time for this in everyday work.”


Christian Hager counts and is astonished himself: “Ten stars! “In the beginning I came because I wanted to know something about monitoring with Nagios and Icinga. Now I come for everything else,” he laughs. Smart Home, Refocus, Maps, SLA Monitoring he has joined. When I ask him what else he is interested in, he pulls out a small notebook. This man is prepared! Hager has made a plan for both days. All the more flexible he is during the breaks. “I like to sit down with people I don’t know. That gives me new ideas.” Other conferences may be bigger, but there are many more sales people on the road. “The OSMC gathers a good group of experts, and it’s easy to find specialists to talk to here.” Hager himself comes from the computer centre at the University of Würzburg.


I meet Holger Koch and Gudrun Schöllhammer at a bar table in front of room Jacobi. They talk to each other. He joins for DB Systel GmbH from Erfurt, she comes from the University of Vienna. Eight OSMC stars are emblazoned on his name tag, six on hers. “We see each other here once a year,” she says – and he: “The OSMC is like a class reunion.” Unfortunately, we don’t get any further: The next talk begins. The two have a well clocked schedule. Apart from the good conversations with other admins, they are mainly on the hunt for input. Well, then I don’t want to stop them!



Have fun at the OSMC 2018!


And save the date for next year’s OSMC, November 4 – 7, 2019!


Julia Hornung

Autor: Julia Hornung

Julia ist seit Juni 2018 Mitglied der NETWAYS-Crew. Vor ihrer Zeit in unserem Marketing Team hat sie als Journalistin und Produktionsassistentin in der freien Theaterszene gearbeitet. Ihre Leidenschaft gilt gutem Storytelling. Privat widmet sie sich dem Klettern und ihrer Ausbildung zur Yogalehrerin.

OSMC 2018 – Day 1

It is always the same, Winter is coming and it brings people to Nuremberg for OSMC. Our Open Source Monitoring conference still grows every year and after giving three parallel tracks a try last year, we changed format again to include also shorter talks and having always three tracks. It also gets more international and topics get more diverse, covering all different monitoring solutions with speakers (and attendees) from all over the worlds. Like every year also the 13th conference started with a day of workshops enabling the interested ones to get hands on Prometheus, Ansible, Graylog and practical example on using the Puppet modules for Icinga 2. Also this year two days of great talks will be followed by a day of hacking and the second issue of the Open Source Camp takes place, this time focusing on Puppet.

OSMC 2018

And another tradition is Bernd starting the conference with a warm welcome before the first talk. Afterwards Michael Medin talked about his journey in monitoring and being a speaker at OSMC for the eleventh time in “10 years of OSMC: Why does my monitoring still look the same?“. It was a very entertaining talk comparing general innovation with the one happening in monitoring. He was showing up that monitoring solutions changed to reflect the change in culture but still stayed the same mechanism and explained all the problems we probably know like finding the correct metrics and interpreting them resulting from this.

Second talk I attended was “Scaling Icinga2 with many heterogeneous projects – and still preserving configurability” by Max Rosin. He started with the technical debt to solve and requirements to fulfill when migrating from Icinga 1 to Icinga 2 like check latency or 100% automation of the configuration. Their high-available production environment had no outage since going live in January, because the infrastructure design and testing updates and configuration changes in a staging setup, what is pretty awesome. The scripting framework they created for the migration will be released on Github. But this was not all they coded to customize their environment, they added some very helpful extensions for the operations team to Icinga Web 2, which will be available on Github somewhere in the future after separating company specific and upstream ready parts.

For the third session I had chosen Matthias Gallinger with “Netzwerkmonitoring mit Prometheus” (Network monitoring with Prometheus). In his case study he showed the migration from Cacti to Prometheus and Grafana done at a international company based in Switzerland. The most important part is here the SNMP Exporter for Prometheus including a generator for its configuration. All required is part of their labs edition of Open Monitoring Distribution (OMD).

After the lunch Serhat Can started with “Building a healthy on-call culture“. He provided and explained his list of rules which should create such a culture: Be transparent – Share responsibilities – Be prepared – Build resilient and sustainable systems – Create actionable alerts – Learn from your experiences. To sum up he tells everyone to care about the on-call people resulting in a good on-call service and user experience which will prevent a loss of users and money.

The Director of UX at Grafana Labs David Kaltschmidt gave an update on whats new and upcoming in Grafana focusing on the logging feature in “Logging is coming to Grafana“. The new menu entry Explore allows to easily querying Prometheus metrics including functions – just one click away – for rate calculation or average and it works the same for logging entries as a new type of datasource. This feature should be very useful in a Kubernetes environment to do some distributed tracing. If you are interested in this feature it should be available as beta in December.

Distributed Tracing FAQ” was also the title of Gianluca Arbezzano‘s talk. I can really recommend his talk for the good explanation on why and how to trace requests through more and more complex, distributed services of nowadays. If you are more interested in tool links, he recommends Opentracing as library, Zipkin as frontend and of course InfluxDB as backend.

This year Bernd’s talk about the “Current State of Icinga” was crowded and interesting as always. I skip the organizational things like interest in the project is growing according to website views, customers talking about their usage, partners, camps and meetups all over the world. From the technical aspects Icinga 2 had a release bringing more stabilization, improved Syntax Highlighting and as new feature Namespacing. The coming Director release brings support for multiple instances helping with staging, health checks and a configuration basket allowing to easily export and import configuration. A new Icinga Web 2 module X509 helps managing your certificate infrastructure, available next week on github. The one for VMware vSphere (sponsored by dmTECH) is already released and was shown in a demo by Tom who developed it. Icinga DB will replace IDO as a backend moving volatile data to Redis and data to be keeped will be stored to MySQL or PostgreSQL and there will also be a new Monitoring Module for Icinga Web 2 to make use of it, all available hopefully in two weeks.

This year’s OSMC provided something special as the last talk of the first day with an authors’ panel including Marianne Spiller (Smart Home mit openHAB 2), Jan Piet Mens (Alternative DNS Servers – Choice and deployment, and optional SQL/LDAP back-ends), Thomas Widhalm and Lennart Betz (Icinga 2 – Ein praktischer Einstieg ins Monitoring) moderated by Bernd and answering questions from the audience.

If you want to get more details or pictures have a look at Twitter. There will also be a post by Julia giving a more personal view on the conference from interviewing some attendees and one of me covering the talks of the second day, but now I am heading for the evening event.

Dirk Götz

Autor: Dirk Götz

Dirk ist Red Hat Spezialist und arbeitet bei NETWAYS im Bereich Consulting für Icinga, Puppet, Ansible, Foreman und andere Systems-Management-Lösungen. Früher war er bei einem Träger der gesetzlichen Rentenversicherung als Senior Administrator beschäftigt und auch für die Ausbildung der Azubis verantwortlich wie nun bei NETWAYS.


Only 16 days to go?

Time for you to speed up: Grab one of our popular Hackathon tickets and join the event on Nov 08!

It’s fun! You’ll start the day with a short round of introductions, then run a brainstorming session for possible topics in addition to these ones that we came up with so far:


Team up, accept the challenge, collaborate to find a solution, succeed!
Find all details at: osmc.de/hackathon


Taking place on Nov 08 the Hackathon directly follows OSMC’s lecture program. Be part of OSMC and round off your stay! Get your conference and add-on ticket here! We would be happy to see you soon at…

#OSMC | November 5 – 8, 2018 | Nuremberg

Julia Hornung

Autor: Julia Hornung

Julia ist seit Juni 2018 Mitglied der NETWAYS-Crew. Vor ihrer Zeit in unserem Marketing Team hat sie als Journalistin und Produktionsassistentin in der freien Theaterszene gearbeitet. Ihre Leidenschaft gilt gutem Storytelling. Privat widmet sie sich dem Klettern und ihrer Ausbildung zur Yogalehrerin.

OSMC #Recap with Kempf & Kraus | Don’t miss this year!

This entry is part 7 of 7 in the series OSMC Recap 2017

2017 sprachen Tobias Kempf & Michael Kraus über “Hochautomatisierte Warenlogistik – Monitoring bei Europas größtem Handelsunternehmen”.

  • 11.460 Filialen in 25 Ländern mit 390.000 Mitarbeiter
  • Es gilt, die Kommunikation über alle Ebenen zu sichern
  • Es braucht neue Monitoring Definitionen

Mehr über die Zusammenarbeit der beiden Herren gibt’s hier:


Don’t miss OSMC 2018! Get your ticket now! Be a part of the Monitoring change.

OSMC | November 5 – 8, 2018 | Nuremberg

Julia Hornung

Autor: Julia Hornung

Julia ist seit Juni 2018 Mitglied der NETWAYS-Crew. Vor ihrer Zeit in unserem Marketing Team hat sie als Journalistin und Produktionsassistentin in der freien Theaterszene gearbeitet. Ihre Leidenschaft gilt gutem Storytelling. Privat widmet sie sich dem Klettern und ihrer Ausbildung zur Yogalehrerin.