Icinga 2 – Monitoring automatisiert mit Puppet Teil 6: Agenten

This entry is part 6 of 7 in the series Icinga 2 Monitoring automatisiert mit Puppet

Nachdem Lennart sich bereits mit vielen Aspekte des Moduls befasst hat, will ich dieses Mal auf die Installation von Icinga 2 als Agenten eingehen. Neben einem generellen Einstieg mit zu beachtenden Konfigurationsoptionen, will ich hierbei auf die verschiedenen Möglichkeiten für die benötigten Zertifikate und Anbindung an die übergeordnete Zone eingehen.


Die Installation und Feature-Konfiguration erfolgt wie Lennart dies in den ersten beiden Teilen der Serie beschrieben hat. Hierbei möchte ich beim Agent sicherstellen, dass keine lokale Konfiguration angezogen wird, da ich für die Verteilung der CheckCommands die Konfigurationssynchronisation von Icinga 2 nutzen möchte. Alternativ können diese zwar auch über Puppet verteilt werden, da ja auch die Plugins installiert werden müssen, aber in den meisten Umgebungen trenne ich an dieser Stelle Konfiguration der Monitoring-Infrastruktur von der eigentlichen Monitoring-Konfiguration. Der Start meines Agenten-Profils sieht also meist so aus:

  class { 'icinga2':
    confd       => false,
    manage_repo => true,
    features    => ['checker','mainlog'],


Dirk Götz

Autor: Dirk Götz

Dirk ist Red Hat Spezialist und arbeitet bei NETWAYS im Bereich Consulting für Icinga, Nagios, Puppet und andere Systems Management Lösungen. Früher war er bei einem Träger der gesetzlichen Rentenversicherung als Senior Administrator beschäftigt und auch für die Ausbildung der Azubis verantwortlich.

Foreman’s 8th birthday – We will give a party

Foreman Logo

After the success last year we decided to celebrate the 8th Birthday of the Foreman Project again at event room Kesselhaus on July, 27th. As feedback was very good about the provided content we decided to provide the same schedule.

We will start right after lunch at 12:30pm with a hands-on session, so you can do your first steps with Foreman, have a look into latest development or get some inspiration which plugins to add to your environment. There will be some experienced users and developers around to help you, give you some tips for your environment or do some in-person hacking. The demos will showcase several ways of provisioning, configuration management with Puppet and Ansible, orchestration, monitoring integration and other plug-ins.

After a short coffee break we will have 3-5 talks for 30-40 minutes depending on how many speakers I can find. Already confirmed speakers are Ewoud Kohl van Wijngaarden, Michael Moll and Timo Goebel, all being long-term users and contributors to the Foreman Project. I am still looking for someone willing to talk about how he is using Foreman, so if you want to give a case study talk simply drop me a mail.

It would not be a party if we would not have at least some Pizza and Beer afterwards. And the Foreman’s Community guy Greg Sutcliffe already promised some swag to take home with you.

So if you want to join us, please register for free here. This will enable us to order enough food and drinks. We will keep you updated on twitter and on the event page. As a teaser you can watch the recording of me giving a demo at OSDC.

If you can not make it to our event, keep your eyes open for an event near you. For example Julien Pivotto already announced one in Belgium.

Dirk Götz

Autor: Dirk Götz

Dirk ist Red Hat Spezialist und arbeitet bei NETWAYS im Bereich Consulting für Icinga, Nagios, Puppet und andere Systems Management Lösungen. Früher war er bei einem Träger der gesetzlichen Rentenversicherung als Senior Administrator beschäftigt und auch für die Ausbildung der Azubis verantwortlich.

Like meeting the family – OSDC 2017: Day 1

I was happy to join our conference crew for OSDC 2017 again because it is like meeting the family as one of our attendees said. Conference started for me already yesterday because I could join Gabriel‘s workshop on Mesos Marathon. It was a quite interesting introduction into this topic with examples and know how from building our Software-As-A-Service platform “Netways Web Services“. But it was also very nice to meet many customers and long-time attendees again as I already knew more than half of the people joining the workshops. So day zero ended with some nice conversation at the hotel’s restaurant.

As always the conference started with a warm welcome from Bernd before the actual talks (and the hard decision which talk to join) started. For the first session I joined Daniel Korn from Red Hat’s Container Management Team on “Automating your data-center with Ansible and ManageIQ“. He gave us an good look behind “one management solution to rule them all” like ManageIQ (the upstream version of Red Hat Cloudform) which is designed as an Open source management platform for Hybrid IT. So it integrates many different solutions like Openshift, Foreman or Ansible Tower in one interface. And as no one wants to configure such things manually today there are some Ansible modules to help with automating the setup. Another topic covered was Hawkular a time series database including triggers and alarming which could be used get alerts from Openshift to ManageIQ.

The second talk was Seth Vargo with “Taming the Modern Data Center” on how to handle the complexity of data centers today. He also covered the issues of life cycles shrinking from timeframes measured in days, weeks and month to seconds and minutes and budget moving from CapEx to OpEx by using cloud or service platforms. With Terraform he introduced one of HashiCorp’s solutions to help with solving these challenges by providing one abstraction layer to manage multiple solutions. Packer was another tool introduced to help with image creation for immutable infrastructure. The third tool shown was Consul providing Service Discovery (utilizing DNS or a HTTP API), Health Checking (and automatic removal from discovered services), Key/Value Store (as configuration backend for these services) and Multi-Datacenter (for delegating service request to nearest available system). In addition Seth gave some good look inside workflows and concepts inside HashCorp like they use their own software and test betas in production before releasing or trust developers of the integrated software to maintain the providers required for this integration.

Next was Mandi Walls on “Building Security Into Your Workflow with InSpec”. The problem she mentioned and is tried to be resolved by InSpec is security reviews can slow down development but moving security reviews to scanning a production environment is to late. So InSpec is giving the administrator a spec dialect to write human-readable compliance tests for Linux and Windows. It addresses being understandable for non-technical compliance officers by doing so and profiles give them a catalog to satisfy all their needs at once. If you want an example have a look at the chef cookbook os-hardening and the InSpec profile /dev-sec/linux-baseline working nicely together by checking compliance and running remediation.

James Shubin giving a big life demo of mgmt was entertaining and informative as always. I have already seen some of the demos on other events, but it is still exciting to see configuration management with parallelization (no unnecessary waiting for resources), event driven (instant recreation of resources), distributed topology (no single point of failure), automatic grouping of resource (no more running the package manager for every package), virtual machines as resources (including managing them from cockpit and hot plug cpus), remote execution (allowing to spread configuration management through SSH from one laptop over your data center). mgmt is not production ready for now, but its very promising. Future work includes a descriptive language, more resource types and more improvements. I can recommend watching the recording when it goes online in the next days.

“Do you trust your containers?” was the question asked by Erez Freiberger in his talk before he gave the audience some tools to increase the trust. After a short introduction into SCAP and OpenSCAP Erez spoke about Image inspector which is build on top of them and is utilized by OpenShift and ManageIQ to inspect container images. It is very good to see security getting nicely integrated into such tools and with the mentioned future work it will be even nicer to use.

For the last talk of today I joined Colin Charles from Percona who let us take part on “Lessons learned from database failures”. On his agenda were backups, replication and security. Without blaming and shaming Colin took many examples which failed and explained how it could be done better with current software and architecture. This remembers me to catch up on MySQL and MariaDB features before they hit enterprise distributions.

So this is it for today, after so many interesting talks I will have some food, drinks and conversation at the evening event taking place at Umspannwerk Ost. Tomorrow I will hand over the blog to Michael because I will give a talk about Foreman myself.

Dirk Götz

Autor: Dirk Götz

Dirk ist Red Hat Spezialist und arbeitet bei NETWAYS im Bereich Consulting für Icinga, Nagios, Puppet und andere Systems Management Lösungen. Früher war er bei einem Träger der gesetzlichen Rentenversicherung als Senior Administrator beschäftigt und auch für die Ausbildung der Azubis verantwortlich.

Contributing to projects on GitHub

GitHub Logo

Today I want to share my workflow for contributing to projects hosted on GitHub because I believe it works very well for me and I regularly contribute to various projects. Of course most of this also applies to other hosted Git platforms like GitLab. It will not involve any Git magic as there are other posts to do that. So let’s use adding a check command to Icinga 2 as my example.

The obvious first step is finding the Git repository for the project and reading contribution guidelines because there are some projects which aren’t hosted on GitHub and some have additional requirements like submitting an issue in addition to a pull request. You should always familiarize yourself with and stick to those policies if you want your pull request to be accepted. For Icinga 2 there currently aren’t any additional project-specific guidelines.

Your next step is to create a fork on GitHub and clone the repository. So in the GitHub web interface click on “Fork” and select your own account (or company account as long as you are allowed to push). Once you’ve forked the repository you can check out a local copy using the following command:

git clone git@github.com/dgoetz/icinga2.git

Afterwards you should add another “remote” for the original Git repository in order to be able to update your own repository with changes from the upstream project:

cd icinga2/
git remote add upstream git@github.com:icinga/icinga2.git

After these initial steps you can create a Git branch for your feature or bug fix. Using the “master” branch for pull requests is strongly discouraged because things tend to get complicated once you have more than one pull request. Another recommendation is to use branch names that match the upstream repository’s style. This however is not a hard requirement:

git checkout -b feature/expand-check-foo

This also automatically switches to the newly-created branch which means you can now start to edit files for your pull request. Once you’re done you can add them to the Git index and create a commit. Typically upstream projects have guidelines for this, so do not forget to include documentation, make only one commit out of your work (perhaps by squashing) and so on.

vi itl/plugins-contrib.d/network-components.conf
vi doc/10-icinga-template-library.md
git add itl/plugins-contrib.d/network-components.conf doc/10-icinga-template-library.md
git commit -m "ITL: expanded check foo"

Afterwards push your commit to your forked repository and then create your pull request using the GitHub webinterface:

git push -u origin feature/expand-check-foo

When creating the pull request make sure to provide a detailed description of your changes and the reason why you feel that your pull request should be merged. Keep the setting checked to allow edits from maintainers. Depending on the project make sure to reference any related issues, fill in their pull request template or do whatever else they require for pull requests.

GitHub pull request

Typically once a pull request is created automated tests will be run and a review process by the project team will start, so it’s possible that you’ll be asked to make changes before your pull request is accepted. If this happens simply edit your branch to fix whatever problems were found during the review, amend your commit and force push it to your fork. This will also automatically update your pull request but you might want to provide a comment for the pull request as to what has changed:

vi doc/10-icinga-template-library.md
git add doc/10-icinga-template-library.md
git commit --amend -m "ITL: expanded check foo"
git push -f

Another commonly requested change is that you rebase your branch before the pull request is accepted. This usually happens when significant changes were made to the upstream repository while your pull request was waiting to be merged. In order to rebase your branch the following commands should be all you need, however in some cases you may also have to manually resolve conflicts:

git pull --rebase upstream master
git push -f

And of course you will sometimes want to create additional pull requests. For these make sure to start with a new branch based on the upstream repository:

git checkout master
git pull upstream master
git checkout -b fix/check-bar
git push -u origin fix/check-bar

So, this is it, this is my basic workflow for easy contributions on GitHub. I hope it helps you to get involved with your favorite projects and your fixes and features to get upstream. If you prefer to do all step in the command line, you can have a look at GitHub’s command-line wrapper for git hub. If you need more general Git knowledge I recommend the Git book and our training. If you need a GitLab system to play around, have a look at our NWS platform.

Dirk Götz

Autor: Dirk Götz

Dirk ist Red Hat Spezialist und arbeitet bei NETWAYS im Bereich Consulting für Icinga, Nagios, Puppet und andere Systems Management Lösungen. Früher war er bei einem Träger der gesetzlichen Rentenversicherung als Senior Administrator beschäftigt und auch für die Ausbildung der Azubis verantwortlich.

Foreman räumt auf

Foreman Logo

Dank Virtualisierung und automatischer Provisionierung sind schnell und einfach Entwicklungs- und Testsysteme zur Verfügung gestellt und jeder bestellt fleißig neue Systeme, aber wie viele melden sich dann wirklich zurück wenn ein System nicht mehr gebraucht wird? Aus meiner Erfahrung heraus möchte ich behaupten die wenigsten tun dies. So kommt es, dass Ressourcen reserviert sind und brach liegen oder schlimmer noch tatsächlich belegt aber nicht mehr wirklich genutzt werden. Manuelle Aufräumaktionen treffen dann entweder zu wenig oder zu viele Systeme und kosten zu viel Zeit und Nerven. Wie schön wäre es wenn man die Kollegen erziehen könnte? Da es mit den Kollegen wohl nicht klappen wird, erziehen wir stattdessen unsere Umgebung und zwar mit dem Plugin “Expire Hosts” für Foreman.

Die Installation übernimmt wie mittlerweile bei fast allen Plugins der Foreman Installer und danach können wir auch direkt konfigurieren wie Systeme zukünftig von diesem Plugin behandelt werden sollen.

Foreman Expire Hosts Settings

Wie man sieht kann man drei Zeiten einstellen und zwar wann die erste und zweite Benachrichtigung relativ zum Ablaufdatum erfolgen soll und nach wie viel Tagen es nach Ablauf gelöscht werden soll nachdem das System am Ablaufdatum heruntergefahren wurde. Außerdem kann eingestellt werden, ob der Besitzer des Systems oder nur der Administrator das Ablaufdatum nachträglich ändern darf und ob die Angabe eine Pflicht ist. Wer dies als Administrator will kann sich auch auf die Liste der zusätzlichen Email-Empfänger setzen, da sonst nur der Besitzer benachrichtigt wird.

Wenn die allgemeinen Einstellungen nun passen erhält jeder Host ein Eingabefeld zur Einstellung seines Ablaufdatums unter den zusätzlichen Informationen.

Foreman Expire Hosts Host

Zusätzlich zur Email-Benachrichtigung zeigt Foreman auch noch im Webinterface den Status an und warnt auch hier vor dem Ablauf.

Foreman Expire Hosts OkForeman Expire Hosts Warning

Zwar ist das Plugin primär für virtuelle Maschinen gedacht, funktioniert aber auch bei physikalischen Systemen. Wenn Foreman das Powermanagement für diese steuern kann, fährt es diese auch herunter. Wenn nicht weiß der Administrator zumindest, dass das System zum Herunterfahren und Löschen freigegeben ist.

Ich denke mal für dieses Plugin finden so einige Administratoren einen sinnvollen Anwendungsfall, weshalb ich es auch neben vielen weiteren Plugins in die Foreman-Schulung aufgenommen habe.

Dirk Götz

Autor: Dirk Götz

Dirk ist Red Hat Spezialist und arbeitet bei NETWAYS im Bereich Consulting für Icinga, Nagios, Puppet und andere Systems Management Lösungen. Früher war er bei einem Träger der gesetzlichen Rentenversicherung als Senior Administrator beschäftigt und auch für die Ausbildung der Azubis verantwortlich.