This week we had the pleasure to welcome Jan Doberstein from Graylog. On Monday our consulting team and myself attended a Graylog workshop held by Jan. Since many of us are already familiar with log management (e.g. Elastic Stack), we’ve skipped the basics and got a deep-dive into the Graylog stack.
You’ll need Elasticsearch, MongoDB and Graylog Server running on your instance and then you are good to go. MongoDB is mainly used for caching and sessions but also as user storage e.g. for dashboards and more. Graylog Server provides a REST API and web interface.
Configuration and Inputs
Once you’ve everything up and running, open your browser and log into Graylog. The default entry page greets you with additional tips and tricks. Graylog is all about usability – you are advised to create inputs to send in data from remote. Everything can be configured via the web interface, or the REST API. Jan also told us that some more advanced settings are only available via the REST API.
If you need more input plugins, you can search the marketplace and install the required one. Or you’ll create your own. By default Graylog supports GELF, Beats, Syslog, Kafka, AMQP, HTTP.
One thing I also learned during our workshop: Graylog also supports Elastic Beats as input. This allows even more possibilities to integrate existing setups with Icingabeat, filebeat, winlogbeat and more.
Graylog supports “internal auth” (manual user creation), sessions/tokens and also LDAP/AD. You can configure and test that via the web interface. One thing to note: The LDAP library doesn’t support nested groups for now. You can create and assign specific roles with restrictions. Even multiple providers and their order can be specified.
Streams and Alerts
Incoming messages can be routed into so-called “streams”. You can inspect an existing message and create a rule set based on these details. That way you can for example route your Icinga 2 notification events into Graylog and correlate events in defined streams.
Alerts can be defined based on existing streams. The idea is to check for a specific message count and apply threshold rules. Alerts can also be reset after a defined grace period. If you dig deeper, you’ll also recognise the alert notifications which could be Email or HTTP. We’ve also discussed an alert handling which connects to the Icinga 2 API similar to the Logstash Icinga output. Keep your fingers crossed.
You can add stream message counters, histograms and more to your own dashboards. Refresh settings and fullscreen mode are available too. You can export and share these dashboards. If you are looking for automated deployments, those dashboards can be imported via the REST API too.
Graylog 2.3 is currently in alpha stages and will be released in Summer 2017. We’ve also learned that it will introduce Elasticsearch 5 as backend. This enables Graylog to use the HTTP API instead of “simulating” a cluster node at the moment. The upcoming release also adds support for lookup tables.
I’ve been fixing a bug inside the Icinga 2 GelfWriter feature lately and was looking for a quick test environment. Turns out that the Graylog project offers Docker compose scripts to bring up a fully running instance. I’ve slightly modified the docker-compose.yml to export the default GELF TCP input port 12201 on localhost.
command: "elasticsearch -Des.cluster.name='graylog'"
Navigate to http://localhost:9000/system/inputs (admin/admin) and add additional inputs, like “Gelf TCP”.
I just enabled the “gelf” feature in Icinga 2 and pointed it to port 12201. As you can see, there’s some data running into. All screenshots above have been taken from that demo too 😉
Jan continued the week with our official two day Graylog training. From a personal view I am really happy to welcome Graylog to our technology stack. I’ve been talking about Graylog with Bernd Ahlers at OSMC 2014 and now am even more excited about the newest additions in v2.x. Hopefully Jan joins us for OSMC 2017, Call for Papers is already open 🙂
My colleagues are already building Graylog clusters and more integrations. Get in touch if you need help with integrating Graylog into your infrastructure stack 🙂
Autor: Michael Friedrich
Michael ist seit vielen Jahren Icinga Developer und hat sich Ende 2012 in das Abenteuer NETWAYS gewagt. Ein Umzug von Wien nach Nürnberg mit der Vorliebe, österreichische Köstlichkeiten zu importieren - so mancher Kollege verzweifelt an den süchtig machenden Dragee-Keksi. Oder schlicht am österreichischen Dialekt der gerne mit Thomas im Büro intensiviert wird ("Jo eh."). Wenn sich Michael mal nicht im Monitoring-Portal helfend meldet, arbeitet er am nächsten LEGO-Projekt oder geniesst das schöne Nürnberg. Oder - at an Icinga Camp near you 😉