logo2_elastic_150x75I’m always trying to look into new devops tools and how they fit best with Icinga 2 as a monitoring solution. Often demanded is an integration with Elastic Stack and Elastic Beats with Icinga 2. Gathering metrics and events, correlated to additional input sources analysing a greater outage and much more.

Last week the first 5.0.0 beta1 release hit my channels and I thought I’d give it a try. The installation is pretty straight forward using packages. Note: This is my first time installing Elastic Stack, still have little knowledge from colleague hero stories and the OSDC talk by Monica Sarbu and earlier conferences.

Requirements

Install Java 1.8.0 as a requirement. I’m using a Centos 7 VM here with the EPEL repository enabled.

# yum install -y java

 

Setup Elasticsearch

Fetch the latest RPM package and install it. It nicely integrates with Systemd, no more hassle with starting it in foreground as suggested in the docs.

# yum install -y https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.0.0-beta1.rpm
# systemctl daemon-reload
# systemctl enable elasticsearch.service

Reduce the memory heap size usage for a small demo VM.

vim /etc/elasticsearch/jvm.options

# Xms represents the initial size of total heap space
# Xmx represents the maximum size of total heap space

-Xms512m
-Xmx512m

Start Elasticsearch.

# systemctl start elasticsearch.service

Test whether the service is talking REST on port 9200.

# curl -X GET http://localhost:9200/
{
  "name" : "-ikeg-B",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "5sZaEqBnST2tW7G3Vo1slg",
  "version" : {
    "number" : "5.0.0-beta1",
    "build_hash" : "7eb6260",
    "build_date" : "2016-09-20T23:10:37.942Z",
    "build_snapshot" : false,
    "lucene_version" : "6.2.0"
  },
  "tagline" : "You Know, for Search"
}

 

Setup Logstash

Straight forward package installation. Specific configuration will be added later.

# yum install -y https://artifacts.elastic.co/downloads/logstash/logstash-5.0.0-beta1.rpm
# systemctl enable logstash.service
# systemctl start logstash.service

 

Setup Kibana

Install the latest RPM package. Kibana listens on localhost by default, change that to “0.0.0.0”.

# yum -y install https://artifacts.elastic.co/downloads/kibana/kibana-5.0.0-beta1-x86_64.rpm

# vim /etc/kibana/kibana.yml

server.host: "0.0.0.0"

# systemctl start kibana

 

Configure Logstash to accept Beats input

Create a new configuration file underneath /etc/logstash/conf.d. Define “beats” as input listening on port 5044. The output is set to “elasticsearch” according to the documentation. Once finished, restart the logstash service.

# vim /etc/logstash/conf.d/logstash.conf

input {
  beats {
    port => "5044"
    client_inactivity_timeout => "1000"
  }
}

output {
  elasticsearch {
    hosts => "localhost:9200"
    manage_template => false
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
  }
}

# systemctl restart logstash

 

Setup Filebeat

Beats are small collector daemons which gather metrics and logs from your system. In this example I’m installing Filebeat to just collect logs. Nice one – again Systemd services for easier handling.

# yum install -y https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.0.0-beta1-x86_64.rpm
# systemctl enable filebeat
# systemctl start filebeat

 

Configure Filebeat

My demo VM has Icinga 2 with some sample configuration pre-installed. I’ll use these logs to generate input for the Filebeat collector. By default the Filebeat configuration will attempt to write to Elasticsearch directly. I’ll change that to Logstash for additional parsing/tagging. More details in the documentation.

# vim /etc/filebeat/filebeat.yml

filebeat.prospectors:
- input_type: log
  paths:
    - /var/log/*.log
    - /var/log/messages
    - /var/log/icinga2/*.log
  fields:
    type: syslog
output.logstash:
  hosts: ["localhost:5044"]


# systemctl restart filebeat
# tail -f /var/log/filebeat/filebeat

 

Import Beats dashboards into Kibana

This comes in handy for configuring the required index patterns in Kibana. Proceed as described in the documentation.

# /usr/share/filebeat/scripts/import_dashboards

 

Configure Kibana

Navigate to http://yourhost:5601 and configure a default index pattern for “filebeat-*”.

elastic_5_beta1_kibana_configure_index_pattern_filebeat_01elastic_5_beta1_kibana_configure_index_pattern_filebeat_02

 

Test Filebeat

While testing Filebeat I found a bug in the beta1 release which causes Logstash to terminate idle connections from Filebeat with EOF. The workaround suggests to configure the Logstash beats input with client_inactivity_timeout>900 – works again.

The screenshot highlights the latest log messages coming from Icinga 2 (icinga2.log).

elastic_5_beta1_kibana_filebeat_icinga2_log

 

More Beats

Topbeat is going to be replaced by Metricbeat. The latter is still experimental but allows you to play with advanced metric collection.

# yum install -y https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-5.0.0-beta1-x86_64.rpm

# vim /etc/metricbeat/metricbeat.yml
...
output.logstash:
  hosts: ["localhost:5044"]

# systemctl enable metricbeat
# systemctl start metricbeat

# /usr/share/metricbeat/scripts/import_dashboards

elastic_5_beta1_kibana_metricbeat

 

 

I’m a Golang beginner, but it shouldn’t be too hard to learn it when you know C++. The Beats already out there look fairly promising. I’ve found for example nagioscheckbeat which executes the Icinga monitoring plugins.

The nice thing is that there is libbeat as framework already, and lots of documentation on how to create your own beats skeleton. I’ve tried to create the “countbeat” example – works like a charm. Let’s see where my journey goes in the future 😉

mkdir $HOME/work
export GOPATH=$HOME/work
go get github.com/elastic/beats
mkdir $GOPATH/src/github.com/dnsmichi
cd $GOPATH/src/github.com/dnsmichi

yum install -y python-pip python-virtualenv
pip install cookiecutter

# cookiecutter $GOPATH/src/github.com/elastic/beats/generate/beat
project_name [Examplebeat]: Countbeat
github_name [your-github-name]: dnsmichi
beat [countbeat]:
beat_path [github.com/dnsmichi]:
full_name [Firstname Lastname]: Michael Friedrich

# cd $GOPATH/src/github.com/dnsmichi/countbeat
# make setup
# make

# vim countbeat.yml

output.logstash:
  hosts: ["localhost:5044"]

# curl -XPUT 'http://localhost:9200/_template/countbeat' -d@countbeat.template.json

# ./countbeat -e -d "*"

2016/09/28 20:34:41.332237 output.go:109: DBG  output worker: publish 1 events
2016/09/28 20:34:41.332272 sync.go:107: DBG  Try to publish 1 events to logstash with window size 10
2016/09/28 20:34:41.332691 client.go:183: DBG  Publish: {
  "@timestamp": "2016-09-28T20:34:41.332Z",
  "beat": {
    "hostname": "icinga2-elastic",
    "name": "icinga2-elastic"
  },
  "counter": 4,
  "type": "countbeat"
}
2016/09/28 20:34:41.332721 countbeat.go:54: INFO Event sent
2016/09/28 20:34:41.334081 sync.go:78: DBG  1 events out of 1 events sent to logstash. Continue sending
2016/09/28 20:34:41.334103 single.go:150: DBG  send completed

 

elastic_5_beta1_kibana_example_countbeat_01elastic_5_beta1_kibana_example_countbeat_02

 

Conclusion

I’m eagerly awaiting the stable 5.0 release, working with the latest and greatest stuff. Meanwhile I’m going to test the beta release in deep, especially in terms of a LogstashWriter feature for Icinga 2 or for preparing a shiny Vagrant box.

We’re thrilled about Monica Sarbu’s OSMC talk on Elastic Beats and of course the OSMC hackathon to work on Beats integration with Icinga 2 and much more. Make sure to join us!

In case you are looking for an in-depth training for configuring Elastic Stack for your environment, my colleagues provide awesome training courses including 5.x already 🙂

Michael Friedrich

Autor: Michael Friedrich

Michael ist seit vielen Jahren Icinga Developer und hat sich Ende 2012 in das Abenteuer NETWAYS gewagt. Ein Umzug von Wien nach Nürnberg mit der Vorliebe, österreichische Köstlichkeiten zu importieren - so mancher Kollege verzweifelt an den süchtig machenden Dragee-Keksi. Oder schlicht am österreichischen Dialekt der gerne mit Thomas im Büro intensiviert wird ("Jo eh."). Wenn sich Michael mal nicht im Monitoring-Portal helfend meldet, arbeitet er am nächsten LEGO-Projekt oder geniesst das schöne Nürnberg. Oder - at an Icinga Camp near you 😉