Saturday, 25 October 2014

A Central Log Server via Logstash ElasticSearch and Kibana

Objective

A central logs server setup example of a distributed application.

Environment

  • Distributed Java application Nodes:
    • Node A : Running on RHEL 6.x OS
    • Node B: Running on RHEL 6.x OS
    • Node C: Running on Solaris 5.9 OS
  • Central Log Server Node:
    • Running on RHEL 6.x OS

Setup

Central Log Server Node:

Node A & B:

input {
  file {
    path => "/opt/xyz/App/jboss-eap-6.0/domain/servers/ APP-linpubb123/log/xyzoperational.log"
                type => "XYZ_App_Operational"
  }
}

filter {
  multiline { pattern => "^%{TIME}"  what => "previous" negate => "true" }
}

output {
  elasticsearch { host => "10.133.8.92" }
  stdout { codec => rubydebug }
}

·         Rung Logstash via command:

o    nohup ./bin/logstash -f mytest.conf > /dev/null 2>&1 &

Node C:

input {
file { path => "/appl/abc/logs/abc_sys0.log" type => "ABC_ENTRY" }
file { path => "/appl/afm/logs/def_sys0.log" type => "DEF_ENTRY" }
}
filter { multiline { pattern => "^%{MONTH}"  what => "previous" negate => "true" } }
output { elasticsearch_http { host => "10.133.8.92" } }

·         Rung Logstash via command:

o    nohup /usr/jdk/jdk1.7.0_45/bin/java -d64 -jar logstash-1.1.9-monolithic.jar agent -f mytest.conf > /dev/null 2>&1 &

Kibana Setup:

·         Login to Central Log Server Node

·         Download kibana under folder /var/www/html/kibana-3.1.0

·         Edit kibana-3.1.0/config.js file with entry (put the hostname of the node):

o    elasticsearch: http://<hostname-of-log-server>:9200,

·         Restart snmpd service

Testing

  • Access the URL from you browser:
    • http://<hostname-of-log-server>/kibana-3.1.0/