Install Kibana 4 and Elasticsearch on Ubuntu

Blog, ElasticSearch, Information Technology, Kibana, Software

Great news, everyone! Kibana 4 is no longer in beta and has been released to the masses.

In this guide, I will show you how to get started installing Kibana 4 and Elasticsearch in a linux environment. I will be using Ubuntu Server 14.04 in my examples.

For those who do not know, Kibana is an interface to data exploration through graphical visualizations. Kibana 4 is the latest release from the people over at Elastic. Kibana 4 offers a much better interface for sorting through your data.

Download and Install Java

If you are doing this on a fresh install of Ubuntu (like me), the first thing you’re going to need to do is install Java. Elasticsearch requires at least Java 7 to run so let’s set that up. I will be using Java 8 in this example but you can run 7 or openjdk if you wish.

Open a terminal window (ctrl+shift+t) and type…

sudo apt-add-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java8-installer

Once you have accepted the license agreement, Java is ready to go.

Download and Install Elasticsearch

Open another terminal window (ctrl+alt+t) and type…

cd ~
wget -O - http://packages.elasticsearch.org/GPG-KEY-elasticsearch | sudo apt-key add -
echo 'deb http://packages.elasticsearch.org/elasticsearch/1.5/debian stable main' | sudo tee /etc/apt/sources.list.d/elasticsearch.list
sudo apt-get update
sudo apt-get install elasticsearch

That commands above will install Elasticsearch on your machine.

The next thing we need to do is to give our Elasticsearch cluster a name. To do this, type in this commands…

sudo sed -i -e 's|#cluster.name: elasticsearch|cluster.name: kibana|' /etc/elasticsearch/elasticsearch.yml

Now you need to tell your machine to run Elasticsearch every time the machine boots up. To do that, type…

sudo update-rc.d elasticsearch defaults 95 10
sudo /etc/init.d/elasticsearch start

And now you have Elasticsearch running on your machine. You can check by typing “sudo service elasticsearch status” in the terminal window, or by opening up a browser and going to http://127.0.0.1:9200 and seeing a result similar to this…

{
  "status" : 200,
  "name" : "Karen Page",
  "cluster_name" : "kibana",
  "version" : {
    "number" : "1.5.2",
    "build_hash" : "62ff9868b4c8a0c45860bebb259e21980778ab1c",
    "build_timestamp" : "2015-04-27T09:21:06Z",
    "build_snapshot" : false,
    "lucene_version" : "4.10.4"
  },
  "tagline" : "You Know, for Search"
}

Download and Install Kibana 4

Now that Elasticsearch is installed on our machine, we can now go ahead and setup Kibana 4. Unlike previous versions of Kibana, Kibana 4 does not use apache to run.

Open up another terminal window (ctrl+shift+t) and type in these commands…

cd ~
wget https://download.elastic.co/kibana/kibana/kibana-4.1.0-linux-x64.tar.gz
tar xvf kibana-4.1.0-linux-x64.tar.gz
sudo mv ~/kibana-4.1.0-linux-x64 /opt/kibana

Now we need to have Kibana 4 start up when the machine boots so we need to have it run as a service. To do that, type in these commands in a terminal window…

sudo wget --output-document="/etc/init.d/kibana4" https://raw.githubusercontent.com/akabdog/scripts/master/kibana4_init
sudo chmod +x /etc/init.d/kibana4
sudo update-rc.d kibana4 defaults 96 9
sudo service kibana4 start

You can check to see if Kibana is working properly by goint to http://127.0.0.1:5601/ in your browser of choice. And that is in in terms of installing Kibana 4 and Elasticsearch. Part 2 of this blog post on setting up Logstash for your Cisco ASA syslogs is over here.

Create a Custom Elasticsearch Template

Blog, ElasticSearch, Information Technology, Networking, Software

This post will show you how to create a custom index template for Elasticsearch.

Why would you need to create a custom template Lets say you are storing ASN data in your Elasticsearch index.
For example…

Google Inc. – 15 messages
Facebook Inc. – 25 messages
Linkedin Inc. – 33 messages

When you query the index for ASN fields, you are going to get 15 hits for Google, 25 hits for Facebook, 33 hits for Linkedin and 73 hits for Inc. This is because, by default, ElasticSearch does automatic index creation which analyzes each field and splits strings at spaces when indexing. So what if you want to just have the whole field item return as a result so that something like “Google Inc” will show up as 15 hits? Well, you have to create an ElasticSearch index template. Note: creating a template will not magically modify old indexes, that data has already been indexed. The template will only work for newly created indices in ElasticSearch after you add the template.

First thing you need to do is figure out the naming scheme for your indices. Knowing the name pattern for new indices will make it so that the template you are about to create only applies to that index and not other indices in ElasticSearch. I use logstash to ship everything to ElasticSearch and the default index naming pattern is logstash-YYYY-MM-DD so, iny my template, I will have logstash* with the asterisk acting as a wildcard. If you’re not using logstash and are unsure of the naming, go to /var/lib/elasticsearch and look in the indices folder to see the names of your current indices. Remember this for when we create the template.

Next you want to find the name inside of a current index so the template will only match the types you want it to match. Open your browser and type http://localhost:9200/_all/_mapping?pretty=1 in the URL bar and hit enter.

For example: I see

"logstash-2014.09.30": {
        "cisco-fw": {
            "properties": {

Because I have my logstash config file set to specify anything coming in on a certain port as type cisco-fw and because of that, the type in ElasticSearch is cisco-fw but yours might be default or something else. Remember this for when we create the template

Next thing is open up a notepad so we can start creating the template.

Here is the template for your template

curl -XPUT http://localhost:9200/_template/logstash_per_index -d '
{
    "template" : "logstash*",
    "mappings" : {
      "cisco-fw" : {
      }
    }
}'

Where logstash_per_index is the name you want to give the template, logstash* is the index naming scheme and cisco-fw is the type.

Now, under properties, you are going to set the field type and options based on field name. For my example, I am doing ASN values, so, under properties, I would write

"asn":{"type":"string", "index":"not_analyzed"}

The ASN type is a string (obviously) and index is set to not_analyzed. Not_analyzed means that it is still searchable with a query, but it does not go through any analysis process and is not broken down into tokens. This will allow us to see Google Inc as one result when querying ElasticSearch. Do this for all the fields in your index. For example, here is my completed template…

#!/bin/sh
curl -XPUT http://localhost:9200/_template/logstash_per_index -d '
{
    "template" : "logstash*",
    "mappings" : {
      "cisco-fw" : {
         "properties": {
            "@timestamp":{"type":"date","format":"dateOptionalTime"},
            "@version":{"type":"string", "index" : "not_analyzed"},
            "action":{"type":"string"},
            "bytes":{"type":"long"},
            "cisco_message":{"type":"string"},
            "ciscotag":{"type":"string", "index" : "not_analyzed"},
            "connection_count":{"type":"long"},
            "connection_count_max":{"type":"long"},
            "connection_id":{"type":"string"},
            "direction":{"type":"string"},
            "dst_interface":{"type":"string"},
            "dst_ip":{"type":"string"},
            "dst_mapped_ip":{"type":"ip"},
            "dst_mapped_port":{"type":"long"},
            "dst_port":{"type":"long"},
            "duration":{"type":"string"},
            "err_dst_interface":{"type":"string"},
            "err_dst_ip":{"type":"ip"},
            "err_icmp_code":{"type":"string"},
            "err_icmp_type":{"type":"string"},
            "err_protocol":{"type":"string"},
            "err_src_interface":{"type":"string"},
            "err_src_ip":{"type":"ip"},
            "geoip":{
               "properties":{
                  "area_code":{"type":"long"},
                  "asn":{"type":"string", "index":"not_analyzed"},
                  "city_name":{"type":"string", "index":"not_analyzed"},
                  "continent_code":{"type":"string"},
                  "country_code2":{"type":"string"},
                  "country_code3":{"type":"string"},
                  "country_name":{"type":"string", "index":"not_analyzed"},
                  "dma_code":{"type":"long"},
                  "ip":{"type":"ip"},
                  "latitude":{"type":"double"},
                  "location":{"type":"geo_point"},
                  "longitude":{"type":"double"},
                  "number":{"type":"string"},
                  "postal_code":{"type":"string"},
                  "real_region_name":{"type":"string", "index":"not_analyzed"},
                  "region_name":{"type":"string", "index":"not_analyzed"},
                  "timezone":{"type":"string"}
               }
            },
            "group":{"type":"string"},
            "hashcode1": {"type": "string"},
            "hashcode2": {"type": "string"},
            "host":{"type":"string"},
            "icmp_code":{"type":"string"},
            "icmp_code_xlated":{"type":"string"},
            "icmp_seq_num":{"type":"string"},
            "icmp_type":{"type":"string"},
            "interface":{"type":"string"},
            "is_local_natted":{"type":"string"},
            "is_remote_natted":{"type":"string"},
            "message":{"type":"string"},
            "orig_dst_ip":{"type":"ip"},
            "orig_dst_port":{"type":"long"},
            "orig_protocol":{"type":"string"},
            "orig_src_ip":{"type":"ip"},
            "orig_src_port":{"type":"long"},
            "policy_id":{"type":"string"},
            "protocol":{"type":"string"},
            "reason":{"type":"string"},
            "seq_num":{"type":"long"},
            "spi":{"type":"string"},
            "src_interface":{"type":"string"},
            "src_ip":{"type":"string"},
            "src_mapped_ip":{"type":"ip"},
            "src_mapped_port":{"type":"long"},
            "src_port":{"type":"long"},
            "src_xlated_interface":{"type":"string"},
            "src_xlated_ip":{"type":"ip"},
            "syslog_facility":{"type":"string"},
            "syslog_facility_code":{"type":"long"},
            "syslog_pri":{"type":"string"},
            "syslog_severity":{"type":"string"},
            "syslog_severity_code":{"type":"long"},
            "tags":{"type":"string"},
            "tcp_flags":{"type":"string"},
            "timestamp":{"type":"string"},
            "tunnel_type":{"type":"string"},
            "type":{"type":"string"},
            "user":{"type":"string"},
            "xlate_type":{"type":"string"}
      }
    }
  }
}'

Once you change all the types you need, it is now time to add the template to ElasticSearch. You can either save you file in notepad, make it a script and run that through a terminal window or you can copy the text from notepad and enter into a terminal window. You should see a {“ok”:true,”acknowledged”:true} response if everything was formatted properly.
ACK

And that is it. You will only see the fruits of your labor when a new index is created and matches the parameters set in your template file (i.e logstash*). Because of my naming scheme with Logstash, new indices are only created at the start of the day (logstash-YYYY-MM-DD) so I had to wait until the next day to see if my template was working properly. If you are impatient, cannot wait to see if it worked or not and don’t care about losing data in your current index then you can delete it from ElasticSearch by issuing the following CURL command in a terminal window

curl -XDELETE localhost:9200/index_name

where index_name is the name of your index (ex. logstash-2014-12-11)

Helpful tip: if you start seeing data not show up in the index, it is very possible that you may have messed up one of the field types in you template file. I am writing this because I ran into this issue and could not figure out why there was no data in my index. To solve this, go to /var/log/elasticsearch and see the log file for the date where data was not properly going into the ElasticSearch index (it should be a lot bigger in file size compared to the other log files). In the log file, I was seeing this error multiple times

org.elasticsearch.index.mapper.MapperParsingException: failed to parse [protocol]

What happened was that, thinking protocol meant port number protocol (ex. 25, 80, 443 etc), I set the protocol field as type LONG. To my surprise, protocol was either TCP or UDP so it should have been set as type string. ElasticSearch was expecting a long to index based off my template but instead was getting strings so the application freaked out. Instead of modifying the template file on the server, I decided to delete it from ElasticSearch, make my changes to the protocol field and then re-upload the template back to ElasticSearch. To do that, I opened a terminal and typed

curl -XDELETE http://localhost:9200/_template/logstash_per_index

where logstash_per_index is the name of the template. That command will delete the template off of your server. Make your changes to your template in notepad and then add the template back to ElasticSearch.

Since the template only applies to newly created indices and your index did not have any data inside of it because of the incorrect template, you can go ahead and just delete that index and create a new one that will work with the newly modified template.

curl -XDELETE localhost:9200/index_name

where index_name is the name of your index (ex. logstash-2014-12-11).

And that is it. Leave a comment down below if you found this information helpful or if you have any questions for me. Good luck!

Install Kibana 4 Beta on Ubuntu

Blog, ElasticSearch, Information Technology, Kibana, Networking, Software

NOTE: This post is out of date as Kibana 4 is no longer in beta. Check out this blog post on how to install a stable release of Kibana 4.

Good news, everyone! The Kibana 4 beta has been released.

If you like to live life on the bleeding edge like me and want to mess around with the new features then follow this guide below. This guide will show you how to install the beta on Ubuntu so you can play around with it before the final version is released to the masses. This guide assumes you have a fresh install of Ubuntu to run this on.

Download and Install Elasticsearch

First thing you will need to do is download and install the latest version of Elasticsearch. You need to have Elasticsearch 1.40 for the beta to work. Running anything earlier than 1.4.0 will result in this error in Kibana

You need to have Java on you machine for this to work. If you are not sure on whether or not you have Java on your machine, open up a terminal window (ctrl + alt + t) and type java -version

If you get java version “1.7.0_65” or similar then skip this next step, otherwise type this into your terminal window

sudo apt-get update
sudo apt-get install openjdk-7-jre-headless -y

Once java is installed, open a terminal session (ctrl + alt + t) and enter these commands.

 cd ~
 sudo apt-get update
 wget https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.4.0.deb
 sudo dpkg -i elasticsearch-1.4.0.deb

This will install Elasticsearch 1.4.0 stable on your machine.

Next you need to configure Elasticsearch on your machine.

sudo sed -i -e 's|# cluster.name: elasticsearch|cluster.name: kibana|' /etc/elasticsearch/elasticsearch.yml

Now you need to tell your machine to run elasticsearch on boot.

sudo update-rc.d elasticsearch defaults 95 10
sudo service elasticsearch restart

Download and Install Kibana 4 Beta

Unlike previous versions of Kibana, this version does not require that you have Apache on your machine so this makes this install much easier for fresh installs.

First download the Kibana 4 beta from here

Extract the archive
In the root dir, open config/kibana.yml in an editor
Set the elasticsearch parameter to the fully qualified hostname of your Elasticsearch server. For us its going to be on the same machine so you can keep the file as is.
Run ./bin/kibana from a terminal window (ctrl + alt + t) & cd to you Kibana root directory
Open your browser to Kibana. (ex: http://127.0.0.1:5601)

And that is it. Start inserting data into your Elasticsearch index and start playing around with the beta.

Jack's Website

Whatever is on my mind

Elasticsearch

Install Kibana 4 and Elasticsearch on Ubuntu

Create a Custom Elasticsearch Template

Install Kibana 4 Beta on Ubuntu