26 thoughts on “Create a Custom Elasticsearch Template”

1. 

   We have custom application related log file.

   (1) how do we determine the various properties from a log file ?
   (2) How does the property name mapped to the real entry in the log file ?
   (3) How to distinguish among themselves ?

   1. 

      (1) how do we determine the various properties from a log file ?

      You just have to know and that may be through just trial and error. What I would do first is create an index and insert your custom data and see how elasticsearch sets that type to be. Then you can see the mapping by issuing curl -XGET 'http://localhost:9200/index_name/_mapping/' to see what it did. Elasticsearch is not perfect, if it is not sure then it will usually set the type as a string. There are a few types to choose from, you just have to look at your data and determine what the type should be. You can see all the types here…http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-types.html Basically you have

      Type	Definition
      string	Text
      integer	32 bit integers
      long	64 bit integers
      float	IEEE float
      double	Double precision floats
      boolean	true or false
      date	UTC Date/Time
      geo_point	Latitude / Longitude
      ip	Numeric ipv4 address

      (2) How does the property name mapped to the real entry in the log file ?

      (3) How to distinguish among themselves ?

      Not sure what you mean here. Care to elaborate?

2. 

   Thanks. Can you please post your logstash.conf associated with your template file ?

   1. 

      Sure, it’s in this blog post http://jackhanington.com/blog/2014/04/21/using-logstash-elasticsearch-and-kibana-for-cisco-asa-syslog-message-analysis/

3. 

   This ELK stack is a fast moving target 🙂
   Indices are now here: /var/lib/elasticsearch/elasticsearch/nodes/0/indices.

   I’m using a test system and there is 1 node, but judging by the path structure indices are probably listed per node.

4. Pingback: 안정적인 서비스 운영을 위한 서버 모니터링 #2 | DRAMA&COMPANY Engineering
5. Pingback: URL
6. 

   Can you please explain the steps to execute the template in windows
   1: I have installed cygwin in my machine
   2: basic curl commands are working
   Question : If Im not running elastic search as a services im just pushing the log with input as a file then
   how could I test this ? we should send the template after sending the log or before?

   1. 

      Cygwin works for simple things but I find it doesn’t work well for things like multi-line curl posts. If I were doing this on windows, the better option I would do is to install the poster extension on firefox https://addons.mozilla.org/en-US/firefox/addon/poster/ and post through that. http://i.imgur.com/yYFDcV8.png

      You do not have to run elasticsearch as a service. You can run it through the command prompt.
      1. Open the command prompt
      2. Change to the root of the elasticsearch dir (ex: cd C:\ELK\elasticsearch)
      3. Run the bat file. (bin\elasticsearch.bat)

      Then you can just post to 127.0.0.1:9200

      You want to post the template before sending the log file because once the file is indexed in elasticsearch then you’re done and the template has no outcome.

   2. 

      Cygwin works for simple things but I find it doesn’t work well for things like multi-line curl posts. If I were doing this on windows, the better option I would do is to install the poster extension on firefox https://addons.mozilla.org/en-US/firefox/addon/poster/ and post through that. http://i.imgur.com/yYFDcV8.png

      You do not have to run elasticsearch as a service. You can run it through the command prompt.
      1. Open the command prompt
      2. Change to the root of the elasticsearch dir (ex: cd C:\ELK\elasticsearch)
      3. Run the bat file. (bin\elasticsearch.bat)

      Then you can just post to 127.0.0.1:9200

      You want to post the template before sending the log file because once the file is indexed in elasticsearch then you’re done and the template has no outcome

7. 

   Hi Jack. Thanks for your sharing. Can I consult you a question? I also created a custom index template named “ddl_template”, but it failed after working some minutes. And the template “ddl_template” still exists, but it’s content has been changed to the content of the default template “logstash”.

   The following is parts of the content of “ddl_template”:
   #!/bin/sh
   curl -XPUT ‘http://localhost:9200/_template/ddl_template’ -d ‘
   {
   “template”: “ddl*”,
   “order”: 1,
   “settings”: {
   “index.number_of_shards”: 5,
   “number_of_replicas”: 1,
   “index.refresh_interval”: “5s”
   },
   “aliases”: {
   “ddl.alias”: { }
   },
   “mappings”: {
   “ddl”: {
   “_source”: {“enabled” : true},
   “_all”: {“enabled” : false},
   “properties”: {
   “@timestamp”: {
   “type”: “date”,
   “format”: “dateOptionalTime”,
   “doc_values”: true
   }
   }
   }
   }
   }’

   If you have some points about my question, please send email to me. Thank you!

   1. 

      Hello Jason.

      Because it is setting to the default logstash template, I am wondering if the setting in logstash is the issue. “template”: “ddl*” is set for any index that starts with ddl. Logstash by default sets the index to Logstash-YYYY.MM.DD. So, if you have this…

      elasticsearch {
      hosts => “10.0.0.123”
      }

      Then logstash will just go to Logstash-YYYY.MM.DD in elasticsearch

      You need to specify the index in your logstash config. So like this…
      output {
      elasticsearch {
      hosts => “10.0.0.123”
      index => “ddl-%{+YYYY.MM.dd}”
      }
      }

      and that should match the template. Let me know if this works.

8. 

   Hi Jack,

   Thank you for your answer. I found the reason. It is not the problem of custom template, but the problem of the setting in logstash output elasticsearch. I take elasticsearch as the output of logstash, and want to use my custom template ddl_template. In fact, I don’t need to do anything in logstash output elasticsearch but to create ddl_template with REST API on Elasticsearch. However, I make some setting in logstash output elasticsearch such as the attributes “manage_template”, “template_name”, “template_overwrite”, which in fact are used for template pushed by logstash and I do nothing on it previously.

   Sum up, I think there are two methods to use custom index template. One is creating template with REST API on Elasticsearch (I used), the other one is pushing template by logstash (need to set for logstash output elasticsearch). Besides, the first way needs to set the attribute “protocol”. However, I failed to test the second method, but I am sure this method works.

   I studied Elasticsearch less than three months, thus maybe there are some problems for my points, I am pleasure to communicate with you. Thanks again, Jack. (ps: English is not my mother language, if there are any points you can not read, please forgive me. ^_^ )

9. 

   First I would like to state that I am new to Elasticsearch, similiar to the above question. I am attempting to create a new index within elasticsearch. I have specified within my logstash.conf to set the index to “mls-%{+YYYY.MM.dd}”. After recycling my elasticsearch instance I expected my logs to be pushed the mls-%{+YYYY.MM.dd} index within elasticsearch but time after time the logs continually show up in the logstash-%{+YYYY.MM.dd} index.

   besides the configuration file is there another place to set the index. Does the version of elasticsearch matter. I am using the latest.

10. 

    I have created a template that is only matching specific fields, while leaving others not mentioned. I imported the template without issue. And the index was created using the new template. However, my concern is, i can still see the some of the fields i do not want in the the index, even though i didn’t add them to the template. What is happening? Here is a link to my template.

    http://pastebin.com/DUEST8EH

    Thanks in advance

    1. 

       If you do not want to see fields, you have to remove them before they are sent to Elasticsearch using logstash in the filter section using mutate. Ex

       mutate {
       replace => [ “DestinationGeo.location”, “” ]
       }

11. 

    Hello
    I’m new in Elasticsearch , and I need to create a name index automatically. Not if I’m wrong in this example of template. I made the following template.
    curl -XPUT http://localhost:9200/_template/prueba -d ‘
    {
    “template” : “test*”,
    “mappings” : {
    “test” : {
    “properties”: {
    “@timestamp”:{“type”:”date”,”format”:”dateOptionalTime”},
    “@version”:{“type”:”string”, “index” : “not_analyzed”},
    “action”:{“type”:”string”}
    }
    }
    }
    }’

    But when I create an index called test curl -XPUT ‘http://localhost:9200/test/’
    not created the way test- YYYY -MM -DD , What I’m doing wrong ?
    Please if somebody could explain to me, with some example and step by step.

    Thx in advance

    1. 

       The dates come from logstash. What you are doing is just making a index called test. Can you should me the elasticsearch portion of your logstash config file? It should looke like this…

       elasticsearch {
       hosts => “IP.OF.ELASTICSEARCH.SERVER”
       index => “test-%{+YYYY.MM.dd}”
       }

12. 

    Thank you for this. It helped me get through a complex mapping procedure and I really appreciate the time you put into this.

13. 

    Thank you sir jack, this post really helped me. Thanks a tone

14. 

    Hi
    i am not getting {“ok”:true,”acknowledged”:true} when I run my config file in my logstsash.
    output {
    elasticsearch {
    hosts => [“localhost:9200”]
    manage_template => false
    template => “C:\Users\1006541\Desktop\RAMA\template2.txt”
    template_name => “logstash_indexPattern_line”
    action => “index”
    index => “data3”
    }
    stdout {codec => rubydebug}
    }
    I am not getting any sort of output
    is this the correct way to do?
    Thanks in advance!!!

15. 

    hi i tried to run that template code in POSTMAN but it is saying 400 error.

16. 

    hi
    i got {“acknowledged”:true} ..the error is when we run the code in postman.
    correct= POST localhost:9200/_template/testindextemplate
    incorrect=POST localhost:9200/_template/testindextemplate -d

    1. 

       The -d flag is used in bash to specify data which you wish to send from curl in bash.

       Its essentially the post data you want to send however you would attach it using postman 🙂

17. 

    How how do you set with template a certain index will use?

18. 

    Hi Jack ,

    I am facing problem with index template .

    I have created template but still new indexes are not created as per index pattern i have mentioned in template .