You can use a pattern in your index name based on the value of one of your fields. Here we use the value of the type field in order to name the index: output { stdout {codec => rubydebug} elasticsearch { host => “localhost” protocol => “http” index => “%{type}_indexer” } } You can also … Read more
You can use the aggregate filter in order to do this. The aggregate filter provides support for aggregating several log lines into one single event based on a common field value. In your case, the common field would be the job_id field. Then we need another field to detect the first event vs the second … Read more
You could use multiple patterns for your grok filter, grok { match => [“fieldname”, “pattern1”, “pattern2”, …, “patternN”] } and they will be applied in order but a) it’s not the best option performance-wise and b) you probably want to treat different types of logs differently anyway, so I suggest you use conditionals based on … Read more
After your json filter add another one called mutate in order to add the two fields that you would take from the parsedJson field. filter { … json { … } mutate { add_field => { “firstname” => “%{[parsedJson][firstname]}” “lastname” => “%{[parsedJson][lastname]}” } } } For your sample log line above that would give: { … Read more
You can definitely have a single config with multiple jdbc input and then parametrize the index and document_type in your elasticsearch output depending on which table the event is coming from. input { jdbc { jdbc_driver_library => “/Users/logstash/mysql-connector-java-5.1.39-bin.jar” jdbc_driver_class => “com.mysql.jdbc.Driver” jdbc_connection_string => “jdbc:mysql://localhost:3306/database_name” jdbc_user => “root” jdbc_password => “password” schedule => “* * * … Read more
For Kibana 4 go to this answer This is easy to do with a terms panel: If you want to select the count of distinct IP that are in your logs, you should specify in the field clientip, you should put a big enough number in length (otherwise, it will join different IP under the … Read more
Just create a template. run curl -XPUT localhost:9200/_template/template_1 -d ‘{ “template”: “*”, “settings”: { “index.refresh_interval”: “5s” }, “mappings”: { “_default_”: { “_all”: { “enabled”: true }, “dynamic_templates”: [ { “string_fields”: { “match”: “*”, “match_mapping_type”: “string”, “mapping”: { “index”: “not_analyzed”, “omit_norms”: true, “type”: “string” } } } ], “properties”: { “@version”: { “type”: “string”, “index”: “not_analyzed” … Read more
Hi I added a grok filter to do just this. I only wanted to have the filename not the path, but you can change this to your needs. filter { grok { match => [“path”,”%{GREEDYDATA}/%{GREEDYDATA:filename}\.log”] } }
As of ES 5.x , they have given this feature out of the box with logstash plugin. This will periodically import data from database and push to ES server. One has to create a simple import file given below (which is also described here) and use logstash to run the script. Logstash supports running this … Read more
This is what you’re looking for: https://github.com/ForgeRock/es-change-feed-plugin Using this plugin, you can register to a websocket channel to receive indexation/deletion events as they happen. It has some limitations, though. Back in the days, it was possible to install river plugins to stream documents to ES. The river feature has been removed, but this plugin above … Read more