Logstash: Merge two logs into one output document

by

You can use the aggregate filter in order to do this. The aggregate filter provides support for aggregating several log lines into one single event based on a common field value. In your case, the common field would be the job_id field.

Then we need another field to detect the first event vs the second event that should be aggregated. In your case, this would be the state field.

So you simply need to add another filter to your existing Logstash configuration, like this:

filter {
    ...your other filters

    if [state] == "processing" {
        aggregate {
            task_id => "%{job_id}"
        }
    } else if [state] == "failed" {
        aggregate {
            task_id => "%{job_id}"
            end_of_task => true
            timeout => 120
        }
    }
}

You are free to adjust the timeout (in seconds) depending on how long your jobs are running.

More Related Contents:

  1. Calculating time between events
  2. Join query in ElasticSearch
  3. Change default mapping of string to “not analyzed” in Elasticsearch
  4. How to retrieve unique count of a field using Kibana + Elastic Search
  5. How to parse json in logstash /grok from a text file line?
  6. How to create multiple indexes in logstash.conf file?
  7. Elasticsearch replication of other system data?
  8. Shards and replicas in Elasticsearch
  9. elasticsearch bool query combine must with OR
  10. ElasticSearch – Return Unique Values
  11. Aggregation + sorting +pagination in elastic search
  12. Elasticsearch: Difference between “Term”, “Match Phrase”, and “Query String”
  13. How can a Elasticsearch client be notified of a new indexed document?
  14. Max limit on the number of values I can specify in the ids filter or generally query clause?
  15. UTF8 encoding is longer than the max length 32766
  16. How to update multiple documents that match a query in elasticsearch
  17. Queries vs. Filters
  18. elasticsearch match vs term query
  19. elasticsearch – what to do with unassigned shards
  20. Elasticsearch vs Cassandra vs Elasticsearch with Cassandra
  21. Find documents with empty string value on elasticsearch
  22. How to get latest values for each group with an Elasticsearch query?
  23. How to create unique constraint in Elasticsearch database?
  24. ElasticSearch: Unassigned Shards, how to fix?
  25. Content-Type header [application/x-www-form-urlencoded] is not supported on Elasticsearch
  26. Elasticsearch, Failed to obtain node lock, is the following location writable
  27. low disk watermark [??%] exceeded on
  28. Elasticsearch – set max_clause_count
  29. How to bind Elasticsearch 2.0 on both Loopback and Non-Loopback interfaces?
  30. How to use the official docker elasticsearch container?

Leave a Comment