elasticsearch

multiple inputs on logstash jdbc

You can definitely have a single config with multiple jdbc input and then parametrize the index and document_type in your elasticsearch output depending on which table the event is coming from. input { jdbc { jdbc_driver_library => “/Users/logstash/mysql-connector-java-5.1.39-bin.jar” jdbc_driver_class => “com.mysql.jdbc.Driver” jdbc_connection_string => “jdbc:mysql://localhost:3306/database_name” jdbc_user => “root” jdbc_password => “password” schedule => “* * * … Read more

ElasticSearch and Regex queries

You should read Elasticsearch’s Regexp Query documentation carefully, you are making some incorrect assumptions about how the regexp query works. Probably the most important thing to understand here is what the string you are trying to match is. You are trying to match terms, not the entire string. If this is being indexed with StandardAnalyzer, … Read more

How to index a pdf file in Elasticsearch 5.0.0 with ingest-attachment plugin?

You need to make sure you have created your ingest pipeline with: PUT _ingest/pipeline/attachment { “description” : “Extract attachment information”, “processors” : [ { “attachment” : { “field” : “data”, “indexed_chars” : -1 } } ] } Then you can make a PUT not POST to your index using the pipeline you’ve created. PUT my_index/my_type/my_id?pipeline=attachment … Read more

ElasticSearch – Get only matching nested objects with All Top level fields in search response

If you’re ok with having all root fields except the nested one and then only the matching inner hits in the nested field, then we can re-use the previous answer like this by specifying a slightly more involved source filtering parameter: { “_source”: { “includes”: [ “*” ], “excludes”: [ “users” ] }, “query”: { … Read more

elasticsearch – querying multiple indexes is possible?

This is quite easy within Elasticsearch itself! Anytime you would specify an index, you can separate additional indices by comma. curl -XGET ‘http://localhost:9200/index1,index2/_search?q=yourQueryHere’ You can also search all indices with _all. curl -XGET ‘http://localhost:9200/_all/_search?q=yourQueryHere’ Here’s some helpful documentation from elasticsearch’s website. This site has TONS of info, but it is a bit difficult to find … Read more

low disk watermark [??%] exceeded on

If you like me have a lot of disk you can tune the watermark setting and use byte values instead of percentages: NB! Use either percentage values or byte values. You cannot mix the usage of percentage/ratio values and byte values within the watermark settings. Either all values are set to percentage/ratio values, or all … Read more