Elasticsearch replication of other system data?

by

You’ve pretty much listed the two main options there are when it comes to search across multiple data stores, i.e. search in one central data store (option #1) or search in all data stores and aggregate the results (option #2).

Both options would work, although option #2 has two main drawbacks:

  1. It will require a substantial amount of logic to be developed in your application in order to “branch out” the searches to the multiple data stores and aggregate the results you get back.
  2. The response times might be different for each data store, and thus, you will have to wait for the slowest data store to respond in order to present the search results to the user (unless you circumvent this by using different asynchronous technologies, such as Ajax, websocket, etc)

If you want to provide a better and more reliable search experience, option #1 would clearly get my vote (I take this way most of the time actually). As you’ve correctly stated, the main “drawback” of this option is that you need to keep Elasticsearch in synch with the changes in your other master data stores.

Since your other data stores will be relational databases, you have a few different options to keep them in synch with Elasticsearch, namely:

These first two options work great but have one main disadvantage, i.e. they don’t capture DELETEs on your table, they will only capture INSERTs and UPDATEs. This means that if you ever delete a user, account, etc, you will not be able to know that you have to delete the corresponding document in Elasticsearch. Unless, of course, you decide to delete the Elasticsearch index before each import session.

To alleviate this, you can use another tool which bases itself on the MySQL binlog and will thus be able to capture every event. There’s one written in Go, one in Java and one in Python.

UPDATE:

Here is another interesting blog article on the subject: How to keep Elasticsearch synchronized with a relational database using Logstash

More Related Contents:

  1. Shards and replicas in Elasticsearch
  2. elasticsearch bool query combine must with OR
  3. ElasticSearch – Return Unique Values
  4. Elasticsearch: Failed to connect to localhost port 9200 – Connection refused
  5. Calculating time between events
  6. Aggregation + sorting +pagination in elastic search
  7. Make elasticsearch only return certain fields?
  8. Elasticsearch: Difference between “Term”, “Match Phrase”, and “Query String”
  9. Join query in ElasticSearch
  10. UTF8 encoding is longer than the max length 32766
  11. Queries vs. Filters
  12. Change default mapping of string to “not analyzed” in Elasticsearch
  13. Find distinct values, not distinct counts in elasticsearch
  14. elasticsearch match vs term query
  15. Elasticsearch vs Cassandra vs Elasticsearch with Cassandra
  16. Elasticsearch difference between MUST and SHOULD bool query
  17. How to change Elasticsearch max memory size
  18. Find documents with empty string value on elasticsearch
  19. How to get latest values for each group with an Elasticsearch query?
  20. What is the maximum Elasticsearch document size?
  21. How to create unique constraint in Elasticsearch database?
  22. How to iterate through a nested array in elasticsearch with filter script?
  23. ElasticSearch: Unassigned Shards, how to fix?
  24. Content-Type header [application/x-www-form-urlencoded] is not supported on Elasticsearch
  25. low disk watermark [??%] exceeded on
  26. Elasticsearch – set max_clause_count
  27. How to bind Elasticsearch 2.0 on both Loopback and Non-Loopback interfaces?
  28. How to use the official docker elasticsearch container?
  29. Remove a field from a Elasticsearch document
  30. How to create multiple indexes in logstash.conf file?

Leave a Comment