Elastic search, multiple indexes vs one index and types for different data sets?

by

There are different implications to both approaches.

Assuming you are using Elasticsearch’s default settings, having 1 index for each model will significantly increase the number of your shards as 1 index will use 5 shards, 5 data models will use 25 shards; while having 5 object types in 1 index is still going to use 5 shards.

Implications for having each data model as index:

  • Efficient and fast to search within index, as amount of data should be smaller in each shard since it is distributed to different indices.
  • Searching a combination of data models from 2 or more indices is going to generate overhead, because the query will have to be sent to more shards across indices, compiled and sent back to the user.
  • Not recommended if your data set is small since you will incur more storage with each additional shard being created and the performance gain is marginal.
  • Recommended if your data set is big and your queries are taking a long time to process, since dedicated shards are storing your specific data and it will be easier for Elasticsearch to process.

Implications for having each data model as an object type within an index:

  • More data will be stored within the 5 shards of an index, which means there is lesser overhead issues when you query across different data models but your shard size will be significantly bigger.
  • More data within the shards is going to take a longer time for Elasticsearch to search through since there are more documents to filter.
  • Not recommended if you know you are going through 1 terabytes of data and you are not distributing your data across different indices or multiple shards in your Elasticsearch mapping.
  • Recommended for small data sets, because you will not waste storage space for marginal performance gain since each shard take up space in your hardware.

If you are asking what is too much data vs small data? Typically it depends on the processor speed and the RAM of your hardware, the amount of data you store within each variable in your mapping for Elasticsearch and your query requirements; using many facets in your queries is going to slow down your response time significantly. There is no straightforward answer to this and you will have to benchmark according to your needs.

More Related Contents:

  1. Database of common name aliases / nicknames of people
  2. Elasticsearch query to return all records
  3. Should each and every table have a primary key?
  4. Database, Table and Column Naming Conventions? [closed]
  5. What are the performance improvement of Sequential Guid over standard Guid? [closed]
  6. When and why are database joins expensive?
  7. Normalization: What does “repeating groups” mean?
  8. Copying data from one SQLite database to another
  9. Difference Between Firestore Set with {merge: true} and Update
  10. Can HTML5 databases and localStorage be shared across subdomains?
  11. What is a database transaction?
  12. Relational table naming convention [closed]
  13. When I should use one to one relationship?
  14. What’s the difference between findAndModify and update in MongoDB?
  15. Are foreign keys really necessary in a database design?
  16. GUID vs INT IDENTITY [duplicate]
  17. What is a sequence (Database)? When would we need it?
  18. Storing Images : DB or File System –
  19. Database optimization orders
  20. What is database pooling?
  21. In what way does denormalization improve database performance?
  22. Storing a small number of images: blob or fs?
  23. have address columns in each table or an address table that is referenced by the other tables?
  24. Minimum no of tables that exists after decomposing relation R into 1NF?
  25. How to unit test an object with database queries
  26. Populate Android Database From CSV file?
  27. Creating test data in a database [closed]
  28. When we need to use 1-to-1 relationship in database design?
  29. CAP theorem – Availability and Partition Tolerance
  30. DB design for microservice architecture [closed]

Leave a Comment