What is the maximum Elasticsearch document size?

by

Lucene uses a byte buffer internally that uses 32bit integers for addressing. By definition this limits the size of the documents. So 2GB is max in theory.

In ElasticSearch:

There is a max http request size in the ES GitHub code, and it is set against Integer.MAX_VALUE or 2^31-1. So, basically, 2GB is the maximum document size for bulk indexing over HTTP. And also to add to it, ES does not process an HTTP request until it completes.

Good Practices:

  • Do not use a very large java heap if you can help it: set it only as large as is necessary (ideally no more than half of the machine’s RAM) to hold the overall maximum working set size for your usage of Elasticsearch. This leaves the remaining (hopefully sizable) RAM for the OS to manage for IO caching.
  • In client side, always use the bulk api, which indexes multiple documents in one request, and experiment with the right number of documents to send with each bulk request. The optimal size depends on many factors, but try to err in the direction of too few rather than too many documents. Use concurrent bulk requests with client-side threads or separate asynchronous requests.

For further study refer to these links:

  1. Performance considerations for elasticsearch indexing

  2. Document maximum size for bulk indexing over HTTP

More Related Contents:

  1. Elasticsearch replication of other system data?
  2. How to search for a part of a word with ElasticSearch
  3. no [query] registered for [filtered]
  4. elasticsearch bool query combine must with OR
  5. ElasticSearch – Return Unique Values
  6. Remove duplicate documents from a search in Elasticsearch
  7. Elasticsearch 2.1: Result window is too large (index.max_result_window)
  8. Max limit on the number of values I can specify in the ids filter or generally query clause?
  9. UTF8 encoding is longer than the max length 32766
  10. How to update multiple documents that match a query in elasticsearch
  11. Queries vs. Filters
  12. Why do I need “store”:”yes” in elasticsearch?
  13. Change default mapping of string to “not analyzed” in Elasticsearch
  14. How to not-analyze in ElasticSearch?
  15. elasticsearch – what to do with unassigned shards
  16. Elasticsearch vs Cassandra vs Elasticsearch with Cassandra
  17. FIELDDATA Data is too large
  18. Can We Retrieve Previous _source Docs with Elastic Search Versions
  19. Find documents with empty string value on elasticsearch
  20. Elasticsearch, Tire, and Nested queries / associations with ActiveRecord
  21. How to create unique constraint in Elasticsearch database?
  22. Elasticsearch, Failed to obtain node lock, is the following location writable
  23. elasticsearch – querying multiple indexes is possible?
  24. ElasticSearch – Get only matching nested objects with All Top level fields in search response
  25. ElasticSearch: aggregation on _score field?
  26. How to parse json in logstash /grok from a text file line?
  27. Logstash: Merge two logs into one output document
  28. Elasticsearch error: cluster_block_exception [FORBIDDEN/12/index read-only / allow delete (api)], flood stage disk watermark exceeded
  29. What is the ElasticSearch equivalent for an SQL subquery?
  30. How to create multiple indexes in logstash.conf file?

Leave a Comment