low disk watermark [??%] exceeded on

by

If you like me have a lot of disk you can tune the watermark setting and use byte values instead of percentages:

NB! Use either percentage values or byte values.

You cannot mix the usage of percentage/ratio values and byte values within the watermark settings. Either all values are set to percentage/ratio values, or all are set to byte values.

Setting: cluster.routing.allocation.disk.watermark.low

Controls the low watermark for disk usage. It defaults to 85%, meaning that Elasticsearch will not allocate shards to nodes that have more than 85% disk used. It can alternatively be set to a ratio value, e.g., 0.85. It can also be set to an absolute byte value (like 500mb) to prevent Elasticsearch from allocating shards if less than the specified amount of space is available. This setting has no effect on the primary shards of newly-created indices but will prevent their replicas from being allocated.

Setting: cluster.routing.allocation.disk.watermark.high

Controls the high watermark. It defaults to 90%, meaning that Elasticsearch will attempt to relocate shards away from a node whose disk usage is above 90%. It can alternatively be set to a ratio value, e.g., 0.9. It can also be set to an absolute byte value (similarly to the low watermark) to relocate shards away from a node if it has less than the specified amount of free space. This setting affects the allocation of all shards, whether previously allocated or not.

Setting:: cluster.routing.allocation.disk.watermark.flood_stage

Controls the flood stage watermark, which defaults to 95%. Elasticsearch enforces a read-only index block (index.blocks.read_only_allow_delete) on every index that has one or more shards allocated on the node, and that has at least one disk exceeding the flood stage. This setting is a last resort to prevent nodes from running out of disk space. The index block is automatically released when the disk utilization falls below the high watermark. Similarly to the low and high watermark values, it can alternatively be set to a ratio value, e.g., 0.95, or an absolute byte value.

https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cluster.html#disk-based-shard-allocation

Please note:

Percentage values refer to used disk space, while byte values refer to free disk space. This can be confusing, since it flips the meaning of high and low. For example, it makes sense to set the low watermark to 10gb and the high watermark to 5gb, but not the other way around.

On my 5TB disk I’ve set:

# /etc/elasticsearch/elasticsearch.yml
cluster.routing.allocation.disk.threshold_enabled: true
cluster.routing.allocation.disk.watermark.flood_stage: 5gb
cluster.routing.allocation.disk.watermark.low: 30gb
cluster.routing.allocation.disk.watermark.high: 20gb

More Related Contents:

  1. Elasticsearch replication of other system data?
  2. How to search for a part of a word with ElasticSearch
  3. no [query] registered for [filtered]
  4. ES keeps returning every document
  5. Elasticsearch: Failed to connect to localhost port 9200 – Connection refused
  6. Calculating time between events
  7. Make elasticsearch only return certain fields?
  8. Remove duplicate documents from a search in Elasticsearch
  9. Elasticsearch 2.1: Result window is too large (index.max_result_window)
  10. Max limit on the number of values I can specify in the ids filter or generally query clause?
  11. How to update multiple documents that match a query in elasticsearch
  12. Elasticsearch searchable synthetic fields
  13. Why do I need “store”:”yes” in elasticsearch?
  14. How to not-analyze in ElasticSearch?
  15. Find distinct values, not distinct counts in elasticsearch
  16. Save Spark Dataframe into Elasticsearch – Can’t handle type exception
  17. How to retrieve unique count of a field using Kibana + Elastic Search
  18. FIELDDATA Data is too large
  19. Elasticsearch difference between MUST and SHOULD bool query
  20. How to change Elasticsearch max memory size
  21. Can We Retrieve Previous _source Docs with Elastic Search Versions
  22. What is the maximum Elasticsearch document size?
  23. Elasticsearch, Tire, and Nested queries / associations with ActiveRecord
  24. How to iterate through a nested array in elasticsearch with filter script?
  25. ElasticSearch: Unassigned Shards, how to fix?
  26. Content-Type header [application/x-www-form-urlencoded] is not supported on Elasticsearch
  27. Elasticsearch, Failed to obtain node lock, is the following location writable
  28. elasticsearch – querying multiple indexes is possible?
  29. ElasticSearch – Get only matching nested objects with All Top level fields in search response
  30. How to use the official docker elasticsearch container?

Leave a Comment