Elasticsearch error: cluster_block_exception [FORBIDDEN/12/index read-only / allow delete (api)], flood stage disk watermark exceeded

by

This happens when Elasticsearch thinks the disk is running low on space so it puts itself into read-only mode.

By default Elasticsearch’s decision is based on the percentage of disk space that’s free, so on big disks this can happen even if you have many gigabytes of free space.

The flood stage watermark is 95% by default, so on a 1TB drive you need at least 50GB of free space or Elasticsearch will put itself into read-only mode.

For docs about the flood stage watermark see https://www.elastic.co/guide/en/elasticsearch/reference/6.2/disk-allocator.html.

The right solution depends on the context – for example a production environment vs a development environment.

Solution 1: free up disk space

Freeing up enough disk space so that more than 5% of the disk is free will solve this problem. Elasticsearch won’t automatically take itself out of read-only mode once enough disk is free though, you’ll have to do something like this to unlock the indices:

$ curl -XPUT -H "Content-Type: application/json" https://[YOUR_ELASTICSEARCH_ENDPOINT]:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'

Solution 2: change the flood stage watermark setting

Change the "cluster.routing.allocation.disk.watermark.flood_stage" setting to something else. It can either be set to a lower percentage or to an absolute value. Here’s an example of how to change the setting from the docs:

PUT _cluster/settings
{
  "transient": {
    "cluster.routing.allocation.disk.watermark.low": "100gb",
    "cluster.routing.allocation.disk.watermark.high": "50gb",
    "cluster.routing.allocation.disk.watermark.flood_stage": "10gb",
    "cluster.info.update.interval": "1m"
  }
}

Again, after doing this you’ll have to use the curl command above to unlock the indices, but after that they should not go into read-only mode again.

More Related Contents:

  1. Elasticsearch replication of other system data?
  2. Shards and replicas in Elasticsearch
  3. elasticsearch bool query combine must with OR
  4. ElasticSearch – Return Unique Values
  5. Calculating time between events
  6. Aggregation + sorting +pagination in elastic search
  7. Elasticsearch: Difference between “Term”, “Match Phrase”, and “Query String”
  8. Join query in ElasticSearch
  9. Max limit on the number of values I can specify in the ids filter or generally query clause?
  10. UTF8 encoding is longer than the max length 32766
  11. How to update multiple documents that match a query in elasticsearch
  12. Queries vs. Filters
  13. Change default mapping of string to “not analyzed” in Elasticsearch
  14. elasticsearch match vs term query
  15. elasticsearch – what to do with unassigned shards
  16. Elasticsearch vs Cassandra vs Elasticsearch with Cassandra
  17. How to change Elasticsearch max memory size
  18. Find documents with empty string value on elasticsearch
  19. How to get latest values for each group with an Elasticsearch query?
  20. How to create unique constraint in Elasticsearch database?
  21. Content-Type header [application/x-www-form-urlencoded] is not supported on Elasticsearch
  22. Elasticsearch, Failed to obtain node lock, is the following location writable
  23. elasticsearch – querying multiple indexes is possible?
  24. ElasticSearch – Get only matching nested objects with All Top level fields in search response
  25. How to use the official docker elasticsearch container?
  26. ElasticSearch: aggregation on _score field?
  27. How to parse json in logstash /grok from a text file line?
  28. Logstash: Merge two logs into one output document
  29. What is the ElasticSearch equivalent for an SQL subquery?
  30. How to create multiple indexes in logstash.conf file?

Leave a Comment