Elasticsearch 索引只读403 read-only

Java日志报错如下

2022-03-30 18:17:02.750 ERROR 22991 --- [task-6] .a.i.SimpleAsyncUncaughtExceptionHandler : Unexpected exception occurred invoking async method: public void com.proinnova.system.log.service.impl.LogServiceImpl.addLog(com.proinnova.system.log.po.SysLogInfo,java.lang.Exception)

org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=cluster_block_exception, reason=index [os_exception_log] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]
        at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177) ~[elasticsearch-6.8.13.jar!/:6.8.13]
        at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2061) ~[elasticsearch-rest-high-level-client-6.8.13.jar!/:6.8.13]
        at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2031) ~[elasticsearch-rest-high-level-client-6.8.13.jar!/:6.8.13]
        at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1778) ~[elasticsearch-rest-high-level-client-6.8.13.jar!/:6.8.13]
        at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1735) ~[elasticsearch-rest-high-level-client-6.8.13.jar!/:6.8.13]
        at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1697) ~[elasticsearch-rest-high-level-client-6.8.13.jar!/:6.8.13]
        at org.elasticsearch.client.RestHighLevelClient.index(RestHighLevelClient.java:929) ~[elasticsearch-rest-high-level-client-6.8.13.jar!/:6.8.13]
        at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.index(ElasticsearchRestTemplate.java:703) ~[spring-data-elasticsearch-3.2.12.RELEASE.jar!/:3.2.12.RELEASE]
        at org.springframework.data.elasticsearch.repository.support.AbstractElasticsearchRepository.save(AbstractElasticsearchRepository.java:182) ~[spring-data-elasticsearch-3.2.12.RELEASE.jar!/:3.2.12.RELEASE]
        at sun.reflect.GeneratedMethodAccessor204.invoke(Unknown Source) ~[na:na]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_301]
        at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_301]
        at org.springframework.data.repository.core.support.RepositoryComposition$RepositoryFragments.invoke(RepositoryComposition.java:371) ~[spring-data-commons-2.2.12.RELEASE.jar!/:2.2.12.RELEASE]
        at org.springframework.data.repository.core.support.RepositoryComposition.invoke(RepositoryComposition.java:204) ~[spring-data-commons-2.2.12.RELEASE.jar!/:2.2.12.RELEASE]
        at org.springframework.data.repository.core.support.RepositoryFactorySupport$ImplementationMethodExecutionInterceptor.invoke(RepositoryFactorySupport.java:658) ~[spring-data-commons-2.2.12.RELEASE.jar!/:2.2.12.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.2.12.RELEASE.jar!/:5.2.12.RELEASE]
        at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.doInvoke(RepositoryFactorySupport.java:622) ~[spring-data-commons-2.2.12.RELEASE.jar!/:2.2.12.RELEASE]
        at org.springframework.data.repository.core.support.RepositoryFactorySupport$QueryExecutorMethodInterceptor.invoke(RepositoryFactorySupport.java:606) ~[spring-data-commons-2.2.12.RELEASE.jar!/:2.2.12.RELEASE]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.2.12.RELEASE.jar!/:5.2.12.RELEASE]

主要报错内容:

org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=cluster_block_exception, reason=index [os_exception_log] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];]

查询得出是ES索引只读了，接下来需要分析为什么es索引会变成只读状态

ES 索引只读原因

• 内存不足
  JVMMemoryPressure 超过92%并持续30分钟时，ES触发保护机制，并且阻止写入操作，以防止集群达到红色状态，启用写保护后，写入操作将失败，并且抛出 ClusterBlockException ，无法创建新索引，并且抛出 IndexCreateBlockException ,当五分钟内恢复不到88%以下时，将禁用写保护。

• 磁盘空间不足
  es的默认磁盘水位警戒线是85%，一旦磁盘使用率超过85%，es不会再为该节点分配分片，es还有一个磁盘水位警戒线是90%，超过后，将尝试将分片重定位到其他节点。

问题排查

接下来我们查看一下ES索引状态，是否为只读

#首先获取索引列表

[root@abcdocker ~]# curl -XGET 'localhost:9200/_cat/indices?v=true&s=index' 
health status index              uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   job_content_sunsea 5-wLLa6YQ_-IEqSNrYfY0w   1   1        611            0    232.7kb        232.7kb
yellow open   job_current_sunsea aXz5NNEFRpuPFGMfKdkzbw   1   1         10            0    107.7kb        107.7kb
yellow open   os_exception_log   zH8f77gLTemzBnUBSFPt7w   5   1        191            0      504kb          504kb
yellow open   space_log          ePVDkDhYRaO1KvPibBKbRQ   5   1      32667            0      2.6mb          2.6mb

查看里面的索引，检查read_only_allow_delete变量是否为true，

[root@abcdocker ~]# curl -s -XGET 'localhost:9200/space_log/_settings' |grep read_only_allow_delete
{"space_log":{"settings":{"index":{"refresh_interval":"1s","number_of_shards":"5","blocks":{"read_only_allow_delete":"true"},"provided_name":"space_log","creation_date":"1639476549574","store":{"type":"fs"},"number_of_replicas":"1","uuid":"ePVDkDhYRaO1KvPibBKbRQ","version":{"created":"7020099"}}}}}

[root@abcdocker ~]# curl -s -XGET 'localhost:9200/os_exception_log/_settings' |grep read_only_allow_delete
{"os_exception_log":{"settings":{"index":{"refresh_interval":"1s","number_of_shards":"5","blocks":{"read_only_allow_delete":"true"},"provided_name":"os_exception_log","creation_date":"1639476469103","store":{"type":"fs"},"number_of_replicas":"1","uuid":"zH8f77gLTemzBnUBSFPt7w","version":{"created":"7020099"}}}}}

[root@abcdocker ~]# curl -s -XGET 'localhost:9200/job_current_sunsea/_settings' |grep read_only_allow_delete
{"job_current_sunsea":{"settings":{"index":{"number_of_shards":"1","blocks":{"read_only_allow_delete":"true"},"provided_name":"job_current_sunsea","creation_date":"1640141880762","number_of_replicas":"1","uuid":"aXz5NNEFRpuPFGMfKdkzbw","version":{"created":"7020099"}}}}}

[root@abcdocker ~]# curl -s -XGET 'localhost:9200/job_current_sunsea/_settings' |grep read_only_allow_delete
{"job_current_sunsea":{"settings":{"index":{"number_of_shards":"1","blocks":{"read_only_allow_delete":"true"},"provided_name":"job_current_sunsea","creation_date":"1640141880762","number_of_replicas":"1","uuid":"aXz5NNEFRpuPFGMfKdkzbw","version":{"created":"7020099"}}}}}

发现es上所有的索引都变成read_only_allow_delete=true

解决方法

• 第一种:使用curl命令，将所有的只读状态修改为false

curl -XPUT -H "Content-Type: application/json" \
   http://localhost:9200/_all/_settings \
     -d '{"index.blocks.read_only_allow_delete": false}'

• 第二种: kibana控制台的运维工具里面执行

PUT /_all/_settings
{"index.blocks.read_only_allow_delete": false}

检查

重启调用的java服务

参考文档:
https://cloud.tencent.com/document/product/845/56276