---
id: 52bdab8c-35bf-4751-a0d5-a6355da59a08
---
# High Unsuccessful Requests Rate in ElasticSearch
---

This incident type typically occurs when the unsuccessful requests rate in ElasticSearch is higher than the expected threshold. This may indicate that there are issues with the ElasticSearch cluster or that it is not able to handle the requests in a timely manner. To prevent further issues, it is important to investigate the root cause and address any underlying issues that may be causing the high unsuccessful requests rate. This will help ensure that the ElasticSearch cluster is able to handle requests efficiently and effectively.

### Parameters
```shell
# Environment Variables

export ELASTICSEARCH_HOST="PLACEHOLDER"

export INDEX_NAME="PLACEHOLDER"

export THRESHOLD="PLACEHOLDER"

export PATH_TO_ELASTICSEARCH_LOGS="PLACEHOLDER"

export SHARD_NUMBER="PLACEHOLDER"

export ELASTICSEARCH_PATH="PLACEHOLDER"

export ELASTICSEARCH_CONFIG="PLACEHOLDER"

export CACHE_TIME="PLACEHOLDER"
```

## Debug

### Check ElasticSearch cluster health
```shell
curl -XGET '${ELASTICSEARCH_HOST}:9200/_cluster/health?pretty'
```

### Check ElasticSearch cluster state
```shell
curl -XGET '${ELASTICSEARCH_HOST}:9200/_cluster/state?pretty'
```

### Check ElasticSearch cluster stats
```shell
curl -XGET '${ELASTICSEARCH_HOST}:9200/_nodes/stats?pretty'
```

### Check ElasticSearch index health
```shell
curl -XGET '${ELASTICSEARCH_HOST}:9200/_cat/indices?v'
```

### Check ElasticSearch index status
```shell
curl -XGET '${ELASTICSEARCH_HOST}:9200/_cat/shards?v'
```

### Check ElasticSearch index settings
```shell
curl -XGET '${ELASTICSEARCH_HOST}:9200/${INDEX_NAME}/_settings?pretty'
```

### Check ElasticSearch index mappings
```shell
curl -XGET '${ELASTICSEARCH_HOST}:9200/${INDEX_NAME}/_mapping?pretty'
```

### Check ElasticSearch index stats
```shell
curl -XGET '${ELASTICSEARCH_HOST}:9200/${INDEX_NAME}/_stats?pretty'
```

### ElasticSearch cluster is overloaded and unable to handle the incoming requests, leading to high unsuccessful request rates.
```shell


#!/bin/bash



# Set Elasticsearch URL and threshold

THRESHOLD=${THRESHOLD}



# Get the high unsuccessful requests rate

RATE=$(curl -s -XGET "${ELASTICSEARCH_HOST}:9200/_nodes/stats/thread_pool" | jq '.nodes[].thread_pool.search.rejected' | awk '{s+=$1} END {print s}')



# Check if the rate is above the threshold

if [ "${RATE}" -gt "${THRESHOLD}" ]; then

  echo "High unsuccessful requests rate detected. Current rate: ${RATE}"

  # Add any additional commands to help diagnose and resolve the issue

else

  echo "Unsuccessful requests rate is within expected range. Current rate: ${RATE}"

fi


```

### The ElasticSearch cluster may be experiencing issues with memory or disk space, causing requests to fail and resulting in high unsuccessful request rates.
```shell


#!/bin/bash



# Check memory usage

MEMORY_USAGE=$(free -m | awk 'NR==2{printf "%.2f%%\n", $3*100/$2 }')

echo "Memory usage: $MEMORY_USAGE"



# Check disk usage

DISK_USAGE=$(df -h | awk '$NF=="/"{printf "%s\n", $5}')

echo "Disk usage: $DISK_USAGE"



# Check ElasticSearch logs for errors

ELASTICSEARCH_LOG_PATH=${PATH_TO_ELASTICSEARCH_LOGS}

ERRORS=$(grep -i "error" $ELASTICSEARCH_LOG_PATH)

if [ -z "$ERRORS" ]

then

  echo "No errors in ElasticSearch logs"

else

  echo "Errors found in ElasticSearch logs: $ERRORS"

fi


```

### ElasticSearch indices or shards may have become corrupted or damaged, leading to frequent unsuccessful requests and high unsuccessful request rates.
```shell


#!/bin/bash



# Set variables

index=${INDEX_NAME}

shard=${SHARD_NUMBER}


# Check if index exists

index_exists=$(curl -s -o /dev/null -w "%{http_code}" -XHEAD http://${ELASTICSEARCH_HOST}:9200/$index)



if [ $index_exists -eq 200 ]; then

  echo "Index $index exists."

else

  echo "Index $index does not exist."

  exit 1

fi

# Check if shard is corrupted

shard_info=$(curl -s -XGET http://${ELASTICSEARCH_HOST}:9200/$index/_shard/$shard/_stats)

if echo $shard_info | grep -q "\"corrupted\":true"; then

  echo "Shard $shard in index $index is corrupted."

  exit 1

else

  echo "Shard $shard in index $index is not corrupted."

fi


```

## Repair

### Optimize the ElasticSearch cluster to improve its performance and reduce the number of unsuccessful requests.
```shell
#!/bin/bash

# Define variables

${ELASTICSEARCH_PATH}="/usr/share/elasticsearch"

${ELASTICSEARCH_CONFIG}="/etc/elasticsearch/elasticsearch.yml"

# Stop the ElasticSearch service

sudo systemctl stop elasticsearch

# Increase the memory allocation for ElasticSearch

sudo sed -i 's/-Xms1g/-Xms4g/g' ${ELASTICSEARCH_CONFIG}

sudo sed -i 's/-Xmx1g/-Xmx4g/g' ${ELASTICSEARCH_CONFIG}

# Optimize the ElasticSearch configuration

sudo sed -i 's/#bootstrap.memory_lock: true/bootstrap.memory_lock: true/g' ${ELASTICSEARCH_CONFIG}

sudo sed -i 's/#thread_pool.search.size: 5/thread_pool.search.size: 10/g' ${ELASTICSEARCH_CONFIG}

sudo sed -i 's/#thread_pool.search.queue_size: 1000/thread_pool.search.queue_size: 5000/g' ${ELASTICSEARCH_CONFIG}

# Start the ElasticSearch service

sudo systemctl start elasticsearch

sudo bash optimize_elasticsearch.sh

```

### Implement caching mechanisms to reduce the number of requests being made to ElasticSearch. This may involve
```shell

#!/bin/bash

# Set variables

CACHE_TIME=${CACHE_TIME}

# Configure caching

curl -XPUT ${ELASTICSEARCH_HOST}:9200 -d '{

    "index" : {

        "cache" : {

            "query" : {

                "enable" : true,

                "cache_count" : 10000,

                "cache_size" : "2gb",

                "expire" : "'$CACHE_TIME'"

            }

        }

    }

}'


```


This incident type typically occurs when the unsuccessful requests rate in ElasticSearch is higher than the expected threshold. This may indicate that there are issues with the ElasticSearch cluster or that it is not able to handle the requests in a timely manner. To prevent further issues, it is important to investigate the root cause and address any underlying issues that may be causing the high unsuccessful requests rate. This will help ensure that the ElasticSearch cluster is able to handle requests efficiently and effectively.


This incident type refers to an increase in the number of errors per second on a Tomcat server, which could indicate an issue with the server itself, the host, a deployed application, or an application servlet. This could include errors generated when the Tomcat server runs out of memory, can't find a requested file or servlet, or is unable to serve a JSP due to syntax errors in the servlet codebase. This incident type requires immediate attention to diagnose and address the underlying issue.


Increase of the errors/second rate for Tomcat server

This incident type refers to a situation where the number of 5xx errors on NGINX upstreams is higher than usual. It is an indication that there could be an issue with the NGINX server or the upstream application server. This type of incident requires immediate attention as it can impact the availability and performance of the application.


Higher than usual 5xx Errors for NGINX

The "High error rate on NGINX" incident type refers to a situation where the error rate on the NGINX server is above 1% for the last 5 minutes. This can result in degraded performance or downtime of the affected service, impacting user experience and potentially leading to lost revenue. The incident requires immediate attention and resolution to minimize the impact on users and prevent further damage.


High error rate on NGINX incident

This incident type refers to a sudden and abnormal drop in the SELECT query rate on a MySQL server. This can potentially cause disruption in the service and affect the user experience. The cause of the drop could be due to a variety of factors such as database issues, server overload, or network problems. It requires immediate attention and investigation to identify the root cause and resolve the issue to prevent further impact.


Unusual drop in SELECT query rate on MySQL server

The Istio High Request Latency incident type refers to an issue where the average execution time for requests in Istio is longer than the expected threshold of 100ms. This may result in slow or unresponsive services, negatively impacting the user experience. It may require investigation and troubleshooting to identify the underlying cause and resolve the issue.


Istio High Request Latency

```shell
# Environment Variables

export ELASTICSEARCH_HOST="PLACEHOLDER"

export INDEX_NAME="PLACEHOLDER"

export THRESHOLD="PLACEHOLDER"

export PATH_TO_ELASTICSEARCH_LOGS="PLACEHOLDER"

export SHARD_NUMBER="PLACEHOLDER"

export ELASTICSEARCH_PATH="PLACEHOLDER"

export ELASTICSEARCH_CONFIG="PLACEHOLDER"

export CACHE_TIME="PLACEHOLDER"
```


### Check ElasticSearch cluster health

```shell
curl -XGET '${ELASTICSEARCH_HOST}:9200/_cluster/health?pretty'
```

### Check ElasticSearch cluster state

```shell
curl -XGET '${ELASTICSEARCH_HOST}:9200/_cluster/state?pretty'
```

### Check ElasticSearch cluster stats

```shell
curl -XGET '${ELASTICSEARCH_HOST}:9200/_nodes/stats?pretty'
```

### Check ElasticSearch index health

```shell
curl -XGET '${ELASTICSEARCH_HOST}:9200/_cat/indices?v'
```

### Check ElasticSearch index status

```shell
curl -XGET '${ELASTICSEARCH_HOST}:9200/_cat/shards?v'
```

### Check ElasticSearch index settings

```shell
curl -XGET '${ELASTICSEARCH_HOST}:9200/${INDEX_NAME}/_settings?pretty'
```

### Check ElasticSearch index mappings

```shell
curl -XGET '${ELASTICSEARCH_HOST}:9200/${INDEX_NAME}/_mapping?pretty'
```

### Check ElasticSearch index stats

```shell
curl -XGET '${ELASTICSEARCH_HOST}:9200/${INDEX_NAME}/_stats?pretty'
```

### ElasticSearch cluster is overloaded and unable to handle the incoming requests, leading to high unsuccessful request rates.

```shell


#!/bin/bash



# Set Elasticsearch URL and threshold

THRESHOLD=${THRESHOLD}



# Get the high unsuccessful requests rate

RATE=$(curl -s -XGET "${ELASTICSEARCH_HOST}:9200/_nodes/stats/thread_pool" | jq '.nodes[].thread_pool.search.rejected' | awk '{s+=$1} END {print s}')



# Check if the rate is above the threshold

if [ "${RATE}" -gt "${THRESHOLD}" ]; then

  echo "High unsuccessful requests rate detected. Current rate: ${RATE}"

  # Add any additional commands to help diagnose and resolve the issue

else

  echo "Unsuccessful requests rate is within expected range. Current rate: ${RATE}"

fi


```

### The ElasticSearch cluster may be experiencing issues with memory or disk space, causing requests to fail and resulting in high unsuccessful request rates.

```shell


#!/bin/bash



# Check memory usage

MEMORY_USAGE=$(free -m | awk 'NR==2{printf "%.2f%%\n", $3*100/$2 }')

echo "Memory usage: $MEMORY_USAGE"



# Check disk usage

DISK_USAGE=$(df -h | awk '$NF=="/"{printf "%s\n", $5}')

echo "Disk usage: $DISK_USAGE"



# Check ElasticSearch logs for errors

ELASTICSEARCH_LOG_PATH=${PATH_TO_ELASTICSEARCH_LOGS}

ERRORS=$(grep -i "error" $ELASTICSEARCH_LOG_PATH)

if [ -z "$ERRORS" ]

then

  echo "No errors in ElasticSearch logs"

else

  echo "Errors found in ElasticSearch logs: $ERRORS"

fi


```

### ElasticSearch indices or shards may have become corrupted or damaged, leading to frequent unsuccessful requests and high unsuccessful request rates.

```shell


#!/bin/bash



# Set variables

index=${INDEX_NAME}

shard=${SHARD_NUMBER}


# Check if index exists

index_exists=$(curl -s -o /dev/null -w "%{http_code}" -XHEAD http://${ELASTICSEARCH_HOST}:9200/$index)



if [ $index_exists -eq 200 ]; then

  echo "Index $index exists."

else

  echo "Index $index does not exist."

  exit 1

fi

# Check if shard is corrupted

shard_info=$(curl -s -XGET http://${ELASTICSEARCH_HOST}:9200/$index/_shard/$shard/_stats)

if echo $shard_info | grep -q "\"corrupted\":true"; then

  echo "Shard $shard in index $index is corrupted."

  exit 1

else

  echo "Shard $shard in index $index is not corrupted."

fi


```


### Optimize the ElasticSearch cluster to improve its performance and reduce the number of unsuccessful requests.

```shell
#!/bin/bash

# Define variables

${ELASTICSEARCH_PATH}="/usr/share/elasticsearch"

${ELASTICSEARCH_CONFIG}="/etc/elasticsearch/elasticsearch.yml"

# Stop the ElasticSearch service

sudo systemctl stop elasticsearch

# Increase the memory allocation for ElasticSearch

sudo sed -i 's/-Xms1g/-Xms4g/g' ${ELASTICSEARCH_CONFIG}

sudo sed -i 's/-Xmx1g/-Xmx4g/g' ${ELASTICSEARCH_CONFIG}

# Optimize the ElasticSearch configuration

sudo sed -i 's/#bootstrap.memory_lock: true/bootstrap.memory_lock: true/g' ${ELASTICSEARCH_CONFIG}

sudo sed -i 's/#thread_pool.search.size: 5/thread_pool.search.size: 10/g' ${ELASTICSEARCH_CONFIG}

sudo sed -i 's/#thread_pool.search.queue_size: 1000/thread_pool.search.queue_size: 5000/g' ${ELASTICSEARCH_CONFIG}

# Start the ElasticSearch service

sudo systemctl start elasticsearch

sudo bash optimize_elasticsearch.sh

```

### Implement caching mechanisms to reduce the number of requests being made to ElasticSearch. This may involve

```shell

#!/bin/bash

# Set variables

CACHE_TIME=${CACHE_TIME}

# Configure caching

curl -XPUT ${ELASTICSEARCH_HOST}:9200 -d '{

    "index" : {

        "cache" : {

            "query" : {

                "enable" : true,

                "cache_count" : 10000,

                "cache_size" : "2gb",

                "expire" : "'$CACHE_TIME'"

            }

        }

    }

}'


```


High Unsuccessful Requests Rate in ElasticSearch

Overview

Parameters

Debug

Check ElasticSearch cluster health

Check ElasticSearch cluster state

Check ElasticSearch cluster stats

Check ElasticSearch index health

Check ElasticSearch index status

Check ElasticSearch index settings

Check ElasticSearch index mappings

Check ElasticSearch index stats

ElasticSearch cluster is overloaded and unable to handle the incoming requests, leading to high unsuccessful request rates.

The ElasticSearch cluster may be experiencing issues with memory or disk space, causing requests to fail and resulting in high unsuccessful request rates.

ElasticSearch indices or shards may have become corrupted or damaged, leading to frequent unsuccessful requests and high unsuccessful request rates.

Repair

Optimize the ElasticSearch cluster to improve its performance and reduce the number of unsuccessful requests.

Implement caching mechanisms to reduce the number of requests being made to ElasticSearch. This may involve

Learn more

Related Runbooks

Increase of the errors/second rate for Tomcat server

Higher than usual 5xx Errors for NGINX

High error rate on NGINX incident

Unusual drop in SELECT query rate on MySQL server

Support