---
id: c84b68c5-a845-493a-b610-4253ab5881c2
---

# High time spent on Elasticsearch queries
---

This incident type refers to an elevated time spent on processing queries in an Elasticsearch cluster. It could be an indication of potential resource bottlenecks or a need to optimize the queries. This incident requires investigation and resolution to prevent further issues and ensure a smooth functioning Elasticsearch cluster.

### Parameters
```shell
# Environment Variables

export INDEX_NAME="PLACEHOLDER"

export ELASTICSEARCH_HOST="PLACEHOLDER"

export QUERY_TO_CACHE="PLACEHOLDER"

export NODE_CPU_CORES="PLACEHOLDER"

export PATH_TO_ELASTICSEARCH_CONFIG_FILE="PLACEHOLDER"

export NODE_MEMORY_SIZE_IN_GB="PLACEHOLDER"

export ELASTICSEARCH_PORT="PLACEHOLDER"

```

## Debug

### Check Elasticsearch cluster health
```shell
curl -X GET ${ELASTICSEARCH_HOST}:9200/_cluster/health?pretty
```

### Check Elasticsearch indices
```shell
curl -X GET ${ELASTICSEARCH_HOST}:9200/_cat/indices?v
```

### Check Elasticsearch node stats
```shell
curl -X GET ${ELASTICSEARCH_HOST}:9200/_nodes/stats?pretty
```

### Check Elasticsearch search and query latency
```shell
curl -X GET ${ELASTICSEARCH_HOST}:9200/_nodes/stats/indices/search?pretty
```

### Check Elasticsearch search and query throughput
```shell
curl -X GET ${ELASTICSEARCH_HOST}:9200/_nodes/stats/indices/search?pretty
```

### Check Elasticsearch query performance
```shell
curl -XGET ${ELASTICSEARCH_HOST}:9200/_cat/indices/${INDEX_NAME}?v&s=query_total
```

### Check Elasticsearch query time
```shell
curl -XGET ${ELASTICSEARCH_HOST}:9200/_cat/indices/${INDEX_NAME}?v&s=query_time
```

### Insufficient resources allocated to Elasticsearch cluster.
```shell
bash

#!/bin/bash



# Check CPU usage

CPU_USAGE=$(top -b -n1 | grep "Cpu(s)" | awk '{print $2}' | awk -F. '{print $1}')

if [ "$CPU_USAGE" -gt ${90} ]; then

  echo "CPU usage is high, which could indicate insufficient resources."

fi



# Check memory usage

TOTAL_MEMORY=$(free -m | awk '/^Mem:/{print $2}')

USED_MEMORY=$(free -m | awk '/^Mem:/{print $3}')

FREE_MEMORY=$(free -m | awk '/^Mem:/{print $4}')

if [ "$FREE_MEMORY" -lt ${2048} ]; then

  echo "Free memory is low, which could indicate insufficient resources."

fi



# Check disk usage

DISK_USAGE=$(df -h | awk '$NF=="/"{printf "%s", $5}')

if [ "${DISK_USAGE%\%}" -gt ${90} ]; then

  echo "Disk usage is high, which could indicate insufficient resources."

fi



# Check Elasticsearch logs for errors

ELASTICSEARCH_LOG="/var/log/elasticsearch/elasticsearch.log"

if [ -f "$ELASTICSEARCH_LOG" ]; then

  ERROR_COUNT=$(grep -i "error" "$ELASTICSEARCH_LOG" | wc -l)

  if [ "$ERROR_COUNT" -gt 0 ]; then

    echo "Elasticsearch logs contain errors, which could indicate insufficient resources."

  fi

else

  echo "Elasticsearch log file not found."

fi


```

## Repair

### Enable query caching for frequently executed queries
```shell


#!/bin/bash



# Set the Elasticsearch host and port

ELASTICSEARCH_HOST=${ELASTICSEARCH_HOST}:9200




# Enable caching for the query

curl -XPUT "http://${ELASTICSEARCH_HOST}:9200:9200/_all/_settings" -H 'Content-Type: application/json' -d'

{

  "index.queries.cache.enabled": true,

  "index.queries.cache.everything": true,

  "index.queries.cache.max_size": "10%",

  "index.queries.cache.expire": "5m",

  "index.queries.cache.recycler.schedule": "5m",

  "index.queries.cache.recycler.page_size": "1m"

}

'



# Refresh the index to apply changes

curl -XPOST "http://${ELASTICSEARCH_HOST}:9200/_refresh"



echo "Caching enabled for query: ${QUERY}"


```

### Fine-tune the Elasticsearch cluster configuration settings.
```shell
bash

#!/bin/bash



# Define variables

ELASTICSEARCH_CONFIG_FILE=${PATH_TO_ELASTICSEARCH_CONFIG_FILE}

NODE_MEMORY=${NODE_MEMORY_SIZE_IN_GB}

NODE_CPU=${NODE_CPU_CORES}



# Increase heap size

sed -i 's/-Xms[[:digit:]]\+g/-Xms'"$NODE_MEMORY"'g/g' $ELASTICSEARCH_CONFIG_FILE

sed -i 's/-Xmx[[:digit:]]\+g/-Xmx'"$NODE_MEMORY"'g/g' $ELASTICSEARCH_CONFIG_FILE



# Set CPU cores

sed -i 's/-Xss[[:digit:]]\+k/-Xss'"$(($NODE_MEMORY * 1024))"'k/g' $ELASTICSEARCH_CONFIG_FILE

sed -i 's/-XX:ParallelGCThreads=[[:digit:]]\+/-XX:ParallelGCThreads='"$NODE_CPU"'/g' $ELASTICSEARCH_CONFIG_FILE



# Restart Elasticsearch service

systemctl restart elasticsearch.service


```


This incident type refers to an elevated time spent on processing queries in an Elasticsearch cluster. It could be an indication of potential resource bottlenecks or a need to optimize the queries. This incident requires investigation and resolution to prevent further issues and ensure a smooth functioning Elasticsearch cluster.


This incident type refers to a situation where the Tomcat server's JDBC connection pool usage surpasses its configured limits. This can lead to a shortage of available connections for incoming requests, resulting in slow response times or even server crashes. It is important to monitor and manage the JDBC connection pool usage to prevent such incidents from occurring.


Tomcat High JDBC Connection Pool Usage Incident.

This incident type refers to an increase in the number of errors per second on a Tomcat server, which could indicate an issue with the server itself, the host, a deployed application, or an application servlet. This could include errors generated when the Tomcat server runs out of memory, can't find a requested file or servlet, or is unable to serve a JSP due to syntax errors in the servlet codebase. This incident type requires immediate attention to diagnose and address the underlying issue.


Increase of the errors/second rate for Tomcat server

This incident type indicates that there is a high latency issue in the execution of a Spark job. Spark is a distributed computing framework that is used for processing large datasets. High latency in this context means that the time taken to execute the Spark job is significantly longer than expected or normal. This can result in delays in processing data and can impact the performance of the application or system that is utilizing Spark.


High Latency Incident for Spark Job Execution.

This incident type refers to a scenario where there is a high rate of statement timeouts in a Postgresql database instance. This can lead to degraded performance and potentially impact the availability of the database. It is important to quickly identify and address the underlying cause of the timeouts to ensure the stability of the system.


Postgresql high rate statement timeout incident.

This incident type refers to a situation where the MySQL database is experiencing a high level of threads running, which could indicate a performance issue. This may cause issues with the application that relies on the database, such as slow response times or even downtime. Resolving this incident typically involves investigating the root cause of the high thread count and optimizing the database configuration or query performance to reduce the load on the database.


MySQL high threads running incident.

```shell
# Environment Variables

export INDEX_NAME="PLACEHOLDER"

export ELASTICSEARCH_HOST="PLACEHOLDER"

export QUERY_TO_CACHE="PLACEHOLDER"

export NODE_CPU_CORES="PLACEHOLDER"

export PATH_TO_ELASTICSEARCH_CONFIG_FILE="PLACEHOLDER"

export NODE_MEMORY_SIZE_IN_GB="PLACEHOLDER"

export ELASTICSEARCH_PORT="PLACEHOLDER"

```


### Check Elasticsearch cluster health

```shell
curl -X GET ${ELASTICSEARCH_HOST}:9200/_cluster/health?pretty
```

### Check Elasticsearch indices

```shell
curl -X GET ${ELASTICSEARCH_HOST}:9200/_cat/indices?v
```

### Check Elasticsearch node stats

```shell
curl -X GET ${ELASTICSEARCH_HOST}:9200/_nodes/stats?pretty
```

### Check Elasticsearch search and query latency

```shell
curl -X GET ${ELASTICSEARCH_HOST}:9200/_nodes/stats/indices/search?pretty
```

### Check Elasticsearch search and query throughput

```shell
curl -X GET ${ELASTICSEARCH_HOST}:9200/_nodes/stats/indices/search?pretty
```

### Check Elasticsearch query performance

```shell
curl -XGET ${ELASTICSEARCH_HOST}:9200/_cat/indices/${INDEX_NAME}?v&s=query_total
```

### Check Elasticsearch query time

```shell
curl -XGET ${ELASTICSEARCH_HOST}:9200/_cat/indices/${INDEX_NAME}?v&s=query_time
```

### Insufficient resources allocated to Elasticsearch cluster.

```shell
bash

#!/bin/bash



# Check CPU usage

CPU_USAGE=$(top -b -n1 | grep "Cpu(s)" | awk '{print $2}' | awk -F. '{print $1}')

if [ "$CPU_USAGE" -gt ${90} ]; then

  echo "CPU usage is high, which could indicate insufficient resources."

fi



# Check memory usage

TOTAL_MEMORY=$(free -m | awk '/^Mem:/{print $2}')

USED_MEMORY=$(free -m | awk '/^Mem:/{print $3}')

FREE_MEMORY=$(free -m | awk '/^Mem:/{print $4}')

if [ "$FREE_MEMORY" -lt ${2048} ]; then

  echo "Free memory is low, which could indicate insufficient resources."

fi



# Check disk usage

DISK_USAGE=$(df -h | awk '$NF=="/"{printf "%s", $5}')

if [ "${DISK_USAGE%\%}" -gt ${90} ]; then

  echo "Disk usage is high, which could indicate insufficient resources."

fi



# Check Elasticsearch logs for errors

ELASTICSEARCH_LOG="/var/log/elasticsearch/elasticsearch.log"

if [ -f "$ELASTICSEARCH_LOG" ]; then

  ERROR_COUNT=$(grep -i "error" "$ELASTICSEARCH_LOG" | wc -l)

  if [ "$ERROR_COUNT" -gt 0 ]; then

    echo "Elasticsearch logs contain errors, which could indicate insufficient resources."

  fi

else

  echo "Elasticsearch log file not found."

fi


```


### Enable query caching for frequently executed queries

```shell


#!/bin/bash



# Set the Elasticsearch host and port

ELASTICSEARCH_HOST=${ELASTICSEARCH_HOST}:9200




# Enable caching for the query

curl -XPUT "http://${ELASTICSEARCH_HOST}:9200:9200/_all/_settings" -H 'Content-Type: application/json' -d'

{

  "index.queries.cache.enabled": true,

  "index.queries.cache.everything": true,

  "index.queries.cache.max_size": "10%",

  "index.queries.cache.expire": "5m",

  "index.queries.cache.recycler.schedule": "5m",

  "index.queries.cache.recycler.page_size": "1m"

}

'



# Refresh the index to apply changes

curl -XPOST "http://${ELASTICSEARCH_HOST}:9200/_refresh"



echo "Caching enabled for query: ${QUERY}"


```

### Fine-tune the Elasticsearch cluster configuration settings.

```shell
bash

#!/bin/bash



# Define variables

ELASTICSEARCH_CONFIG_FILE=${PATH_TO_ELASTICSEARCH_CONFIG_FILE}

NODE_MEMORY=${NODE_MEMORY_SIZE_IN_GB}

NODE_CPU=${NODE_CPU_CORES}



# Increase heap size

sed -i 's/-Xms[[:digit:]]\+g/-Xms'"$NODE_MEMORY"'g/g' $ELASTICSEARCH_CONFIG_FILE

sed -i 's/-Xmx[[:digit:]]\+g/-Xmx'"$NODE_MEMORY"'g/g' $ELASTICSEARCH_CONFIG_FILE



# Set CPU cores

sed -i 's/-Xss[[:digit:]]\+k/-Xss'"$(($NODE_MEMORY * 1024))"'k/g' $ELASTICSEARCH_CONFIG_FILE

sed -i 's/-XX:ParallelGCThreads=[[:digit:]]\+/-XX:ParallelGCThreads='"$NODE_CPU"'/g' $ELASTICSEARCH_CONFIG_FILE



# Restart Elasticsearch service

systemctl restart elasticsearch.service


```


High time spent on Elasticsearch queries

Overview

Parameters

Debug

Check Elasticsearch cluster health

Check Elasticsearch indices

Check Elasticsearch node stats

Check Elasticsearch search and query latency

Check Elasticsearch search and query throughput

Check Elasticsearch query performance

Check Elasticsearch query time

Insufficient resources allocated to Elasticsearch cluster.

Repair

Enable query caching for frequently executed queries

Fine-tune the Elasticsearch cluster configuration settings.

Learn more

Related Runbooks

Tomcat High JDBC Connection Pool Usage Incident.

Increase of the errors/second rate for Tomcat server

High Latency Incident for Spark Job Execution.

Postgresql high rate statement timeout incident.

Support