---
id: b11196e5-23f5-4926-942c-32e57c6a1c44
---

# MySQL high threads running incident.
---

This incident type refers to a situation where the MySQL database is experiencing a high level of threads running, which could indicate a performance issue. This may cause issues with the application that relies on the database, such as slow response times or even downtime. Resolving this incident typically involves investigating the root cause of the high thread count and optimizing the database configuration or query performance to reduce the load on the database.

### Parameters
```shell
# Environment Variables

export HOSTNAME="PLACEHOLDER"

export USERNAME="PLACEHOLDER"

export PASSWORD="PLACEHOLDER"

export MYSQL_DATA_DIRECTORY="PLACEHOLDER"

export MYSQL_ERROR_LOG="PLACEHOLDER"

export MYSQL_SLOW_QUERY_LOG="PLACEHOLDER"

```

## Debug

### Check MySQL processlist to see the running queries and threads
```shell
mysql -h ${HOSTNAME} -u ${USERNAME} -p${PASSWORD} -e "SHOW FULL PROCESSLIST"
```

### Check the MySQL connection limit
```shell
mysql -h ${HOSTNAME} -u ${USERNAME} -p${PASSWORD} -e "SHOW VARIABLES LIKE 'max_connections'"
```

### Check the number of threads currently running on the server
```shell
ps -ef | grep mysql | grep -v grep | wc -l
```

### Check the system load average
```shell
uptime
```

### Check the available memory and swap space
```shell
free -m
```

### Check the disk usage of MySQL data directory
```shell
du -sh ${MYSQL_DATA_DIRECTORY}
```

### Check the MySQL error log for any relevant errors or warnings
```shell
tail -n 100 ${MYSQL_ERROR_LOG}
```

### Check the slow query log for queries that might be contributing to high thread count
```shell
tail -n 100 ${MYSQL_SLOW_QUERY_LOG}
```

### Check the dmesg output for any kernel-level errors or warnings
```shell
dmesg | tail
```

### A long-running or poorly optimized query, causing threads to stay active for longer periods of time.
```shell


#!/bin/bash


# Connect to MySQL and run diagnostic query

echo "Checking for long-running or poorly optimized queries..."

mysql -u $MYSQL_USER -p$MYSQL_PASSWORD -h $MYSQL_HOST -P $MYSQL_PORT -D $MYSQL_DATABASE -e "SELECT * FROM INFORMATION_SCHEMA.PROCESSLIST WHERE TIME > 60 OR Command = 'Query' ORDER BY TIME DESC;"



echo "Done."


```

## Repair

### Identify the queries causing the high thread count and optimize them.
```shell
bash

#!/bin/bash

 INDEX_NAME="PLACEHOLDER"

 TABLE_NAME="PLACEHOLDER"

 COLUMN_NAME="PLACEHOLDER"

# Connect to the MySQL server using the ${USERNAME} and ${PASSWORD}

mysql -u ${USERNAME} -p${PASSWORD}



# Identify the queries causing the high thread count

SHOW PROCESSLIST;



# Analyze the slow performing queries

EXPLAIN ${SLOW_QUERY};



# Optimize the queries by adding missing indexes or rewriting the query

ALTER TABLE ${TABLE_NAME} ADD INDEX ${INDEX_NAME} (${COLUMN_NAME});


```

### Restart the MySQL server to clear all running threads.
```shell


#!/bin/bash



# Stop the MySQL service

sudo systemctl stop mysql



# Wait for the service to stop

while [ "$(systemctl is-active mysql)" != "inactive" ]; do

    sleep 1

done



# Start the MySQL service

sudo systemctl start mysql


```

This incident type refers to a situation where the MySQL database is experiencing a high level of threads running, which could indicate a performance issue. This may cause issues with the application that relies on the database, such as slow response times or even downtime. Resolving this incident typically involves investigating the root cause of the high thread count and optimizing the database configuration or query performance to reduce the load on the database.


When too many requests are sent to a Tomcat server, the thread pools that handle these requests can run out of available threads. This means that incoming requests will not be processed, leading to service degradation or even a complete outage. This incident type is critical for web applications that rely on Tomcat as their application server.


Tomcat thread pool exhaustion.

This incident type is related to the Tomcat server, which is a popular open-source Java application server. It occurs when the available threads in the connection pool are running low, which means that there are not enough resources to serve incoming requests. This can cause delays or even failures in the application's performance, impacting the end-user experience. It is important to monitor and address this issue promptly to ensure that the application runs smoothly and efficiently.


Tomcat Low Available Threads in Connection Pool

In software engineering, a large number of suspended threads in Tomcat is an incident type that occurs when a high number of threads are created but are blocked from executing. This can happen due to a variety of reasons, such as slow database queries, network latency, or resource contention. When this occurs, the system may become unresponsive or slow, causing a degradation in performance or even a complete system failure. It is important to investigate and resolve this issue as quickly as possible to ensure the system can continue to function properly.


Tomcat Large Number of Suspended Threads Incident.

This incident type refers to a situation where the Tomcat server's JDBC connection pool usage surpasses its configured limits. This can lead to a shortage of available connections for incoming requests, resulting in slow response times or even server crashes. It is important to monitor and manage the JDBC connection pool usage to prevent such incidents from occurring.


Tomcat High JDBC Connection Pool Usage Incident.

This incident type is triggered when the percentage of busy threads on a Tomcat host exceeds a certain threshold. This can indicate that the host is experiencing high traffic or that there is a problem with the configuration of the Tomcat server. The incident is resolved when the issue is addressed and the percentage of busy threads returns to a normal level.


High % of Busy Threads on Tomcat Host (TEST)

```shell
# Environment Variables

export HOSTNAME="PLACEHOLDER"

export USERNAME="PLACEHOLDER"

export PASSWORD="PLACEHOLDER"

export MYSQL_DATA_DIRECTORY="PLACEHOLDER"

export MYSQL_ERROR_LOG="PLACEHOLDER"

export MYSQL_SLOW_QUERY_LOG="PLACEHOLDER"

```


### Check MySQL processlist to see the running queries and threads

```shell
mysql -h ${HOSTNAME} -u ${USERNAME} -p${PASSWORD} -e "SHOW FULL PROCESSLIST"
```

### Check the MySQL connection limit

```shell
mysql -h ${HOSTNAME} -u ${USERNAME} -p${PASSWORD} -e "SHOW VARIABLES LIKE 'max_connections'"
```

### Check the number of threads currently running on the server

```shell
ps -ef | grep mysql | grep -v grep | wc -l
```

### Check the system load average

```shell
uptime
```

### Check the available memory and swap space

```shell
free -m
```

### Check the disk usage of MySQL data directory

```shell
du -sh ${MYSQL_DATA_DIRECTORY}
```

### Check the MySQL error log for any relevant errors or warnings

```shell
tail -n 100 ${MYSQL_ERROR_LOG}
```

### Check the slow query log for queries that might be contributing to high thread count

```shell
tail -n 100 ${MYSQL_SLOW_QUERY_LOG}
```

### Check the dmesg output for any kernel-level errors or warnings

```shell
dmesg | tail
```

### A long-running or poorly optimized query, causing threads to stay active for longer periods of time.

```shell


#!/bin/bash


# Connect to MySQL and run diagnostic query

echo "Checking for long-running or poorly optimized queries..."

mysql -u $MYSQL_USER -p$MYSQL_PASSWORD -h $MYSQL_HOST -P $MYSQL_PORT -D $MYSQL_DATABASE -e "SELECT * FROM INFORMATION_SCHEMA.PROCESSLIST WHERE TIME > 60 OR Command = 'Query' ORDER BY TIME DESC;"



echo "Done."


```


### Identify the queries causing the high thread count and optimize them.

```shell
bash

#!/bin/bash

 INDEX_NAME="PLACEHOLDER"

 TABLE_NAME="PLACEHOLDER"

 COLUMN_NAME="PLACEHOLDER"

# Connect to the MySQL server using the ${USERNAME} and ${PASSWORD}

mysql -u ${USERNAME} -p${PASSWORD}



# Identify the queries causing the high thread count

SHOW PROCESSLIST;



# Analyze the slow performing queries

EXPLAIN ${SLOW_QUERY};



# Optimize the queries by adding missing indexes or rewriting the query

ALTER TABLE ${TABLE_NAME} ADD INDEX ${INDEX_NAME} (${COLUMN_NAME});


```

### Restart the MySQL server to clear all running threads.

```shell


#!/bin/bash



# Stop the MySQL service

sudo systemctl stop mysql



# Wait for the service to stop

while [ "$(systemctl is-active mysql)" != "inactive" ]; do

    sleep 1

done



# Start the MySQL service

sudo systemctl start mysql


```


MySQL high threads running incident.

Overview

Parameters

Debug

Check MySQL processlist to see the running queries and threads

Check the MySQL connection limit

Check the number of threads currently running on the server

Check the system load average

Check the available memory and swap space

Check the disk usage of MySQL data directory

Check the MySQL error log for any relevant errors or warnings

Check the slow query log for queries that might be contributing to high thread count

Check the dmesg output for any kernel-level errors or warnings

A long-running or poorly optimized query, causing threads to stay active for longer periods of time.

Repair

Identify the queries causing the high thread count and optimize them.

Restart the MySQL server to clear all running threads.

Learn more

Related Runbooks

Tomcat thread pool exhaustion.

Tomcat Low Available Threads in Connection Pool

Tomcat Large Number of Suspended Threads Incident.

Tomcat High JDBC Connection Pool Usage Incident.

Support