---
id: cecfa00e-f266-4b0b-a478-d615d67facd8
---

# Tomcat thread pool exhaustion.
---

When too many requests are sent to a Tomcat server, the thread pools that handle these requests can run out of available threads. This means that incoming requests will not be processed, leading to service degradation or even a complete outage. This incident type is critical for web applications that rely on Tomcat as their application server.

### Parameters
```shell
export TOMCAT_HOME="PLACEHOLDER"

export TOMCAT_PID="PLACEHOLDER"

export BLOCKED_THREAD_ID="PLACEHOLDER"

export PATH_TO_TOMCAT="PLACEHOLDER"
```

## Debug

### Check the number of available threads in the thread pool
```shell
sudo ${TOMCAT_HOME}/bin/catalina.sh status | grep "http-nio-8080" | awk '{print $7}'
```

### Check the number of threads currently in use by Tomcat
```shell
sudo lsof -p $(sudo lsof -i:8080 -t) | awk '{print $9}' | sort | uniq -c | sort -rn
```

### Check the thread dump of the JVM to identify the root cause of the thread pool exhaustion
```shell
sudo jstack ${TOMCAT_PID} | grep "java.util.concurrent.ThreadPoolExecutor" -A 20
```

### Check the number of threads blocked by a particular thread
```shell
sudo jstack ${TOMCAT_PID} | grep "${BLOCKED_THREAD_ID}" -A 20
```

### Check the overall thread usage and status of the JVM
```shell
sudo jstat -gcutil ${TOMCAT_PID} 1000 5
```

## Repair

### Tune thread pools - Adjust the thread pool size and configuration to match the workload and resources available.
```shell


#!/bin/bash



# Set the path to the Tomcat server.xml file

TOMCAT_CONFIG=${PATH_TO_TOMCAT}/conf/server.xml



# Set the new maxThreads value (adjust as needed)

NEW_MAX_THREADS=500



# Backup the original server.xml file

cp $TOMCAT_CONFIG $TOMCAT_CONFIG.bak



# Update the maxThreads parameter in the server.xml file

sed -i "s/<Connector port=\"8080\"/<Connector port=\"8080\" maxThreads=\"$NEW_MAX_THREADS\" /g" $TOMCAT_CONFIG



# Restart the Tomcat service

systemctl restart tomcat.service


```

When too many requests are sent to a Tomcat server, the thread pools that handle these requests can run out of available threads. This means that incoming requests will not be processed, leading to service degradation or even a complete outage. This incident type is critical for web applications that rely on Tomcat as their application server.


Tomcat Server Crash incident type refers to a situation where the Tomcat Server, an open-source web server and servlet container, becomes unresponsive and stops functioning. This can result in a complete or partial system outage, leading to a disruption in the delivery of web applications and services. The root cause of the crash can be attributed to a variety of factors such as misconfiguration, network or hardware issues, software bugs, or excessive server load. It is critical to address this incident quickly to minimize the impact on users and restore normal system functionality.


Tomcat Server Crash

This incident type occurs when the PermGen or Metaspace usage of the Tomcat server exceeds 90%. PermGen and Metaspace are memory pools used for storing class definitions and metadata of the Java virtual machine (JVM). When these memory pools reach their maximum capacity, the server may experience performance issues or even crash. This incident requires immediate attention from the software engineer to investigate the root cause and take appropriate actions to resolve the issue.


Tomcat PermGen or Metaspace Usage exceeds 90%

This incident type is related to the Tomcat server, which is a popular open-source Java application server. It occurs when the available threads in the connection pool are running low, which means that there are not enough resources to serve incoming requests. This can cause delays or even failures in the application's performance, impacting the end-user experience. It is important to monitor and address this issue promptly to ensure that the application runs smoothly and efficiently.


Tomcat Low Available Threads in Connection Pool

In software engineering, a large number of suspended threads in Tomcat is an incident type that occurs when a high number of threads are created but are blocked from executing. This can happen due to a variety of reasons, such as slow database queries, network latency, or resource contention. When this occurs, the system may become unresponsive or slow, causing a degradation in performance or even a complete system failure. It is important to investigate and resolve this issue as quickly as possible to ensure the system can continue to function properly.


Tomcat Large Number of Suspended Threads Incident.

Tomcat JVM OutOfMemory incident occurs when the Java Virtual Machine (JVM) running the Apache Tomcat server runs out of memory, which causes the server to crash or become unresponsive. This type of incident can happen due to a variety of reasons such as incorrect configuration settings, memory leaks, or insufficient memory allocation. When this incident occurs, it can impact the availability and performance of the web application running on the Tomcat server.


Tomcat JVM OutOfMemory Incident

```shell
export TOMCAT_HOME="PLACEHOLDER"

export TOMCAT_PID="PLACEHOLDER"

export BLOCKED_THREAD_ID="PLACEHOLDER"

export PATH_TO_TOMCAT="PLACEHOLDER"
```


### Check the number of available threads in the thread pool

```shell
sudo ${TOMCAT_HOME}/bin/catalina.sh status | grep "http-nio-8080" | awk '{print $7}'
```

### Check the number of threads currently in use by Tomcat

```shell
sudo lsof -p $(sudo lsof -i:8080 -t) | awk '{print $9}' | sort | uniq -c | sort -rn
```

### Check the thread dump of the JVM to identify the root cause of the thread pool exhaustion

```shell
sudo jstack ${TOMCAT_PID} | grep "java.util.concurrent.ThreadPoolExecutor" -A 20
```

### Check the number of threads blocked by a particular thread

```shell
sudo jstack ${TOMCAT_PID} | grep "${BLOCKED_THREAD_ID}" -A 20
```

### Check the overall thread usage and status of the JVM

```shell
sudo jstat -gcutil ${TOMCAT_PID} 1000 5
```


### Tune thread pools - Adjust the thread pool size and configuration to match the workload and resources available.

```shell


#!/bin/bash



# Set the path to the Tomcat server.xml file

TOMCAT_CONFIG=${PATH_TO_TOMCAT}/conf/server.xml



# Set the new maxThreads value (adjust as needed)

NEW_MAX_THREADS=500



# Backup the original server.xml file

cp $TOMCAT_CONFIG $TOMCAT_CONFIG.bak



# Update the maxThreads parameter in the server.xml file

sed -i "s/<Connector port=\"8080\"/<Connector port=\"8080\" maxThreads=\"$NEW_MAX_THREADS\" /g" $TOMCAT_CONFIG



# Restart the Tomcat service

systemctl restart tomcat.service


```


Tomcat thread pool exhaustion.

Overview

Parameters

Debug

Check the number of available threads in the thread pool

Check the number of threads currently in use by Tomcat

Check the thread dump of the JVM to identify the root cause of the thread pool exhaustion

Check the number of threads blocked by a particular thread

Check the overall thread usage and status of the JVM

Repair

Tune thread pools - Adjust the thread pool size and configuration to match the workload and resources available.

Learn more

Related Runbooks

Tomcat Server Crash

Tomcat PermGen or Metaspace Usage exceeds 90%

Tomcat Low Available Threads in Connection Pool

Tomcat Large Number of Suspended Threads Incident.

Support