---
id: a5e519bb-4e99-4785-9784-cef49041e0da
---

# Tomcat Large Number of Suspended Threads Incident.
---

In software engineering, a large number of suspended threads in Tomcat is an incident type that occurs when a high number of threads are created but are blocked from executing. This can happen due to a variety of reasons, such as slow database queries, network latency, or resource contention. When this occurs, the system may become unresponsive or slow, causing a degradation in performance or even a complete system failure. It is important to investigate and resolve this issue as quickly as possible to ensure the system can continue to function properly.

### Parameters
```shell
export PORT_NUMBER="PLACEHOLDER"

export PID="PLACEHOLDER"

export MAX_THREADS="PLACEHOLDER"

export MAX_IDLE_TIME="PLACEHOLDER"
```

## Debug

### Next Step
```shell
sh
```

### Identify the PID of the Tomcat process
```shell
PID=$(sudo lsof -t -i:${PORT_NUMBER})
```

### Check the number of threads blocked in the JVM
```shell
sudo jstack -l $PID | grep "java.lang.Thread.State" | grep -c "BLOCKED"
```

### Check the number of threads waiting on a monitor lock
```shell
sudo jstack -l $PID | grep "java.lang.Thread.State" | grep -c "WAITING"
```

### Check the number of threads in the "TIMED_WAITING" state
```shell
sudo jstack -l $PID | grep "java.lang.Thread.State" | grep -c "TIMED_WAITING"
```

### Check the number of threads in the "RUNNABLE" state
```shell
sudo jstack -l $PID | grep "java.lang.Thread.State" | grep -c "RUNNABLE"
```

### Check the current heap usage of the Tomcat JVM
```shell
sudo jmap -heap $PID | grep -A 1 "used" | tail -n 1 | awk '{print $1}'
```

### Check the current CPU usage for the Tomcat process
```shell
sudo top -p $PID -b -n 1 | grep "Cpu(s)" | awk '{print $2 + $4}'
```

## Repair


### Tune the thread pool settings in Tomcat to prevent the creation of too many threads and limit the amount of time a thread is allowed to remain suspended.
```shell


#!/bin/bash

# Set the thread pool settings in Tomcat configuration

sed -i 's/<Connector port="8080"/<Connector port="8080" maxThreads="${MAX_THREADS}" maxIdleTime="${MAX_IDLE_TIME}"/' /path/to/your/server.xml


# Restart Tomcat service

systemctl restart tomcat


```

In software engineering, a large number of suspended threads in Tomcat is an incident type that occurs when a high number of threads are created but are blocked from executing. This can happen due to a variety of reasons, such as slow database queries, network latency, or resource contention. When this occurs, the system may become unresponsive or slow, causing a degradation in performance or even a complete system failure. It is important to investigate and resolve this issue as quickly as possible to ensure the system can continue to function properly.


When too many requests are sent to a Tomcat server, the thread pools that handle these requests can run out of available threads. This means that incoming requests will not be processed, leading to service degradation or even a complete outage. This incident type is critical for web applications that rely on Tomcat as their application server.


Tomcat thread pool exhaustion.

Tomcat Server Crash incident type refers to a situation where the Tomcat Server, an open-source web server and servlet container, becomes unresponsive and stops functioning. This can result in a complete or partial system outage, leading to a disruption in the delivery of web applications and services. The root cause of the crash can be attributed to a variety of factors such as misconfiguration, network or hardware issues, software bugs, or excessive server load. It is critical to address this incident quickly to minimize the impact on users and restore normal system functionality.


Tomcat Server Crash

This incident type occurs when the PermGen or Metaspace usage of the Tomcat server exceeds 90%. PermGen and Metaspace are memory pools used for storing class definitions and metadata of the Java virtual machine (JVM). When these memory pools reach their maximum capacity, the server may experience performance issues or even crash. This incident requires immediate attention from the software engineer to investigate the root cause and take appropriate actions to resolve the issue.


Tomcat PermGen or Metaspace Usage exceeds 90%

This incident type is related to the Tomcat server, which is a popular open-source Java application server. It occurs when the available threads in the connection pool are running low, which means that there are not enough resources to serve incoming requests. This can cause delays or even failures in the application's performance, impacting the end-user experience. It is important to monitor and address this issue promptly to ensure that the application runs smoothly and efficiently.


Tomcat Low Available Threads in Connection Pool

Tomcat JVM OutOfMemory incident occurs when the Java Virtual Machine (JVM) running the Apache Tomcat server runs out of memory, which causes the server to crash or become unresponsive. This type of incident can happen due to a variety of reasons such as incorrect configuration settings, memory leaks, or insufficient memory allocation. When this incident occurs, it can impact the availability and performance of the web application running on the Tomcat server.


Tomcat JVM OutOfMemory Incident

```shell
export PORT_NUMBER="PLACEHOLDER"

export PID="PLACEHOLDER"

export MAX_THREADS="PLACEHOLDER"

export MAX_IDLE_TIME="PLACEHOLDER"
```


### Next Step

```shell
sh
```

### Identify the PID of the Tomcat process

```shell
PID=$(sudo lsof -t -i:${PORT_NUMBER})
```

### Check the number of threads blocked in the JVM

```shell
sudo jstack -l $PID | grep "java.lang.Thread.State" | grep -c "BLOCKED"
```

### Check the number of threads waiting on a monitor lock

```shell
sudo jstack -l $PID | grep "java.lang.Thread.State" | grep -c "WAITING"
```

### Check the number of threads in the "TIMED\_WAITING" state

```shell
sudo jstack -l $PID | grep "java.lang.Thread.State" | grep -c "TIMED_WAITING"
```

### Check the number of threads in the "RUNNABLE" state

```shell
sudo jstack -l $PID | grep "java.lang.Thread.State" | grep -c "RUNNABLE"
```

### Check the current heap usage of the Tomcat JVM

```shell
sudo jmap -heap $PID | grep -A 1 "used" | tail -n 1 | awk '{print $1}'
```

### Check the current CPU usage for the Tomcat process

```shell
sudo top -p $PID -b -n 1 | grep "Cpu(s)" | awk '{print $2 + $4}'
```


### Tune the thread pool settings in Tomcat to prevent the creation of too many threads and limit the amount of time a thread is allowed to remain suspended.

```shell


#!/bin/bash

# Set the thread pool settings in Tomcat configuration

sed -i 's/<Connector port="8080"/<Connector port="8080" maxThreads="${MAX_THREADS}" maxIdleTime="${MAX_IDLE_TIME}"/' /path/to/your/server.xml


# Restart Tomcat service

systemctl restart tomcat


```


Tomcat Large Number of Suspended Threads Incident.

Overview

Parameters

Debug

Next Step

Identify the PID of the Tomcat process

Check the number of threads blocked in the JVM

Check the number of threads waiting on a monitor lock

Check the number of threads in the "TIMED_WAITING" state

Check the number of threads in the "RUNNABLE" state

Check the current heap usage of the Tomcat JVM

Check the current CPU usage for the Tomcat process

Repair

Tune the thread pool settings in Tomcat to prevent the creation of too many threads and limit the amount of time a thread is allowed to remain suspended.

Learn more

Related Runbooks

Tomcat thread pool exhaustion.

Tomcat Server Crash

Tomcat PermGen or Metaspace Usage exceeds 90%

Tomcat Low Available Threads in Connection Pool

Support