---
id: 4167a34f-70a2-41e2-be73-cd1876b70d73
---

# Anomalous max processing time for Tomcat host
---

This incident type typically refers to a situation where there is an anomaly in the Tomcat max processing time on a host. The maximum amount of time it takes for the server to process one request, from the time an available thread starts processing the request to the time it returns a response, has exceeded the expected threshold. This could indicate that a JSP page isn’t loading or an associated process (such as a database query) is taking too long to complete.

### Parameters
```shell
# Environment Variables

export PID="PLACEHOLDER"

export USERNAME="PLACEHOLDER"

export DATABASE_NAME="PLACEHOLDER"

export DESIRED_NETWORK_THRESHOLD="PLACEHOLDER"

export DESIRED_MEMORY_THRESHOLD="PLACEHOLDER"
```

## Debug

### Check if the Tomcat service is running
```shell
systemctl status tomcat
```

### Check the Tomcat access and error logs for any relevant messages
```shell
tail -n 50 catalina_out.log

tail -n 50 localhost_access.log

tail -n 50 localhost.log
```

### Check the CPU and memory usage of the server
```shell
top
```

### Check the network connections to the server
```shell
netstat -an
```

### Check the current JVM settings
```shell
jcmd ${PID} VM.flags
```

### Check the database server for performance issues
```shell
psql -U ${USERNAME} -d ${DATABASE_NAME} -c "SELECT * FROM pg_stat_activity;"
```

### The server may be experiencing a high load of requests, causing the Tomcat max processing time to spike.
```shell


#!/bin/bash



# Check CPU usage

CPU_USAGE=$(top -bn1 | grep load | awk '{print $(NF-2)}' | sed 's/,//')

CPU_THRESHOLD=80



if (( $(echo "$CPU_USAGE > $CPU_THRESHOLD" | bc -l) )); then

    echo "CPU usage is high: ${CPU_USAGE}%"

else

    echo "CPU usage is normal: ${CPU_USAGE}%"

fi



# Check memory usage

MEM_USAGE=$(free | awk '/Mem/{printf("%.2f\n"), $3/$2*100}')

MEM_THRESHOLD=${DESIRED_MEMORY_THRESHOLD}



if (( $(echo "$MEM_USAGE > $MEM_THRESHOLD" | bc -l) )); then

    echo "Memory usage is high: ${MEM_USAGE}%"

else

    echo "Memory usage is normal: ${MEM_USAGE}%"

fi



# Check network usage

NET_USAGE=$(netstat -an | grep :80 | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn | head -n 10)

NET_THRESHOLD=${DESIRED_NETWORK_THRESHOLD}



if (( $(echo "$NET_USAGE > $NET_THRESHOLD" | bc -l) )); then

    echo "Network usage is high: ${NET_USAGE}"

else

    echo "Network usage is normal: ${NET_USAGE}"

fi


```

## Repair

### Increase the resources allocated to the Tomcat server, such as CPU, memory, or network bandwidth.
```shell


#!/bin/bash



# Define variables

TOMCAT_HOME="PLACEHOLDER"

TOMCAT_USER="PLACEHOLDER"



# Stop Tomcat service

sudo systemctl stop tomcat



# Increase memory allocation

sudo sed -i 's/-Xms512m/-Xms1024m/g' $TOMCAT_HOME/bin/setenv.sh

sudo sed -i 's/-Xmx1024m/-Xmx2048m/g' $TOMCAT_HOME/bin/setenv.sh



# Start Tomcat service

sudo systemctl start tomcat


```

This incident type typically refers to a situation where there is an anomaly in the Tomcat max processing time on a host. The maximum amount of time it takes for the server to process one request, from the time an available thread starts processing the request to the time it returns a response, has exceeded the expected threshold. This could indicate that a JSP page isn’t loading or an associated process (such as a database query) is taking too long to complete.


When too many requests are sent to a Tomcat server, the thread pools that handle these requests can run out of available threads. This means that incoming requests will not be processed, leading to service degradation or even a complete outage. This incident type is critical for web applications that rely on Tomcat as their application server.


Tomcat thread pool exhaustion.

Tomcat Server Crash incident type refers to a situation where the Tomcat Server, an open-source web server and servlet container, becomes unresponsive and stops functioning. This can result in a complete or partial system outage, leading to a disruption in the delivery of web applications and services. The root cause of the crash can be attributed to a variety of factors such as misconfiguration, network or hardware issues, software bugs, or excessive server load. It is critical to address this incident quickly to minimize the impact on users and restore normal system functionality.


Tomcat Server Crash

This incident type occurs when the PermGen or Metaspace usage of the Tomcat server exceeds 90%. PermGen and Metaspace are memory pools used for storing class definitions and metadata of the Java virtual machine (JVM). When these memory pools reach their maximum capacity, the server may experience performance issues or even crash. This incident requires immediate attention from the software engineer to investigate the root cause and take appropriate actions to resolve the issue.


Tomcat PermGen or Metaspace Usage exceeds 90%

This incident type is related to the Tomcat server, which is a popular open-source Java application server. It occurs when the available threads in the connection pool are running low, which means that there are not enough resources to serve incoming requests. This can cause delays or even failures in the application's performance, impacting the end-user experience. It is important to monitor and address this issue promptly to ensure that the application runs smoothly and efficiently.


Tomcat Low Available Threads in Connection Pool

In software engineering, a large number of suspended threads in Tomcat is an incident type that occurs when a high number of threads are created but are blocked from executing. This can happen due to a variety of reasons, such as slow database queries, network latency, or resource contention. When this occurs, the system may become unresponsive or slow, causing a degradation in performance or even a complete system failure. It is important to investigate and resolve this issue as quickly as possible to ensure the system can continue to function properly.


Tomcat Large Number of Suspended Threads Incident.

```shell
# Environment Variables

export PID="PLACEHOLDER"

export USERNAME="PLACEHOLDER"

export DATABASE_NAME="PLACEHOLDER"

export DESIRED_NETWORK_THRESHOLD="PLACEHOLDER"

export DESIRED_MEMORY_THRESHOLD="PLACEHOLDER"
```


### Check if the Tomcat service is running

```shell
systemctl status tomcat
```

### Check the Tomcat access and error logs for any relevant messages

```shell
tail -n 50 catalina_out.log

tail -n 50 localhost_access.log

tail -n 50 localhost.log
```

### Check the CPU and memory usage of the server

```shell
top
```

### Check the network connections to the server

```shell
netstat -an
```

### Check the current JVM settings

```shell
jcmd ${PID} VM.flags
```

### Check the database server for performance issues

```shell
psql -U ${USERNAME} -d ${DATABASE_NAME} -c "SELECT * FROM pg_stat_activity;"
```

### The server may be experiencing a high load of requests, causing the Tomcat max processing time to spike.

```shell


#!/bin/bash



# Check CPU usage

CPU_USAGE=$(top -bn1 | grep load | awk '{print $(NF-2)}' | sed 's/,//')

CPU_THRESHOLD=80



if (( $(echo "$CPU_USAGE > $CPU_THRESHOLD" | bc -l) )); then

    echo "CPU usage is high: ${CPU_USAGE}%"

else

    echo "CPU usage is normal: ${CPU_USAGE}%"

fi



# Check memory usage

MEM_USAGE=$(free | awk '/Mem/{printf("%.2f\n"), $3/$2*100}')

MEM_THRESHOLD=${DESIRED_MEMORY_THRESHOLD}



if (( $(echo "$MEM_USAGE > $MEM_THRESHOLD" | bc -l) )); then

    echo "Memory usage is high: ${MEM_USAGE}%"

else

    echo "Memory usage is normal: ${MEM_USAGE}%"

fi



# Check network usage

NET_USAGE=$(netstat -an | grep :80 | awk '{print $5}' | cut -d: -f1 | sort | uniq -c | sort -rn | head -n 10)

NET_THRESHOLD=${DESIRED_NETWORK_THRESHOLD}



if (( $(echo "$NET_USAGE > $NET_THRESHOLD" | bc -l) )); then

    echo "Network usage is high: ${NET_USAGE}"

else

    echo "Network usage is normal: ${NET_USAGE}"

fi


```


### Increase the resources allocated to the Tomcat server, such as CPU, memory, or network bandwidth.

```shell


#!/bin/bash



# Define variables

TOMCAT_HOME="PLACEHOLDER"

TOMCAT_USER="PLACEHOLDER"



# Stop Tomcat service

sudo systemctl stop tomcat



# Increase memory allocation

sudo sed -i 's/-Xms512m/-Xms1024m/g' $TOMCAT_HOME/bin/setenv.sh

sudo sed -i 's/-Xmx1024m/-Xmx2048m/g' $TOMCAT_HOME/bin/setenv.sh



# Start Tomcat service

sudo systemctl start tomcat


```


Anomalous max processing time for Tomcat host

Overview

Parameters

Debug

Check if the Tomcat service is running

Check the Tomcat access and error logs for any relevant messages

Check the CPU and memory usage of the server

Check the network connections to the server

Check the current JVM settings

Check the database server for performance issues

The server may be experiencing a high load of requests, causing the Tomcat max processing time to spike.

Repair

Increase the resources allocated to the Tomcat server, such as CPU, memory, or network bandwidth.

Learn more

Related Runbooks

Tomcat thread pool exhaustion.

Tomcat Server Crash

Tomcat PermGen or Metaspace Usage exceeds 90%

Tomcat Low Available Threads in Connection Pool

Support