---
id: deb4e616-d8a9-4e7f-b487-1f0adb9b3ad2
---

# High 5xx Errors on Traffic Server
---

This incident type occurs when the number of 5xx errors on Traffic Server is higher than usual. This can indicate issues with server performance or connectivity problems. It requires investigation to identify the root cause and resolve the issue.

### Parameters
```shell
# Environment Variables

export APP_LABEL="PLACEHOLDER"

export NAMESPACE="PLACEHOLDER"

export POD_NAME="PLACEHOLDER"

export NODE_NAME="PLACEHOLDER"

export DEPENDENT_SERVICE_NAME="PLACEHOLDER"

export DEPLOYMENT_NAME="PLACEHOLDER"

```

## Debug

### Check if the Traffic Server service is running
```shell
kubectl get pods -n ${NAMESPACE} -l app=${APP_LABEL} # Replace with actual values
```

### Check the logs of the Traffic Server pod for any errors
```shell
kubectl logs ${POD_NAME} -n ${NAMESPACE} # Replace with actual values
```

### Check the resource usage of the Traffic Server pod
```shell
kubectl top pod ${POD_NAME} -n ${NAMESPACE} # Replace with actual values
```

### Check if any other pods in the same node as the Traffic Server pod are consuming excessive resources
```shell
kubectl top pod -n ${NAMESPACE} --sort-by cpu | grep ${NODE_NAME} # Replace with actual values
```

### Check if there are any network issues between the Traffic Server pod and its dependant services
```shell
kubectl exec -it ${POD_NAME} -n ${NAMESPACE} -- traceroute ${DEPENDENT_SERVICE_NAME} # Replace with actual values
```

### Check the configuration of the Traffic Server deployment and its dependant services
```shell
kubectl describe deployment ${DEPLOYMENT_NAME} -n ${NAMESPACE} # Replace with actual values
```

### Check the health status of the Traffic Server pod and its dependant services
```shell
kubectl describe pod ${POD_NAME} -n ${NAMESPACE} # Replace with actual values
```

### Cyber attack or malicious activity targeting the Traffic Server.
```shell


#!/bin/bash



# Set the namespace and pod name for the Traffic Server

NAMESPACE=${NAMESPACE}

POD_NAME=${POD_NAME}



# Check the logs of the Traffic Server pod for any suspicious activity

kubectl logs $POD_NAME -n $NAMESPACE | grep -iE "attack|malicious|hack|exploit|vulnerability"



# Check the network traffic to and from the Traffic Server pod for any suspicious activity

kubectl exec -it $POD_NAME -n $NAMESPACE -- tcpdump -i eth0 -nn -s0 -w traffic.pcap &

TCPDUMP_PID=$!



sleep 10



kill -SIGINT $TCPDUMP_PID



# Analyze the captured network traffic for any suspicious activity

tcpdump -r traffic.pcap | grep -iE "attack|malicious|hack|exploit|vulnerability"



# Clean up the captured traffic file

rm -f traffic.pcap


```

### Traffic Server overloaded due to high traffic.
```shell


#!/bin/bash



# Set the namespace and pod name of the Traffic Server deployment

NAMESPACE=${NAMESPACE}

POD_NAME=$(kubectl get pods -n $NAMESPACE -l app=traffic-server -o jsonpath='{.items[0].metadata.name}')



# Check the CPU and memory usage of the Traffic Server pod

CPU_USAGE=$(kubectl top pod $POD_NAME -n $NAMESPACE --containers | awk '{print $2}' | tail -n 1 | sed 's/m$//')

MEMORY_USAGE=$(kubectl top pod $POD_NAME -n $NAMESPACE --containers | awk '{print $3}' | tail -n 1 | sed 's/Mi$//')



# Check the number of requests to the Traffic Server pod

REQUESTS=$(kubectl logs $POD_NAME -n $NAMESPACE | grep -c "HTTP/1.1\" 5")



# Check the number of open connections to the Traffic Server pod

OPEN_CONNECTIONS=$(kubectl exec -n $NAMESPACE $POD_NAME -- netstat -an | grep ESTABLISHED | grep -c ":80")



# Print the diagnostic information

echo "Traffic Server CPU usage: $CPU_USAGE m"

echo "Traffic Server memory usage: $MEMORY_USAGE Mi"

echo "Number of 5xx requests to Traffic Server: $REQUESTS"

echo "Number of open connections to Traffic Server: $OPEN_CONNECTIONS"


```

## Repair

### Implement HPA to handle increased traffic load.
```shell
# Set up HPA
CPU_THRESHOLD="PLACEHOLDER"
MIN_REPLICAS="PLACEHOLDER"
MAX_REPLICAS="PLACEHOLDER"

kubectl autoscale deployment $DEPLOYMENT_NAME --namespace $NAMESPACE --cpu-percent=$CPU_THRESHOLD --min=$MIN_REPLICAS --max=$MAX_REPLICAS


```

This incident type occurs when the number of 5xx errors on Traffic Server is higher than usual. This can indicate issues with server performance or connectivity problems. It requires investigation to identify the root cause and resolve the issue.


This incident type occurs when the number of 4xx errors on Traffic Server is at an anomalous level, higher than usual. It could be an indicator of an issue with the server or an increase in traffic that is causing errors. It requires investigation and resolution to ensure the server is functioning correctly and not impacting users.


High 4xx Errors on Traffic Server

This incident occurs when the Tomcat server is experiencing an unusually high number of active sessions. Active sessions refer to the number of users currently accessing a web application hosted on the Tomcat server. When the number of active sessions exceeds the server's capacity, it can lead to slow response times, unresponsive applications, or even server crashes. This incident requires immediate attention to resolve the issue and prevent any further impact on the web application's performance.


Tomcat High Number of Active Sessions.

This incident type refers to a situation where the Tomcat server's JDBC connection pool usage surpasses its configured limits. This can lead to a shortage of available connections for incoming requests, resulting in slow response times or even server crashes. It is important to monitor and manage the JDBC connection pool usage to prevent such incidents from occurring.


Tomcat High JDBC Connection Pool Usage Incident.

This incident type occurs when the size of HTTP request headers or payload exceeds the limits set by Tomcat server. When this happens, the server may reject the request or return an error message, causing disruption to the normal functioning of the web application. This can be caused by a variety of factors, including large file uploads, excessive use of cookies or query parameters, or attacks that intentionally send oversized requests.


Tomcat HTTP Request Headers or Payload Size Exceeds Limits.

This incident type refers to an increase in the number of errors per second on a Tomcat server, which could indicate an issue with the server itself, the host, a deployed application, or an application servlet. This could include errors generated when the Tomcat server runs out of memory, can't find a requested file or servlet, or is unable to serve a JSP due to syntax errors in the servlet codebase. This incident type requires immediate attention to diagnose and address the underlying issue.


Increase of the errors/second rate for Tomcat server

```shell
# Environment Variables

export APP_LABEL="PLACEHOLDER"

export NAMESPACE="PLACEHOLDER"

export POD_NAME="PLACEHOLDER"

export NODE_NAME="PLACEHOLDER"

export DEPENDENT_SERVICE_NAME="PLACEHOLDER"

export DEPLOYMENT_NAME="PLACEHOLDER"

```


### Check if the Traffic Server service is running

```shell
kubectl get pods -n ${NAMESPACE} -l app=${APP_LABEL} # Replace with actual values
```

### Check the logs of the Traffic Server pod for any errors

```shell
kubectl logs ${POD_NAME} -n ${NAMESPACE} # Replace with actual values
```

### Check the resource usage of the Traffic Server pod

```shell
kubectl top pod ${POD_NAME} -n ${NAMESPACE} # Replace with actual values
```

### Check if any other pods in the same node as the Traffic Server pod are consuming excessive resources

```shell
kubectl top pod -n ${NAMESPACE} --sort-by cpu | grep ${NODE_NAME} # Replace with actual values
```

### Check if there are any network issues between the Traffic Server pod and its dependant services

```shell
kubectl exec -it ${POD_NAME} -n ${NAMESPACE} -- traceroute ${DEPENDENT_SERVICE_NAME} # Replace with actual values
```

### Check the configuration of the Traffic Server deployment and its dependant services

```shell
kubectl describe deployment ${DEPLOYMENT_NAME} -n ${NAMESPACE} # Replace with actual values
```

### Check the health status of the Traffic Server pod and its dependant services

```shell
kubectl describe pod ${POD_NAME} -n ${NAMESPACE} # Replace with actual values
```

### Cyber attack or malicious activity targeting the Traffic Server.

```shell


#!/bin/bash



# Set the namespace and pod name for the Traffic Server

NAMESPACE=${NAMESPACE}

POD_NAME=${POD_NAME}



# Check the logs of the Traffic Server pod for any suspicious activity

kubectl logs $POD_NAME -n $NAMESPACE | grep -iE "attack|malicious|hack|exploit|vulnerability"



# Check the network traffic to and from the Traffic Server pod for any suspicious activity

kubectl exec -it $POD_NAME -n $NAMESPACE -- tcpdump -i eth0 -nn -s0 -w traffic.pcap &

TCPDUMP_PID=$!



sleep 10



kill -SIGINT $TCPDUMP_PID



# Analyze the captured network traffic for any suspicious activity

tcpdump -r traffic.pcap | grep -iE "attack|malicious|hack|exploit|vulnerability"



# Clean up the captured traffic file

rm -f traffic.pcap


```

### Traffic Server overloaded due to high traffic.

```shell


#!/bin/bash



# Set the namespace and pod name of the Traffic Server deployment

NAMESPACE=${NAMESPACE}

POD_NAME=$(kubectl get pods -n $NAMESPACE -l app=traffic-server -o jsonpath='{.items[0].metadata.name}')



# Check the CPU and memory usage of the Traffic Server pod

CPU_USAGE=$(kubectl top pod $POD_NAME -n $NAMESPACE --containers | awk '{print $2}' | tail -n 1 | sed 's/m$//')

MEMORY_USAGE=$(kubectl top pod $POD_NAME -n $NAMESPACE --containers | awk '{print $3}' | tail -n 1 | sed 's/Mi$//')



# Check the number of requests to the Traffic Server pod

REQUESTS=$(kubectl logs $POD_NAME -n $NAMESPACE | grep -c "HTTP/1.1\" 5")



# Check the number of open connections to the Traffic Server pod

OPEN_CONNECTIONS=$(kubectl exec -n $NAMESPACE $POD_NAME -- netstat -an | grep ESTABLISHED | grep -c ":80")



# Print the diagnostic information

echo "Traffic Server CPU usage: $CPU_USAGE m"

echo "Traffic Server memory usage: $MEMORY_USAGE Mi"

echo "Number of 5xx requests to Traffic Server: $REQUESTS"

echo "Number of open connections to Traffic Server: $OPEN_CONNECTIONS"


```


### Implement HPA to handle increased traffic load.

```shell
# Set up HPA
CPU_THRESHOLD="PLACEHOLDER"
MIN_REPLICAS="PLACEHOLDER"
MAX_REPLICAS="PLACEHOLDER"

kubectl autoscale deployment $DEPLOYMENT_NAME --namespace $NAMESPACE --cpu-percent=$CPU_THRESHOLD --min=$MIN_REPLICAS --max=$MAX_REPLICAS


```


High 5xx Errors on Traffic Server

Overview

Parameters

Debug

Check if the Traffic Server service is running

Check the logs of the Traffic Server pod for any errors

Check the resource usage of the Traffic Server pod

Check if any other pods in the same node as the Traffic Server pod are consuming excessive resources

Check if there are any network issues between the Traffic Server pod and its dependant services

Check the configuration of the Traffic Server deployment and its dependant services

Check the health status of the Traffic Server pod and its dependant services

Cyber attack or malicious activity targeting the Traffic Server.

Traffic Server overloaded due to high traffic.

Repair

Implement HPA to handle increased traffic load.

Learn more

Related Runbooks

High 4xx Errors on Traffic Server

Tomcat High Number of Active Sessions.

Tomcat High JDBC Connection Pool Usage Incident.

Tomcat HTTP Request Headers or Payload Size Exceeds Limits.

Support