---
id: 4dca48fa-bc83-4038-94a0-1f26a98cb191
---

# High 4xx Errors on Traffic Server
---

This incident type occurs when the number of 4xx errors on Traffic Server is at an anomalous level, higher than usual. It could be an indicator of an issue with the server or an increase in traffic that is causing errors. It requires investigation and resolution to ensure the server is functioning correctly and not impacting users.

### Parameters
```shell
# Environment Variables

export POD_NAME="PLACEHOLDER"

export NAMESPACE="PLACEHOLDER"

export DEPLOYMENT_NAME="PLACEHOLDER"

export SERVICE_NAME="PLACEHOLDER"

export CONTAINER_NAME="PLACEHOLDER"

export PATH_TO_CONFIG_FILE="PLACEHOLDER"

```

## Debug

### Check if the affected pod is running
```shell
kubectl get pods -n ${NAMESPACE} | grep ${POD_NAME}
```

### Check the logs of the pod for any relevant error messages
```shell
kubectl logs ${POD_NAME} -n ${NAMESPACE}
```

### Check the status of the deployment
```shell
kubectl describe deployment ${DEPLOYMENT_NAME} -n ${NAMESPACE}
```

### Check the status of the service
```shell
kubectl describe service ${SERVICE_NAME} -n ${NAMESPACE}
```

### Check the logs of the Traffic Server container
```shell
kubectl logs ${POD_NAME} -c ${CONTAINER_NAME} -n ${NAMESPACE}
```

### Check the number of 4xx errors in the last 15 minutes
```shell
kubectl exec ${POD_NAME} -c ${CONTAINER_NAME} -n ${NAMESPACE} -- curl -s http://localhost:8000/stats | grep '4xx'
```

### Check the configuration file for Traffic Server
```shell
kubectl exec ${POD_NAME} -c ${CONTAINER_NAME} -n ${NAMESPACE} -- cat ${PATH_TO_CONFIG_FILE}
```

## Repair

### Set the namespace where Traffic Server is deployed
```shell
NAMESPACE=${NAMESPACE}
```

### Set the name of the Traffic Server pod
```shell
POD_NAME=${POD_NAME}
```

### Set the name of the Traffic Server container
```shell
CONTAINER_NAME=${CONTAINER_NAME}
```

### Set the path to the Traffic Server log file
```shell
LOG_FILE=${LOG_FILE}
```

### Get the logs from the Traffic Server pod and container
```shell
kubectl logs $POD_NAME -n $NAMESPACE -c $CONTAINER_NAME > /tmp/application.log
```

### Search for 4xx errors in the log file
```shell
grep " 4[0-9][0-9] " /tmp/application.log
```

### Check the exit code of the grep command to see if any 4xx errors were found
```shell
if [ $? -eq 0 ]

then

    echo "4xx errors found in the log file"

else

    echo "No 4xx errors found in the log file"

fi
```

### Increase the server capacity to handle the increased traffic, if applicable.
```shell


#!/bin/bash



# Set variables

DEPLOYMENT_NAME=${DEPLOYMENT_NAME}

NAMESPACE=${NAMESPACE}

REPLICAS="PLACEHOLDER"



# Scale up the deployment

kubectl scale deployment $DEPLOYMENT_NAME --namespace $NAMESPACE --replicas $REPLICAS


```

### Optimize the Traffic Server configuration to improve performance and reduce the number of 4xx errors.
```shell
bash

#!/bin/bash



# Set the Kubernetes namespace and the Traffic Server deployment name

NAMESPACE=${NAMESPACE}

DEPLOYMENT=${DEPLOYMENT}



# Get the current Traffic Server configuration

CURRENT_CONFIG=$(kubectl get configmap -n $NAMESPACE $DEPLOYMENT -o json | jq -r '.data.traffic_server_config')



# Modify the configuration to optimize performance

NEW_CONFIG=$(echo $CURRENT_CONFIG | sed 's/CONFIG_OPTION1=.*$/CONFIG_OPTION1=new_value/')

NEW_CONFIG=$(echo $NEW_CONFIG | sed 's/CONFIG_OPTION2=.*$/CONFIG_OPTION2=new_value/')



# Update the Traffic Server configuration

kubectl patch configmap -n $NAMESPACE $DEPLOYMENT --type merge -p "{\"data\":{\"traffic_server_config\":\"$NEW_CONFIG\"}}"



# Restart the Traffic Server deployment to apply the new configuration

kubectl rollout restart deployment -n $NAMESPACE $DEPLOYMENT


```

This incident type occurs when the number of 4xx errors on Traffic Server is at an anomalous level, higher than usual. It could be an indicator of an issue with the server or an increase in traffic that is causing errors. It requires investigation and resolution to ensure the server is functioning correctly and not impacting users.


This incident type occurs when the number of 5xx errors on Traffic Server is higher than usual. This can indicate issues with server performance or connectivity problems. It requires investigation to identify the root cause and resolve the issue.


High 5xx Errors on Traffic Server

This incident occurs when the Tomcat server is experiencing an unusually high number of active sessions. Active sessions refer to the number of users currently accessing a web application hosted on the Tomcat server. When the number of active sessions exceeds the server's capacity, it can lead to slow response times, unresponsive applications, or even server crashes. This incident requires immediate attention to resolve the issue and prevent any further impact on the web application's performance.


Tomcat High Number of Active Sessions.

This incident type refers to a situation where the Tomcat server's JDBC connection pool usage surpasses its configured limits. This can lead to a shortage of available connections for incoming requests, resulting in slow response times or even server crashes. It is important to monitor and manage the JDBC connection pool usage to prevent such incidents from occurring.


Tomcat High JDBC Connection Pool Usage Incident.

This incident type occurs when the size of HTTP request headers or payload exceeds the limits set by Tomcat server. When this happens, the server may reject the request or return an error message, causing disruption to the normal functioning of the web application. This can be caused by a variety of factors, including large file uploads, excessive use of cookies or query parameters, or attacks that intentionally send oversized requests.


Tomcat HTTP Request Headers or Payload Size Exceeds Limits.

This incident type refers to an increase in the number of errors per second on a Tomcat server, which could indicate an issue with the server itself, the host, a deployed application, or an application servlet. This could include errors generated when the Tomcat server runs out of memory, can't find a requested file or servlet, or is unable to serve a JSP due to syntax errors in the servlet codebase. This incident type requires immediate attention to diagnose and address the underlying issue.


Increase of the errors/second rate for Tomcat server

```shell
# Environment Variables

export POD_NAME="PLACEHOLDER"

export NAMESPACE="PLACEHOLDER"

export DEPLOYMENT_NAME="PLACEHOLDER"

export SERVICE_NAME="PLACEHOLDER"

export CONTAINER_NAME="PLACEHOLDER"

export PATH_TO_CONFIG_FILE="PLACEHOLDER"

```


### Check if the affected pod is running

```shell
kubectl get pods -n ${NAMESPACE} | grep ${POD_NAME}
```

### Check the logs of the pod for any relevant error messages

```shell
kubectl logs ${POD_NAME} -n ${NAMESPACE}
```

### Check the status of the deployment

```shell
kubectl describe deployment ${DEPLOYMENT_NAME} -n ${NAMESPACE}
```

### Check the status of the service

```shell
kubectl describe service ${SERVICE_NAME} -n ${NAMESPACE}
```

### Check the logs of the Traffic Server container

```shell
kubectl logs ${POD_NAME} -c ${CONTAINER_NAME} -n ${NAMESPACE}
```

### Check the number of 4xx errors in the last 15 minutes

```shell
kubectl exec ${POD_NAME} -c ${CONTAINER_NAME} -n ${NAMESPACE} -- curl -s http://localhost:8000/stats | grep '4xx'
```

### Check the configuration file for Traffic Server

```shell
kubectl exec ${POD_NAME} -c ${CONTAINER_NAME} -n ${NAMESPACE} -- cat ${PATH_TO_CONFIG_FILE}
```


### Set the namespace where Traffic Server is deployed

```shell
NAMESPACE=${NAMESPACE}
```

### Set the name of the Traffic Server pod

```shell
POD_NAME=${POD_NAME}
```

### Set the name of the Traffic Server container

```shell
CONTAINER_NAME=${CONTAINER_NAME}
```

### Set the path to the Traffic Server log file

```shell
LOG_FILE=${LOG_FILE}
```

### Get the logs from the Traffic Server pod and container

```shell
kubectl logs $POD_NAME -n $NAMESPACE -c $CONTAINER_NAME > /tmp/application.log
```

### Search for 4xx errors in the log file

```shell
grep " 4[0-9][0-9] " /tmp/application.log
```

### Check the exit code of the grep command to see if any 4xx errors were found

```shell
if [ $? -eq 0 ]

then

    echo "4xx errors found in the log file"

else

    echo "No 4xx errors found in the log file"

fi
```

### Increase the server capacity to handle the increased traffic, if applicable.

```shell


#!/bin/bash



# Set variables

DEPLOYMENT_NAME=${DEPLOYMENT_NAME}

NAMESPACE=${NAMESPACE}

REPLICAS="PLACEHOLDER"



# Scale up the deployment

kubectl scale deployment $DEPLOYMENT_NAME --namespace $NAMESPACE --replicas $REPLICAS


```

### Optimize the Traffic Server configuration to improve performance and reduce the number of 4xx errors.

```shell
bash

#!/bin/bash



# Set the Kubernetes namespace and the Traffic Server deployment name

NAMESPACE=${NAMESPACE}

DEPLOYMENT=${DEPLOYMENT}



# Get the current Traffic Server configuration

CURRENT_CONFIG=$(kubectl get configmap -n $NAMESPACE $DEPLOYMENT -o json | jq -r '.data.traffic_server_config')



# Modify the configuration to optimize performance

NEW_CONFIG=$(echo $CURRENT_CONFIG | sed 's/CONFIG_OPTION1=.*$/CONFIG_OPTION1=new_value/')

NEW_CONFIG=$(echo $NEW_CONFIG | sed 's/CONFIG_OPTION2=.*$/CONFIG_OPTION2=new_value/')



# Update the Traffic Server configuration

kubectl patch configmap -n $NAMESPACE $DEPLOYMENT --type merge -p "{\"data\":{\"traffic_server_config\":\"$NEW_CONFIG\"}}"



# Restart the Traffic Server deployment to apply the new configuration

kubectl rollout restart deployment -n $NAMESPACE $DEPLOYMENT


```


High 4xx Errors on Traffic Server

Overview

Parameters

Debug

Check if the affected pod is running

Check the logs of the pod for any relevant error messages

Check the status of the deployment

Check the status of the service

Check the logs of the Traffic Server container

Check the number of 4xx errors in the last 15 minutes

Check the configuration file for Traffic Server

Repair

Set the namespace where Traffic Server is deployed

Set the name of the Traffic Server pod

Set the name of the Traffic Server container

Set the path to the Traffic Server log file

Get the logs from the Traffic Server pod and container

Search for 4xx errors in the log file

Check the exit code of the grep command to see if any 4xx errors were found

Increase the server capacity to handle the increased traffic, if applicable.

Optimize the Traffic Server configuration to improve performance and reduce the number of 4xx errors.

Learn more

Related Runbooks

High 5xx Errors on Traffic Server

Tomcat High Number of Active Sessions.

Tomcat High JDBC Connection Pool Usage Incident.

Tomcat HTTP Request Headers or Payload Size Exceeds Limits.

Support