---
id: aebf5979-0406-4fc2-a445-6e69f5887bd0
---
# Container high throttle rate incident.
---

This incident type typically occurs when a container is being throttled, meaning it is being limited in the amount of resources it can use. This can happen due to various reasons such as exceeding resource limits, network connectivity issues, or other performance problems. This can cause an interruption in the normal functioning of the application or service running in the container. It requires immediate attention to identify and resolve the underlying cause of the throttling to ensure normal operation of the application or service.

### Parameters
```shell
export CONTAINER_NAME="PLACEHOLDER"

export CONTAINER_PORT="PLACEHOLDER"

export MAX_THROTTLE_RATE="PLACEHOLDER"

export IMAGE_NAME="PLACEHOLDER"

export ADDITIONAL_PARAMETERS="PLACEHOLDER"
```

## Debug

### 1. Check the CPU usage of the container instance
```shell
docker stats ${CONTAINER_NAME}
```

### 2. Check the logs of the container for any errors or exceptions
```shell
docker logs ${CONTAINER_NAME}
```

### 3. Check if the container is running out of memory
```shell
docker stats --format "table {{.Name}}\t{{.MemUsage}}" ${CONTAINER_NAME}
```

### 4. Check the network traffic and connections of the container
```shell
sudo netstat -tulpn | grep ${CONTAINER_PORT}
```

### 7. Check the running processes inside the container
```shell
docker exec -it ${CONTAINER_NAME} ps aux
```

### 6. Check the container's resource limits
```shell
docker inspect ${CONTAINER_NAME} | grep -i mem
```

### 8. Check the container's environment variables
```shell
docker exec -it ${CONTAINER_NAME} env
```

### 10. Check the container's uptime and resource usage history
```shell
docker stats ${CONTAINER_NAME}
```

### The container may be processing more requests than it can handle, leading to a high throttle rate.
```shell


#!/bin/bash



# Set the container name to ${CONTAINER_NAME}

container_name="${CONTAINER_NAME}"



# Get the current throttle rate for the container

throttle_rate=$(docker stats --no-stream $container_name | awk 'NR==2{print $3}' | cut -d'%' -f1)



# Set the maximum throttle rate to ${MAX_THROTTLE_RATE}

max_throttle_rate=${MAX_THROTTLE_RATE}



# Check if the throttle rate is above the maximum

if [ "$throttle_rate" -gt "$max_throttle_rate" ]; then

  echo "Warning: Container is being throttled at $throttle_rate%"

  echo "Checking logs for high request rate..."

  

  # Look for log entries indicating a high request rate

  if docker logs $container_name | grep "high request rate"; then

    echo "Found log entries indicating a high request rate."

    echo "Checking network traffic..."

    

    # Check if there is high network traffic to the container

    if netstat -an | grep ":80" | grep "ESTABLISHED" | wc -l | awk '{print $1}' | tr -d '\n' > 100; then

      echo "High network traffic detected."

      echo "Suggest reducing the number of requests to the container or scaling up resources."

    else

      echo "No high network traffic detected."

      echo "Suggest checking for performance issues in the application or service running inside the container."

    fi

  else

    echo "No log entries indicating a high request rate."

    echo "Suggest checking for performance issues in the application or service running inside the container."

  fi

else

  echo "Container is not being throttled."

fi


```

## Repair

### A Script to restart the docker container
```shell
bash

#!/bin/bash



# Set the name of the container to restart

container_name=${CONTAINER_NAME}



# Stop the container

docker stop $container_name



# Remove the container

docker rm $container_name



# Pull the latest image

docker pull ${IMAGE_NAME}



# Start a new container with the same parameters as the old one

docker run --name $container_name ${ADDITIONAL_PARAMETERS} ${IMAGE_NAME}


```



This incident type typically occurs when a container is being throttled, meaning it is being limited in the amount of resources it can use. This can happen due to various reasons such as exceeding resource limits, network connectivity issues, or other performance problems. This can cause an interruption in the normal functioning of the application or service running in the container. It requires immediate attention to identify and resolve the underlying cause of the throttling to ensure normal operation of the application or service.


This incident type refers to an increase in the number of errors per second on a Tomcat server, which could indicate an issue with the server itself, the host, a deployed application, or an application servlet. This could include errors generated when the Tomcat server runs out of memory, can't find a requested file or servlet, or is unable to serve a JSP due to syntax errors in the servlet codebase. This incident type requires immediate attention to diagnose and address the underlying issue.


Increase of the errors/second rate for Tomcat server

Spark YARN container failure is an incident type that occurs when a container running on the YARN (Yet Another Resource Negotiator) resource manager in a Spark cluster fails to complete its assigned task. This can be caused by a variety of factors such as resource constraints, hardware failures, or software bugs. When a container fails, it can cause delays or failures in the overall Spark job, leading to reduced performance or data loss.


Spark YARN Container Failure

This incident type refers to the failure of a systemd service on a particular host instance. The incident could be triggered by various causes such as a software bug, hardware failure, or system overload. This type of incident can cause downtime or service disruption to the affected host instance, which may require immediate resolution to restore normal operations.


Host systemd service crashed (instance) incident.

This incident type relates to a high urgency issue regarding host context switching. The incident is triggered when the context switching grows on the node beyond a certain level, typically over 10000 per second. This issue can cause performance degradation and impact the stability of the system. The incident requires immediate attention from a software engineer to identify the root cause and take the necessary steps to resolve the issue.


High urgency incident related to host context switching.

A high CPU load incident occurs when the CPU usage of a host or server exceeds a certain threshold, typically 80%. This can cause slowdowns, unresponsiveness, or even crashes in the affected system. It is critical to address this issue promptly to minimize impact on users and prevent further damage to the system.


High CPU Load Incident

```shell
export CONTAINER_NAME="PLACEHOLDER"

export CONTAINER_PORT="PLACEHOLDER"

export MAX_THROTTLE_RATE="PLACEHOLDER"

export IMAGE_NAME="PLACEHOLDER"

export ADDITIONAL_PARAMETERS="PLACEHOLDER"
```


### 1. Check the CPU usage of the container instance

```shell
docker stats ${CONTAINER_NAME}
```

### 2. Check the logs of the container for any errors or exceptions

```shell
docker logs ${CONTAINER_NAME}
```

### 3. Check if the container is running out of memory

```shell
docker stats --format "table {{.Name}}\t{{.MemUsage}}" ${CONTAINER_NAME}
```

### 4. Check the network traffic and connections of the container

```shell
sudo netstat -tulpn | grep ${CONTAINER_PORT}
```

### 7. Check the running processes inside the container

```shell
docker exec -it ${CONTAINER_NAME} ps aux
```

### 6. Check the container's resource limits

```shell
docker inspect ${CONTAINER_NAME} | grep -i mem
```

### 8. Check the container's environment variables

```shell
docker exec -it ${CONTAINER_NAME} env
```

### 10. Check the container's uptime and resource usage history

```shell
docker stats ${CONTAINER_NAME}
```

### The container may be processing more requests than it can handle, leading to a high throttle rate.

```shell


#!/bin/bash



# Set the container name to ${CONTAINER_NAME}

container_name="${CONTAINER_NAME}"



# Get the current throttle rate for the container

throttle_rate=$(docker stats --no-stream $container_name | awk 'NR==2{print $3}' | cut -d'%' -f1)



# Set the maximum throttle rate to ${MAX_THROTTLE_RATE}

max_throttle_rate=${MAX_THROTTLE_RATE}



# Check if the throttle rate is above the maximum

if [ "$throttle_rate" -gt "$max_throttle_rate" ]; then

  echo "Warning: Container is being throttled at $throttle_rate%"

  echo "Checking logs for high request rate..."

  

  # Look for log entries indicating a high request rate

  if docker logs $container_name | grep "high request rate"; then

    echo "Found log entries indicating a high request rate."

    echo "Checking network traffic..."

    

    # Check if there is high network traffic to the container

    if netstat -an | grep ":80" | grep "ESTABLISHED" | wc -l | awk '{print $1}' | tr -d '\n' > 100; then

      echo "High network traffic detected."

      echo "Suggest reducing the number of requests to the container or scaling up resources."

    else

      echo "No high network traffic detected."

      echo "Suggest checking for performance issues in the application or service running inside the container."

    fi

  else

    echo "No log entries indicating a high request rate."

    echo "Suggest checking for performance issues in the application or service running inside the container."

  fi

else

  echo "Container is not being throttled."

fi


```


### A Script to restart the docker container

```shell
bash

#!/bin/bash



# Set the name of the container to restart

container_name=${CONTAINER_NAME}



# Stop the container

docker stop $container_name



# Remove the container

docker rm $container_name



# Pull the latest image

docker pull ${IMAGE_NAME}



# Start a new container with the same parameters as the old one

docker run --name $container_name ${ADDITIONAL_PARAMETERS} ${IMAGE_NAME}


```


Container high throttle rate incident.

Overview

Parameters

Debug

1. Check the CPU usage of the container instance

2. Check the logs of the container for any errors or exceptions

3. Check if the container is running out of memory

4. Check the network traffic and connections of the container

7. Check the running processes inside the container

6. Check the container's resource limits

8. Check the container's environment variables

10. Check the container's uptime and resource usage history

The container may be processing more requests than it can handle, leading to a high throttle rate.

Repair

A Script to restart the docker container

Learn more

Related Runbooks

Increase of the errors/second rate for Tomcat server

Spark YARN Container Failure

Host systemd service crashed (instance) incident.

High urgency incident related to host context switching.

Support