---
id: 2c7bab8e-21ff-4df3-88b1-7cb2fcd052bc
---

# Container Volume Usage Incident.
---

A Container Volume Usage Incident occurs when the usage of a container volume exceeds a certain threshold. This incident is triggered by an alert when the usage of the container volume goes above the predefined limit. The incident is created to notify the relevant team or individual about the issue so that they can take necessary actions to resolve it. The incident may include information about the affected instance, service, team, and the details of the incident, such as the volume usage value and labels. The incident is resolved when the volume usage goes below the threshold, and the team or individual acknowledges the resolution.

### Parameters
```shell
export PVC_NAME="PLACEHOLDER"

export NAMESPACE="PLACEHOLDER"

export POD_NAME="PLACEHOLDER"

export CONTAINER_NAME="PLACEHOLDER"

export EVENT_NAME="PLACEHOLDER"

export DEPLOYMENT_NAME="PLACEHOLDER"

export NEW_VOLUME_SIZE="PLACEHOLDER"

export VOLUME_NAME="PLACEHOLDER"
```

## Debug

### List all persistent volumes (PVs)
```shell
kubectl get pv
```

### List all persistent volume claims (PVCs)
```shell
kubectl get pvc
```

### Describe a specific PVC
```shell
kubectl describe pvc ${PVC_NAME}
```

### List all pods in the namespace
```shell
kubectl get pods -n ${NAMESPACE}
```

### Describe a specific pod
```shell
kubectl describe pod ${POD_NAME} -n ${NAMESPACE}
```

### List all containers in a pod
```shell
kubectl get pods ${POD_NAME} -n ${NAMESPACE} -o jsonpath='{.spec.containers[*].name}'
```

### Check container logs for a specific container in a pod
```shell
kubectl logs ${POD_NAME} -n ${NAMESPACE} ${CONTAINER_NAME}
```

### List all events in the namespace
```shell
kubectl get events -n ${NAMESPACE}
```

### Describe a specific event
```shell
kubectl describe event ${EVENT_NAME} -n ${NAMESPACE}
```

### The container has consumed an excessive amount of data, causing the volume usage to exceed its limit.
```shell


#!/bin/bash



# Set the namespace and pod name variables

NAMESPACE=${NAMESPACE}

POD_NAME=${POD_NAME}



# Get the disk usage for the pod

DISK_USAGE=$(kubectl exec -n $NAMESPACE $POD_NAME -- df -h / | awk 'NR==2{print $5}')



# Get the disk limit for the pod

DISK_LIMIT=$(kubectl describe pod -n $NAMESPACE $POD_NAME | grep -i 'limit' | awk '{print $2}')



# Check if the disk usage is greater than the limit

if [ $DISK_USAGE -gt $DISK_LIMIT ]; then

  # Alert the team about the incident

  echo "Container has exceeded the disk limit. Disk usage: $DISK_USAGE, Disk limit: $DISK_LIMIT"

  # Collect logs and metrics for further analysis

  kubectl logs -n $NAMESPACE $POD_NAME > pod_logs.txt

  kubectl describe pod -n $NAMESPACE $POD_NAME > pod_description.txt

  # Restart the pod to release the disk space

  kubectl delete pod -n $NAMESPACE $POD_NAME

fi


```

### The container is not set up to auto-scale, and the amount of data being processed is too large for the current container configuration.
```shell


#!/bin/bash



# Set the namespace and deployment name

NAMESPACE=${NAMESPACE}

DEPLOYMENT_NAME=${DEPLOYMENT_NAME}



# Get the current replica count

REPLICA_COUNT=$(kubectl get deployment $DEPLOYMENT_NAME -n $NAMESPACE -o=jsonpath='{.spec.replicas}')



# Get the CPU and memory usage of the current pods

CPU_USAGE=$(kubectl top pods -n $NAMESPACE | grep $DEPLOYMENT_NAME | awk '{print $2}')

MEMORY_USAGE=$(kubectl top pods -n $NAMESPACE | grep $DEPLOYMENT_NAME | awk '{print $3}')



# Get the CPU and memory requests and limits of the deployment

CPU_REQUEST=$(kubectl get deployment $DEPLOYMENT_NAME -n $NAMESPACE -o=jsonpath='{.spec.template.spec.containers[0].resources.requests.cpu}')

MEMORY_REQUEST=$(kubectl get deployment $DEPLOYMENT_NAME -n $NAMESPACE -o=jsonpath='{.spec.template.spec.containers[0].resources.requests.memory}')

CPU_LIMIT=$(kubectl get deployment $DEPLOYMENT_NAME -n $NAMESPACE -o=jsonpath='{.spec.template.spec.containers[0].resources.limits.cpu}')

MEMORY_LIMIT=$(kubectl get deployment $DEPLOYMENT_NAME -n $NAMESPACE -o=jsonpath='{.spec.template.spec.containers[0].resources.limits.memory}')



# Check if the CPU or memory usage is exceeding the requests or limits

if [[ $CPU_USAGE -gt $CPU_REQUEST ]] || [[ $MEMORY_USAGE -gt $MEMORY_REQUEST ]]; then

    echo "CPU or memory usage is exceeding the requests. Please check the container configuration."

    exit 1

elif [[ $CPU_USAGE -gt $CPU_LIMIT ]] || [[ $MEMORY_USAGE -gt $MEMORY_LIMIT ]]; then

    echo "CPU or memory usage is exceeding the limits. Please check the container configuration."

    exit 1

elif [[ $REPLICA_COUNT -lt 2 ]]; then

    echo "The current replica count is too low. Please consider increasing the replica count to handle the workload."

    exit 1

else

    echo "The container configuration is sufficient to handle the workload."

    exit 0

fi


```

## Repair

### Increase the size of the container volume or add a new volume to accommodate additional data.
```shell


#!/bin/bash



# Set the name of the deployment and container volume.

DEPLOYMENT_NAME=${DEPLOYMENT_NAME}

VOLUME_NAME=${VOLUME_NAME}



# Set the size of the new container volume.

NEW_VOLUME_SIZE=${NEW_VOLUME_SIZE}



# Resize the container volume.

# If the volume already exists, use this command to resize it:

kubectl patch persistentvolumeclaim $VOLUME_NAME -p '{"spec":{"resources":{"requests":{"storage":"'$NEW_VOLUME_SIZE'"}}}}'



# If a new volume needs to be added, use this command instead:

kubectl apply -f - <<EOF

apiVersion: v1

kind: PersistentVolumeClaim

metadata:

  name: $VOLUME_NAME

spec:

  accessModes:

  - ReadWriteOnce

  resources:

    requests:

      storage: $NEW_VOLUME_SIZE

EOF



# Scale the deployment to apply the changes.

kubectl scale deployment $DEPLOYMENT_NAME --replicas=0

kubectl scale deployment $DEPLOYMENT_NAME --replicas=1


```

A Container Volume Usage Incident occurs when the usage of a container volume exceeds a certain threshold. This incident is triggered by an alert when the usage of the container volume goes above the predefined limit. The incident is created to notify the relevant team or individual about the issue so that they can take necessary actions to resolve it. The incident may include information about the affected instance, service, team, and the details of the incident, such as the volume usage value and labels. The incident is resolved when the volume usage goes below the threshold, and the team or individual acknowledges the resolution.


​
AWS EBS (Elastic Block Store) Burst Credits Exhausted is an incident that occurs when an Amazon EBS volume has exhausted its burst credit balance. This happens when the volume has been used continuously at its maximum IOPS (input/output operations per second) rate for an extended period of time, causing the volume to consume its burst credits faster than it can earn them. As a result, the volume's performance is significantly degraded until the burst credits are replenished.
​


AWS EBS Burst Credits Exhausted

The Vault too many pending tokens incident refers to an issue where the Vault server has too many pending tokens. This can happen when the number of tokens generated exceeds the limit of what the server can handle, causing a backlog of requests. As a result, users may experience difficulty accessing certain resources or functions that require token authentication. This incident type typically requires the attention of a system administrator or engineer to investigate and resolve the underlying cause, such as increasing the token limit or optimizing the token generation process.


Vault too many pending tokens incident on kubernetes

A high CPU load incident occurs when the CPU usage of a host or server exceeds a certain threshold, typically 80%. This can cause slowdowns, unresponsiveness, or even crashes in the affected system. It is critical to address this issue promptly to minimize impact on users and prevent further damage to the system.


High CPU Load Incident

This incident type involves monitoring the replicas of a Kubernetes Statefulset, which is a type of workload in Kubernetes used for stateful applications. The incident is triggered when more than one replica's pods are down, creating an unsafe situation for manual operations. This incident is critical and requires immediate attention to resolve the issue and ensure the smooth functioning of the stateful applications.


Kubernetes Statefulset Replicas Monitoring Incident

A Kubernetes Pod ImagePullBackOff incident occurs when a pod in a Kubernetes cluster is unable to pull its container image. This can happen due to various reasons, such as incorrect image path or tag, or misconfigured image pulling credentials. This incident can cause the pod to fail to start and impact the availability of the application running in the pod. It requires investigation and resolution to ensure the pod can pull its container image and restart successfully.


Kubernetes Pod ImagePullBackOff Incident

```shell
export PVC_NAME="PLACEHOLDER"

export NAMESPACE="PLACEHOLDER"

export POD_NAME="PLACEHOLDER"

export CONTAINER_NAME="PLACEHOLDER"

export EVENT_NAME="PLACEHOLDER"

export DEPLOYMENT_NAME="PLACEHOLDER"

export NEW_VOLUME_SIZE="PLACEHOLDER"

export VOLUME_NAME="PLACEHOLDER"
```


### List all persistent volumes (PVs)

```shell
kubectl get pv
```

### List all persistent volume claims (PVCs)

```shell
kubectl get pvc
```

### Describe a specific PVC

```shell
kubectl describe pvc ${PVC_NAME}
```

### List all pods in the namespace

```shell
kubectl get pods -n ${NAMESPACE}
```

### Describe a specific pod

```shell
kubectl describe pod ${POD_NAME} -n ${NAMESPACE}
```

### List all containers in a pod

```shell
kubectl get pods ${POD_NAME} -n ${NAMESPACE} -o jsonpath='{.spec.containers[*].name}'
```

### Check container logs for a specific container in a pod

```shell
kubectl logs ${POD_NAME} -n ${NAMESPACE} ${CONTAINER_NAME}
```

### List all events in the namespace

```shell
kubectl get events -n ${NAMESPACE}
```

### Describe a specific event

```shell
kubectl describe event ${EVENT_NAME} -n ${NAMESPACE}
```

### The container has consumed an excessive amount of data, causing the volume usage to exceed its limit.

```shell


#!/bin/bash



# Set the namespace and pod name variables

NAMESPACE=${NAMESPACE}

POD_NAME=${POD_NAME}



# Get the disk usage for the pod

DISK_USAGE=$(kubectl exec -n $NAMESPACE $POD_NAME -- df -h / | awk 'NR==2{print $5}')



# Get the disk limit for the pod

DISK_LIMIT=$(kubectl describe pod -n $NAMESPACE $POD_NAME | grep -i 'limit' | awk '{print $2}')



# Check if the disk usage is greater than the limit

if [ $DISK_USAGE -gt $DISK_LIMIT ]; then

  # Alert the team about the incident

  echo "Container has exceeded the disk limit. Disk usage: $DISK_USAGE, Disk limit: $DISK_LIMIT"

  # Collect logs and metrics for further analysis

  kubectl logs -n $NAMESPACE $POD_NAME > pod_logs.txt

  kubectl describe pod -n $NAMESPACE $POD_NAME > pod_description.txt

  # Restart the pod to release the disk space

  kubectl delete pod -n $NAMESPACE $POD_NAME

fi


```

### The container is not set up to auto-scale, and the amount of data being processed is too large for the current container configuration.

```shell


#!/bin/bash



# Set the namespace and deployment name

NAMESPACE=${NAMESPACE}

DEPLOYMENT_NAME=${DEPLOYMENT_NAME}



# Get the current replica count

REPLICA_COUNT=$(kubectl get deployment $DEPLOYMENT_NAME -n $NAMESPACE -o=jsonpath='{.spec.replicas}')



# Get the CPU and memory usage of the current pods

CPU_USAGE=$(kubectl top pods -n $NAMESPACE | grep $DEPLOYMENT_NAME | awk '{print $2}')

MEMORY_USAGE=$(kubectl top pods -n $NAMESPACE | grep $DEPLOYMENT_NAME | awk '{print $3}')



# Get the CPU and memory requests and limits of the deployment

CPU_REQUEST=$(kubectl get deployment $DEPLOYMENT_NAME -n $NAMESPACE -o=jsonpath='{.spec.template.spec.containers[0].resources.requests.cpu}')

MEMORY_REQUEST=$(kubectl get deployment $DEPLOYMENT_NAME -n $NAMESPACE -o=jsonpath='{.spec.template.spec.containers[0].resources.requests.memory}')

CPU_LIMIT=$(kubectl get deployment $DEPLOYMENT_NAME -n $NAMESPACE -o=jsonpath='{.spec.template.spec.containers[0].resources.limits.cpu}')

MEMORY_LIMIT=$(kubectl get deployment $DEPLOYMENT_NAME -n $NAMESPACE -o=jsonpath='{.spec.template.spec.containers[0].resources.limits.memory}')



# Check if the CPU or memory usage is exceeding the requests or limits

if [[ $CPU_USAGE -gt $CPU_REQUEST ]] || [[ $MEMORY_USAGE -gt $MEMORY_REQUEST ]]; then

    echo "CPU or memory usage is exceeding the requests. Please check the container configuration."

    exit 1

elif [[ $CPU_USAGE -gt $CPU_LIMIT ]] || [[ $MEMORY_USAGE -gt $MEMORY_LIMIT ]]; then

    echo "CPU or memory usage is exceeding the limits. Please check the container configuration."

    exit 1

elif [[ $REPLICA_COUNT -lt 2 ]]; then

    echo "The current replica count is too low. Please consider increasing the replica count to handle the workload."

    exit 1

else

    echo "The container configuration is sufficient to handle the workload."

    exit 0

fi


```


### Increase the size of the container volume or add a new volume to accommodate additional data.

```shell


#!/bin/bash



# Set the name of the deployment and container volume.

DEPLOYMENT_NAME=${DEPLOYMENT_NAME}

VOLUME_NAME=${VOLUME_NAME}



# Set the size of the new container volume.

NEW_VOLUME_SIZE=${NEW_VOLUME_SIZE}



# Resize the container volume.

# If the volume already exists, use this command to resize it:

kubectl patch persistentvolumeclaim $VOLUME_NAME -p '{"spec":{"resources":{"requests":{"storage":"'$NEW_VOLUME_SIZE'"}}}}'



# If a new volume needs to be added, use this command instead:

kubectl apply -f - <<EOF

apiVersion: v1

kind: PersistentVolumeClaim

metadata:

  name: $VOLUME_NAME

spec:

  accessModes:

  - ReadWriteOnce

  resources:

    requests:

      storage: $NEW_VOLUME_SIZE

EOF



# Scale the deployment to apply the changes.

kubectl scale deployment $DEPLOYMENT_NAME --replicas=0

kubectl scale deployment $DEPLOYMENT_NAME --replicas=1


```


Container Volume Usage Incident.

Overview

Parameters

Debug

List all persistent volumes (PVs)

List all persistent volume claims (PVCs)

Describe a specific PVC

List all pods in the namespace

Describe a specific pod

List all containers in a pod

Check container logs for a specific container in a pod

List all events in the namespace

Describe a specific event

The container has consumed an excessive amount of data, causing the volume usage to exceed its limit.

The container is not set up to auto-scale, and the amount of data being processed is too large for the current container configuration.

Repair

Increase the size of the container volume or add a new volume to accommodate additional data.

Learn more

Related Runbooks

AWS EBS Burst Credits Exhausted

Vault too many pending tokens incident on kubernetes

High CPU Load Incident

Kubernetes Statefulset Replicas Monitoring Incident

Support