---
id: 039acc01-f48b-446a-9230-73851a731417
---

# Kubernetes CPU Availability Low
---

The "Kubernetes CPU Availability Low" incident type refers to an incident where the available CPU for requests in a Kubernetes cluster is too low. This can cause performance issues and potential downtime for the affected service. This incident may be triggered by a monitoring system or an alert from Kubernetes itself. It is important to investigate and resolve this incident promptly to ensure the service is operating optimally.

### Parameters
```shell
# Environment Variables

export POD_NAME="PLACEHOLDER"

export CONTAINER_NAME="PLACEHOLDER"

export CLUSTER_NAME="PLACEHOLDER"

export NODE_INSTANCE_TYPE="PLACEHOLDER"

export NODE_COUNT="PLACEHOLDER"

export NAMESPACE="PLACEHOLDER"

export DEPLOYMENT="PLACEHOLDER"

export CPU_THRESHOLD="PLACEHOLDER"
```

## Debug

### Check the CPU usage of all the pods in the Kubernetes cluster
```shell
kubectl top pods --all-namespaces
```

### Check the CPU usage of all the nodes in the Kubernetes cluster
```shell
kubectl top nodes
```

### Check the CPU limits and requests of the pods in the Kubernetes cluster
```shell
kubectl describe pods --all-namespaces | grep -E 'Limits|Requests'
```

### Check the CPU usage of the containers in a specific pod
```shell
kubectl exec ${POD_NAME} -c ${CONTAINER_NAME} -- top -n 1 -b | grep -E 'CPU'
```

### Check the Kubernetes events to see if there are any relevant error messages
```shell
kubectl get events --sort-by=.metadata.creationTimestamp
```

### Resource limits were not set properly in containers, leading to overconsumption of CPU resources.
```shell


#!/bin/bash



# Set the Kubernetes namespace and deployment name

NAMESPACE=${NAMESPACE}

DEPLOYMENT=${DEPLOYMENT}



# Get the pod name for the deployment

POD=$(kubectl get pods -n $NAMESPACE -l app=$DEPLOYMENT -o jsonpath='{.items[0].metadata.name}')



# Get the CPU usage for the pod

CPU_USAGE=$(kubectl top pod $POD -n $NAMESPACE --containers | awk '{print $2}' | tail -n1)



# Get the CPU limit for the pod

CPU_LIMIT=$(kubectl get pod $POD -n $NAMESPACE -o jsonpath='{.spec.containers[0].resources.limits.cpu}')



# Get the CPU request for the pod

CPU_REQUEST=$(kubectl get pod $POD -n $NAMESPACE -o jsonpath='{.spec.containers[0].resources.requests.cpu}')



# Compare the CPU usage to the limit and request

if (( $(bc ${<< "$CPU_USAGE } $CPU_LIMIT") )); then

    echo "CPU usage for $POD is above the limit of $CPU_LIMIT"

elif (( $(bc ${<< "$CPU_USAGE } $CPU_REQUEST") )); then

    echo "CPU usage for $POD is above the request of $CPU_REQUEST"

else

    echo "CPU usage is within limits for $POD"

fi


```

### Inadequate resource allocation by the Kubernetes cluster, leading to insufficient CPU resources being available.
```shell


#!/bin/bash



# Define variables

NAMESPACE=${NAMESPACE}

DEPLOYMENT=${DEPLOYMENT}

CONTAINER=${CONTAINER}

CPU_THRESHOLD=${CPU_THRESHOLD}



# Get the CPU usage for the specified container in the specified deployment

CPU_USAGE=$(kubectl top pod -n $NAMESPACE -l app=$DEPLOYMENT | grep $CONTAINER | awk '{print $2}')



# Check if the CPU usage exceeds the threshold

if (( $(echo "$CPU_USAGE > $CPU_THRESHOLD" | bc -l) )); then

  # If the CPU usage exceeds the threshold, print a warning and exit with status 1

  echo "Warning: CPU usage for $CONTAINER in $DEPLOYMENT is above the threshold of $CPU_THRESHOLD"

  exit 1

else

  # If the CPU usage is below the threshold, print a success message and exit with status 0

  echo "CPU usage for $CONTAINER in $DEPLOYMENT is below the threshold of $CPU_THRESHOLD"

  exit 0

fi


```

## Repair

### Increase the resources allocated to the Kubernetes cluster to avoid CPU contention.
```shell


#!/bin/bash



# Set variables

CLUSTER_NAME=${CLUSTER_NAME}

NODE_COUNT=${NODE_COUNT}

NODE_INSTANCE_TYPE=${NODE_INSTANCE_TYPE}



# Update node group

eksctl get nodegroup --cluster=$CLUSTER_NAME -o yaml > nodegroup.yaml

sed -i "s/instanceType:.*/instanceType: $NODE_INSTANCE_TYPE/" nodegroup.yaml

sed -i "s/minSize:.*/minSize: $NODE_COUNT/" nodegroup.yaml

sed -i "s/maxSize:.*/maxSize: $NODE_COUNT/" nodegroup.yaml

eksctl apply nodegroup -f nodegroup.yaml --approve



# Wait for nodes to be ready

while [ `kubectl get nodes | grep Ready | wc -l` -lt $NODE_COUNT ];

do

    sleep 5

done



echo "Node group updated successfully."


```

### Identify and terminate idle or unused containers to free up resources.
```shell


#!/bin/bash



# Set the namespace where the pods are running

NAMESPACE=${NAMESPACE}



# Get the name of the deployment or replicaset

DEPLOYMENT=${DEPLOYMENT}



# Get the threshold for CPU usage

THRESHOLD=${THRESHOLD}



# Get the list of pods running in the deployment or replicaset

PODS=$(kubectl get pods -n $NAMESPACE -l app=$DEPLOYMENT -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}')



# Loop through each pod and check CPU usage

for POD in $PODS

do

  # Get the current CPU usage for the pod

  CPU=$(kubectl top pod $POD -n $NAMESPACE --containers | awk '/^'$POD'/ {print $2}')

  

  # Check if the CPU usage is below the threshold

  if (( $(echo "$CPU < $THRESHOLD" | bc -l) ))

  then

    # If the CPU usage is below the threshold, delete the pod

    kubectl delete pod $POD -n $NAMESPACE

    echo "Deleted pod $POD to free up resources."

  else

    echo "Skipping pod $POD as its CPU usage is above the threshold."

  fi

done



echo "Completed remediation of identifying and terminating idle or unused containers."


```

The "Kubernetes CPU Availability Low" incident type refers to an incident where the available CPU for requests in a Kubernetes cluster is too low. This can cause performance issues and potential downtime for the affected service. This incident may be triggered by a monitoring system or an alert from Kubernetes itself. It is important to investigate and resolve this incident promptly to ensure the service is operating optimally.


This incident type involves an issue with Kubernetes deployments where the expected number of pods to run is not matching the actual number of pods running. This can lead to alerts being triggered and potential disruptions in the system.


Kubernetes pods not starting - Deployment issue

This incident type involves monitoring the replicas of a Kubernetes Statefulset, which is a type of workload in Kubernetes used for stateful applications. The incident is triggered when more than one replica's pods are down, creating an unsafe situation for manual operations. This incident is critical and requires immediate attention to resolve the issue and ensure the smooth functioning of the stateful applications.


Kubernetes Statefulset Replicas Monitoring Incident

A Kubernetes Replicaset Incomplete incident typically occurs when a specific number of pods that should be running are not, due to reasons such as failed pod initialization, unavailability of resources in the cluster, or inability to pull the image. This incident is usually triggered when the difference between desired and running pods is greater than zero, and it can be detected through monitoring tools like Datadog.


Kubernetes Replicaset Incomplete

Kubernetes Pods Pending incident indicates that one or more pods in a Kubernetes cluster are not running as expected and are in a pending state. This can happen due to various reasons such as resource constraints, scheduling issues, or network problems. This incident can impact the availability and performance of the application running on the Kubernetes cluster. It requires immediate attention to diagnose and resolve the underlying issue to ensure the pods are running as expected.


Kubernetes Pods Pending

A Kubernetes Pod Restarting Monitoring incident is triggered when a pod running on a Kubernetes cluster restarts multiple times within a certain time frame. This incident type is usually used to detect issues with the application or infrastructure running on the cluster, and can be caused by various factors such as resource constraints, misconfigurations, or bugs in the application code. The incident is typically resolved by identifying and addressing the underlying cause of the pod restarts.


Kubernetes Pod Restarting Monitoring

```shell
# Environment Variables

export POD_NAME="PLACEHOLDER"

export CONTAINER_NAME="PLACEHOLDER"

export CLUSTER_NAME="PLACEHOLDER"

export NODE_INSTANCE_TYPE="PLACEHOLDER"

export NODE_COUNT="PLACEHOLDER"

export NAMESPACE="PLACEHOLDER"

export DEPLOYMENT="PLACEHOLDER"

export CPU_THRESHOLD="PLACEHOLDER"
```


### Check the CPU usage of all the pods in the Kubernetes cluster

```shell
kubectl top pods --all-namespaces
```

### Check the CPU usage of all the nodes in the Kubernetes cluster

```shell
kubectl top nodes
```

### Check the CPU limits and requests of the pods in the Kubernetes cluster

```shell
kubectl describe pods --all-namespaces | grep -E 'Limits|Requests'
```

### Check the CPU usage of the containers in a specific pod

```shell
kubectl exec ${POD_NAME} -c ${CONTAINER_NAME} -- top -n 1 -b | grep -E 'CPU'
```

### Check the Kubernetes events to see if there are any relevant error messages

```shell
kubectl get events --sort-by=.metadata.creationTimestamp
```

### Resource limits were not set properly in containers, leading to overconsumption of CPU resources.

```shell


#!/bin/bash



# Set the Kubernetes namespace and deployment name

NAMESPACE=${NAMESPACE}

DEPLOYMENT=${DEPLOYMENT}



# Get the pod name for the deployment

POD=$(kubectl get pods -n $NAMESPACE -l app=$DEPLOYMENT -o jsonpath='{.items[0].metadata.name}')



# Get the CPU usage for the pod

CPU_USAGE=$(kubectl top pod $POD -n $NAMESPACE --containers | awk '{print $2}' | tail -n1)



# Get the CPU limit for the pod

CPU_LIMIT=$(kubectl get pod $POD -n $NAMESPACE -o jsonpath='{.spec.containers[0].resources.limits.cpu}')



# Get the CPU request for the pod

CPU_REQUEST=$(kubectl get pod $POD -n $NAMESPACE -o jsonpath='{.spec.containers[0].resources.requests.cpu}')



# Compare the CPU usage to the limit and request

if (( $(bc ${<< "$CPU_USAGE } $CPU_LIMIT") )); then

    echo "CPU usage for $POD is above the limit of $CPU_LIMIT"

elif (( $(bc ${<< "$CPU_USAGE } $CPU_REQUEST") )); then

    echo "CPU usage for $POD is above the request of $CPU_REQUEST"

else

    echo "CPU usage is within limits for $POD"

fi


```

### Inadequate resource allocation by the Kubernetes cluster, leading to insufficient CPU resources being available.

```shell


#!/bin/bash



# Define variables

NAMESPACE=${NAMESPACE}

DEPLOYMENT=${DEPLOYMENT}

CONTAINER=${CONTAINER}

CPU_THRESHOLD=${CPU_THRESHOLD}



# Get the CPU usage for the specified container in the specified deployment

CPU_USAGE=$(kubectl top pod -n $NAMESPACE -l app=$DEPLOYMENT | grep $CONTAINER | awk '{print $2}')



# Check if the CPU usage exceeds the threshold

if (( $(echo "$CPU_USAGE > $CPU_THRESHOLD" | bc -l) )); then

  # If the CPU usage exceeds the threshold, print a warning and exit with status 1

  echo "Warning: CPU usage for $CONTAINER in $DEPLOYMENT is above the threshold of $CPU_THRESHOLD"

  exit 1

else

  # If the CPU usage is below the threshold, print a success message and exit with status 0

  echo "CPU usage for $CONTAINER in $DEPLOYMENT is below the threshold of $CPU_THRESHOLD"

  exit 0

fi


```


### Increase the resources allocated to the Kubernetes cluster to avoid CPU contention.

```shell


#!/bin/bash



# Set variables

CLUSTER_NAME=${CLUSTER_NAME}

NODE_COUNT=${NODE_COUNT}

NODE_INSTANCE_TYPE=${NODE_INSTANCE_TYPE}



# Update node group

eksctl get nodegroup --cluster=$CLUSTER_NAME -o yaml > nodegroup.yaml

sed -i "s/instanceType:.*/instanceType: $NODE_INSTANCE_TYPE/" nodegroup.yaml

sed -i "s/minSize:.*/minSize: $NODE_COUNT/" nodegroup.yaml

sed -i "s/maxSize:.*/maxSize: $NODE_COUNT/" nodegroup.yaml

eksctl apply nodegroup -f nodegroup.yaml --approve



# Wait for nodes to be ready

while [ `kubectl get nodes | grep Ready | wc -l` -lt $NODE_COUNT ];

do

    sleep 5

done



echo "Node group updated successfully."


```

### Identify and terminate idle or unused containers to free up resources.

```shell


#!/bin/bash



# Set the namespace where the pods are running

NAMESPACE=${NAMESPACE}



# Get the name of the deployment or replicaset

DEPLOYMENT=${DEPLOYMENT}



# Get the threshold for CPU usage

THRESHOLD=${THRESHOLD}



# Get the list of pods running in the deployment or replicaset

PODS=$(kubectl get pods -n $NAMESPACE -l app=$DEPLOYMENT -o jsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}')



# Loop through each pod and check CPU usage

for POD in $PODS

do

  # Get the current CPU usage for the pod

  CPU=$(kubectl top pod $POD -n $NAMESPACE --containers | awk '/^'$POD'/ {print $2}')

  

  # Check if the CPU usage is below the threshold

  if (( $(echo "$CPU < $THRESHOLD" | bc -l) ))

  then

    # If the CPU usage is below the threshold, delete the pod

    kubectl delete pod $POD -n $NAMESPACE

    echo "Deleted pod $POD to free up resources."

  else

    echo "Skipping pod $POD as its CPU usage is above the threshold."

  fi

done



echo "Completed remediation of identifying and terminating idle or unused containers."


```


Kubernetes CPU Availability Low

Overview

Parameters

Debug

Check the CPU usage of all the pods in the Kubernetes cluster

Check the CPU usage of all the nodes in the Kubernetes cluster

Check the CPU limits and requests of the pods in the Kubernetes cluster

Check the CPU usage of the containers in a specific pod

Check the Kubernetes events to see if there are any relevant error messages

Resource limits were not set properly in containers, leading to overconsumption of CPU resources.

Inadequate resource allocation by the Kubernetes cluster, leading to insufficient CPU resources being available.

Repair

Increase the resources allocated to the Kubernetes cluster to avoid CPU contention.

Identify and terminate idle or unused containers to free up resources.

Learn more

Related Runbooks

Kubernetes pods not starting - Deployment issue

Kubernetes Statefulset Replicas Monitoring Incident

Kubernetes Replicaset Incomplete

Kubernetes Pods Pending

Support