---
id: ed48696d-fbbe-44ce-a92e-23d5d4e5fb29
---

# Kubernetes - Available CPU for Limits in percentages Low
---

This incident type relates to a situation where the available CPU for limits in percentages in a Kubernetes cluster is low. The incident is triggered when a container uses more CPU resources than its specified request and limits, eventually leading to resource exhaustion. This can cause service disruptions and impact the performance of the Kubernetes cluster. The incident requires immediate attention to prevent further degradation of service quality.

### Parameters
```shell
# Environment Variables

export POD_NAME="PLACEHOLDER"

export NAMESPACE="PLACEHOLDER"

export DEPLOYMENT_NAME="PLACEHOLDER"

export CONTAINER_NAME="PLACEHOLDER"

export SELECTOR_FOR_THE_AFFECTED_PODS="PLACEHOLDER"

export NUMBER_OF_NODES_TO_ADD="PLACEHOLDER"
```

## Debug

### Check the Kubernetes cluster status
```shell
kubectl cluster-info
```

### Check the status of the Kubernetes nodes
```shell
kubectl get nodes
```

### Check the status of the Kubernetes pods
```shell
kubectl get pods
```

### Check the CPU resources for the Kubernetes nodes
```shell
kubectl describe nodes | grep -i cpu
```

### Check the CPU resources for the Kubernetes pods
```shell
kubectl describe pods ${POD_NAME} | grep -i cpu
```

### Check the resource limits for the Kubernetes pods
```shell
kubectl describe pods ${POD_NAME} | grep -i limits
```

### Check the CPU usage for the Kubernetes pods
```shell
kubectl top pods
```

### Check the resource requests for the Kubernetes pods
```shell
kubectl describe pods ${POD_NAME} | grep -i requests
```

### Check the CPU usage of the Kubernetes nodes
```shell
kubectl top nodes
```

### Heavy load on the Kubernetes cluster which has caused the CPU usage to spike and exceed the limits set in place.
```shell


#!/bin/bash



# Set the namespace for the deployment

NAMESPACE=${NAMESPACE}



# Set the deployment name

DEPLOYMENT=${DEPLOYMENT_NAME}



# Get the CPU usage for the deployment

CPU_USAGE=$(kubectl top pods -n $NAMESPACE | grep $DEPLOYMENT | awk '{print $2}')



# Get the CPU limit for the deployment

CPU_LIMIT=$(kubectl describe deployment $DEPLOYMENT -n $NAMESPACE | grep -i cpu | awk '{print $3}')



# Compare the CPU usage to the CPU limit

if (( $(echo "$CPU_USAGE > $CPU_LIMIT" |bc -l) )); then

    # Alert that the CPU usage has exceeded the limit

    echo "CPU usage for deployment $DEPLOYMENT in namespace $NAMESPACE has exceeded the limit of $CPU_LIMIT."

else

    # Alert that the CPU usage is within the limit

    echo "CPU usage for deployment $DEPLOYMENT in namespace $NAMESPACE is within the limit of $CPU_LIMIT."

fi


```

### Misconfiguration of Kubernetes resources such as incorrect CPU limit values or not enough resources allocated to the cluster.
```shell


#!/bin/bash



# Set variables

NAMESPACE=${NAMESPACE}

CONTAINER=${CONTAINER_NAME}



# Check if kubectl is installed

if ! command -v kubectl &> /dev/null

then

    echo "kubectl could not be found. Please install kubectl and try again."

    exit 1

fi



# Check if namespace exists

if ! kubectl get namespace $NAMESPACE &> /dev/null

then

    echo "Namespace $NAMESPACE not found. Please specify the correct namespace name."

    exit 1

fi



# Check if container exists in the namespace

if ! kubectl get pod -n $NAMESPACE -o jsonpath="{.items[*].metadata.name}" | grep $CONTAINER &> /dev/null

then

    echo "Container $CONTAINER not found in namespace $NAMESPACE. Please specify the correct container name."

    exit 1

fi



# Get CPU limits and requests for the container

CPU_LIMITS=$(kubectl get pod -n $NAMESPACE -o jsonpath="{.items[*].spec.containers[?(@.name=='$CONTAINER')].resources.limits.cpu}")

CPU_REQUESTS=$(kubectl get pod -n $NAMESPACE -o jsonpath="{.items[*].spec.containers[?(@.name=='$CONTAINER')].resources.requests.cpu}")



# Check if CPU limits and requests are correctly configured

if [ -z "$CPU_LIMITS" ] || [ -z "$CPU_REQUESTS" ] || [ "$CPU_LIMITS" -lt "$CPU_REQUESTS" ]

then

    echo "Misconfigured Kubernetes resources detected. CPU limits and requests are either not set or set incorrectly."

    exit 1

fi



echo "No issues detected with Kubernetes resources configuration."

exit 0


```

## Repair

### Increase the CPU limits for the affected Kubernetes Pods: Insufficient CPU limits may cause this incident. Increasing the limits can help address the issue.
```shell


#!/bin/bash



# Set variables

NAMESPACE=${NAMESPACE}

POD_SELECTOR=${SELECTOR_FOR_THE_AFFECTED_PODS}



# Increase CPU limits for the affected pods

kubectl -n $NAMESPACE patch pod -l $POD_SELECTOR --type=json -p='[{"op": "replace", "path": "/spec/containers/0/resources/limits/cpu", "value":{"cpu": "1"}}]'


```

### Add more CPU resources to the Kubernetes cluster: If the cluster is already running at maximum capacity and the CPU limits are set appropriately, you may need to add more CPU resources to the cluster to avoid this incident.
```shell


#!/bin/bash



# Set variables

KUBECONFIG=${PATH_TO_KUBECONFIG_FILE} # e.g. /home/user/kubeconfig.yaml

NODES=${NUMBER_OF_NODES_TO_ADD} # e.g. 2



# Increase CPU resources in Kubernetes cluster

kubectl --replicas=$NODES deployment/kube-system/kube-dns

kubectl scale --replicas=$NODES deployment/kube-system/kube-proxy


```

This incident type relates to a situation where the available CPU for limits in percentages in a Kubernetes cluster is low. The incident is triggered when a container uses more CPU resources than its specified request and limits, eventually leading to resource exhaustion. This can cause service disruptions and impact the performance of the Kubernetes cluster. The incident requires immediate attention to prevent further degradation of service quality.


This incident type involves monitoring the replicas of a Kubernetes Statefulset, which is a type of workload in Kubernetes used for stateful applications. The incident is triggered when more than one replica's pods are down, creating an unsafe situation for manual operations. This incident is critical and requires immediate attention to resolve the issue and ensure the smooth functioning of the stateful applications.


Kubernetes Statefulset Replicas Monitoring Incident

A Kubernetes Replicaset Incomplete incident typically occurs when a specific number of pods that should be running are not, due to reasons such as failed pod initialization, unavailability of resources in the cluster, or inability to pull the image. This incident is usually triggered when the difference between desired and running pods is greater than zero, and it can be detected through monitoring tools like Datadog.


Kubernetes Replicaset Incomplete

Kubernetes Pods Pending incident indicates that one or more pods in a Kubernetes cluster are not running as expected and are in a pending state. This can happen due to various reasons such as resource constraints, scheduling issues, or network problems. This incident can impact the availability and performance of the application running on the Kubernetes cluster. It requires immediate attention to diagnose and resolve the underlying issue to ensure the pods are running as expected.


Kubernetes Pods Pending

A Kubernetes Pod Restarting Monitoring incident is triggered when a pod running on a Kubernetes cluster restarts multiple times within a certain time frame. This incident type is usually used to detect issues with the application or infrastructure running on the cluster, and can be caused by various factors such as resource constraints, misconfigurations, or bugs in the application code. The incident is typically resolved by identifying and addressing the underlying cause of the pod restarts.


Kubernetes Pod Restarting Monitoring

The Kubernetes Nodes with Memorypressure incident type occurs when a Kubernetes cluster node is running low on memory, which can be caused by a memory leak in an application. This incident type requires immediate attention to prevent any downtime and ensure the proper functioning of the Kubernetes cluster. Typically, this incident type is monitored by DevOps teams using various monitoring tools, including PagerDuty, to identify and address memory pressure issues quickly.


Kubernetes Nodes with Memorypressure incident

```shell
# Environment Variables

export POD_NAME="PLACEHOLDER"

export NAMESPACE="PLACEHOLDER"

export DEPLOYMENT_NAME="PLACEHOLDER"

export CONTAINER_NAME="PLACEHOLDER"

export SELECTOR_FOR_THE_AFFECTED_PODS="PLACEHOLDER"

export NUMBER_OF_NODES_TO_ADD="PLACEHOLDER"
```


### Check the Kubernetes cluster status

```shell
kubectl cluster-info
```

### Check the status of the Kubernetes nodes

```shell
kubectl get nodes
```

### Check the status of the Kubernetes pods

```shell
kubectl get pods
```

### Check the CPU resources for the Kubernetes nodes

```shell
kubectl describe nodes | grep -i cpu
```

### Check the CPU resources for the Kubernetes pods

```shell
kubectl describe pods ${POD_NAME} | grep -i cpu
```

### Check the resource limits for the Kubernetes pods

```shell
kubectl describe pods ${POD_NAME} | grep -i limits
```

### Check the CPU usage for the Kubernetes pods

```shell
kubectl top pods
```

### Check the resource requests for the Kubernetes pods

```shell
kubectl describe pods ${POD_NAME} | grep -i requests
```

### Check the CPU usage of the Kubernetes nodes

```shell
kubectl top nodes
```

### Heavy load on the Kubernetes cluster which has caused the CPU usage to spike and exceed the limits set in place.

```shell


#!/bin/bash



# Set the namespace for the deployment

NAMESPACE=${NAMESPACE}



# Set the deployment name

DEPLOYMENT=${DEPLOYMENT_NAME}



# Get the CPU usage for the deployment

CPU_USAGE=$(kubectl top pods -n $NAMESPACE | grep $DEPLOYMENT | awk '{print $2}')



# Get the CPU limit for the deployment

CPU_LIMIT=$(kubectl describe deployment $DEPLOYMENT -n $NAMESPACE | grep -i cpu | awk '{print $3}')



# Compare the CPU usage to the CPU limit

if (( $(echo "$CPU_USAGE > $CPU_LIMIT" |bc -l) )); then

    # Alert that the CPU usage has exceeded the limit

    echo "CPU usage for deployment $DEPLOYMENT in namespace $NAMESPACE has exceeded the limit of $CPU_LIMIT."

else

    # Alert that the CPU usage is within the limit

    echo "CPU usage for deployment $DEPLOYMENT in namespace $NAMESPACE is within the limit of $CPU_LIMIT."

fi


```

### Misconfiguration of Kubernetes resources such as incorrect CPU limit values or not enough resources allocated to the cluster.

```shell


#!/bin/bash



# Set variables

NAMESPACE=${NAMESPACE}

CONTAINER=${CONTAINER_NAME}



# Check if kubectl is installed

if ! command -v kubectl &> /dev/null

then

    echo "kubectl could not be found. Please install kubectl and try again."

    exit 1

fi



# Check if namespace exists

if ! kubectl get namespace $NAMESPACE &> /dev/null

then

    echo "Namespace $NAMESPACE not found. Please specify the correct namespace name."

    exit 1

fi



# Check if container exists in the namespace

if ! kubectl get pod -n $NAMESPACE -o jsonpath="{.items[*].metadata.name}" | grep $CONTAINER &> /dev/null

then

    echo "Container $CONTAINER not found in namespace $NAMESPACE. Please specify the correct container name."

    exit 1

fi



# Get CPU limits and requests for the container

CPU_LIMITS=$(kubectl get pod -n $NAMESPACE -o jsonpath="{.items[*].spec.containers[?(@.name=='$CONTAINER')].resources.limits.cpu}")

CPU_REQUESTS=$(kubectl get pod -n $NAMESPACE -o jsonpath="{.items[*].spec.containers[?(@.name=='$CONTAINER')].resources.requests.cpu}")



# Check if CPU limits and requests are correctly configured

if [ -z "$CPU_LIMITS" ] || [ -z "$CPU_REQUESTS" ] || [ "$CPU_LIMITS" -lt "$CPU_REQUESTS" ]

then

    echo "Misconfigured Kubernetes resources detected. CPU limits and requests are either not set or set incorrectly."

    exit 1

fi



echo "No issues detected with Kubernetes resources configuration."

exit 0


```


### Increase the CPU limits for the affected Kubernetes Pods: Insufficient CPU limits may cause this incident. Increasing the limits can help address the issue.

```shell


#!/bin/bash



# Set variables

NAMESPACE=${NAMESPACE}

POD_SELECTOR=${SELECTOR_FOR_THE_AFFECTED_PODS}



# Increase CPU limits for the affected pods

kubectl -n $NAMESPACE patch pod -l $POD_SELECTOR --type=json -p='[{"op": "replace", "path": "/spec/containers/0/resources/limits/cpu", "value":{"cpu": "1"}}]'


```

### Add more CPU resources to the Kubernetes cluster: If the cluster is already running at maximum capacity and the CPU limits are set appropriately, you may need to add more CPU resources to the cluster to avoid this incident.

```shell


#!/bin/bash



# Set variables

KUBECONFIG=${PATH_TO_KUBECONFIG_FILE} # e.g. /home/user/kubeconfig.yaml

NODES=${NUMBER_OF_NODES_TO_ADD} # e.g. 2



# Increase CPU resources in Kubernetes cluster

kubectl --replicas=$NODES deployment/kube-system/kube-dns

kubectl scale --replicas=$NODES deployment/kube-system/kube-proxy


```


Kubernetes - Available CPU for Limits in percentages Low

Overview

Parameters

Debug

Check the Kubernetes cluster status

Check the status of the Kubernetes nodes

Check the status of the Kubernetes pods

Check the CPU resources for the Kubernetes nodes

Check the CPU resources for the Kubernetes pods

Check the resource limits for the Kubernetes pods

Check the CPU usage for the Kubernetes pods

Check the resource requests for the Kubernetes pods

Check the CPU usage of the Kubernetes nodes

Heavy load on the Kubernetes cluster which has caused the CPU usage to spike and exceed the limits set in place.

Misconfiguration of Kubernetes resources such as incorrect CPU limit values or not enough resources allocated to the cluster.

Repair

Increase the CPU limits for the affected Kubernetes Pods: Insufficient CPU limits may cause this incident. Increasing the limits can help address the issue.

Add more CPU resources to the Kubernetes cluster: If the cluster is already running at maximum capacity and the CPU limits are set appropriately, you may need to add more CPU resources to the cluster to avoid this incident.

Learn more

Related Runbooks

Kubernetes Statefulset Replicas Monitoring Incident

Kubernetes Replicaset Incomplete

Kubernetes Pods Pending

Kubernetes Pod Restarting Monitoring

Support