---
id: 683df8af-dd34-4a4e-a3f1-0f1f2ab9e883
---

# Kubernetes Pods Pending
---

Kubernetes Pods Pending incident indicates that one or more pods in a Kubernetes cluster are not running as expected and are in a pending state. This can happen due to various reasons such as resource constraints, scheduling issues, or network problems. This incident can impact the availability and performance of the application running on the Kubernetes cluster. It requires immediate attention to diagnose and resolve the underlying issue to ensure the pods are running as expected.

### Parameters
```shell
# Environment Variables

export NAMESPACE="PLACEHOLDER"

export POD_NAME="PLACEHOLDER"

export CONTAINER_NAME="PLACEHOLDER"

export RESOURCE_TYPE="PLACEHOLDER"

export RESOURCE_LIMIT="PLACEHOLDER"
```

## Debug

### Get the list of namespaces in the Kubernetes cluster
```shell
kubectl get namespaces
```

### Get the list of pods in a specific namespace
```shell
kubectl get pods -n ${NAMESPACE}
```

### Get the details of a specific pod
```shell
kubectl describe pod ${POD_NAME} -n ${NAMESPACE}
```

### Get the list of events related to a specific pod
```shell
kubectl get events -n ${NAMESPACE} --field-selector involvedObject.name=${POD_NAME}
```

### Check the pod's YAML configuration file for any issues
```shell
kubectl get pod ${POD_NAME} -n ${NAMESPACE} -o yaml
```
### Check the status of the nodes in the Kubernetes cluster
```shell
kubectl get nodes
```

### Check the resource usage of the nodes in the Kubernetes cluster
```shell
kubectl top nodes
```

### Insufficient resources allocated to the Kubernetes cluster, such as CPU, memory, or storage.
```shell
#!/bin/bash

# Set variables

NAMESPACE=${NAMESPACE}

POD_NAME=${POD_NAME}

RESOURCE=${RESOURCE_TYPE} # e.g. cpu, memory, storage

# Get the resource limits and requests for the pod

LIMIT=$(kubectl describe pod $POD_NAME -n $NAMESPACE | grep "Limits" | awk '{print $2}')

REQUEST=$(kubectl describe pod $POD_NAME -n $NAMESPACE | grep "Requests" | awk '{print $2}')

# Remove the unit from the limit and request values

LIMIT=${LIMIT//Mi/}

LIMIT=${LIMIT//Gi/}

REQUEST=${REQUEST//Mi/}

REQUEST=${REQUEST//Gi/}

# Convert the limit and request values to integers

LIMIT=$(echo $LIMIT | awk '{print int($1)}')

REQUEST=$(echo $REQUEST | awk '{print int($1)}')

# Get the usage of the resource for the pod

USAGE=$(kubectl top pod $POD_NAME -n $NAMESPACE | awk '{print $3}')

USAGE=${USAGE//Mi/}

USAGE=${USAGE//Gi/}

USAGE=$(echo $USAGE | awk '{print int($1)}')

# Calculate the available resource for the pod

AVAILABLE=$((LIMIT - USAGE))

# Check if the available resource is less than the requested resource

if [ $AVAILABLE -lt $REQUEST ]; then

    echo "Insufficient $RESOURCE allocated to the Kubernetes cluster for pod $POD_NAME"

    echo "Limit: $LIMIT$RESOURCE, Request: $REQUEST$RESOURCE, Usage: $USAGE$RESOURCE, Available: $AVAILABLE$RESOURCE"

else

    echo "Sufficient $RESOURCE allocated to the Kubernetes cluster for pod $POD_NAME"

    echo "Limit: $LIMIT$RESOURCE, Request: $REQUEST$RESOURCE, Usage: $USAGE$RESOURCE, Available: $AVAILABLE$RESOURCE"

fi


```

### Misconfiguration of the pod specifications, such as incorrect image names or container ports, preventing the pods from starting.
```shell
#!/bin/bash

# Replace ${POD_NAME} and ${NAMESPACE} with the actual values

POD_NAME="${POD_NAME}"

NAMESPACE="${NAMESPACE}"

# Get the pod description and check the status

POD_DESC=$(kubectl describe pod $POD_NAME -n $NAMESPACE)

STATUS=$(echo "$POD_DESC" | awk '/Status:/{print $2}')

if [[ "$STATUS" == "Error" ]]; then

  # Check the container status and logs

  CONTAINER_STATUS=$(echo "$POD_DESC" | awk '/Container Status:/{print $3}')

  if [[ "$CONTAINER_STATUS" == "Error" ]]; then

    CONTAINER_NAME=$(echo "$POD_DESC" | awk '/Container ID:/{print $3}' | cut -d '/' -f 3)

    CONTAINER_LOGS=$(kubectl logs $POD_NAME -c $CONTAINER_NAME -n $NAMESPACE)

    echo "Container logs: $CONTAINER_LOGS"

  else

    echo "Container status: $CONTAINER_STATUS"

  fi

else

  echo "Pod status: $STATUS"

fi

```

## Repair

### Check the resource allocation for the pods to ensure that they have sufficient resources to run. Increase the resource allocation if necessary.
```shell
#!/bin/bash

# Set parameters

NAMESPACE=${NAMESPACE}

POD_NAME=${POD_NAME}

RESOURCE=${RESOURCE_TYPE}

RESOURCE_LIMIT=${RESOURCE_LIMIT}

# Check current resource allocation

CURRENT_LIMIT=$(kubectl get pod $POD_NAME -n $NAMESPACE -o=jsonpath="{.spec.containers[0].resources.limits.$RESOURCE}")

echo "Current $RESOURCE limit for $POD_NAME is $CURRENT_LIMIT"

# Increase resource allocation if necessary

if [ "$CURRENT_LIMIT" -lt "$RESOURCE_LIMIT" ]; then

    kubectl patch pod $POD_NAME -n $NAMESPACE --type='json' -p='[{"op": "replace", "path": "/spec/containers/0/resources/limits/'$RESOURCE'", "value": "'$RESOURCE_LIMIT'"}]'

    echo "Increased $RESOURCE limit for $POD_NAME to $RESOURCE_LIMIT"

else

    echo "Current $RESOURCE limit for $POD_NAME is already greater than or equal to $RESOURCE_LIMIT"

fi

```

Kubernetes Pods Pending incident indicates that one or more pods in a Kubernetes cluster are not running as expected and are in a pending state. This can happen due to various reasons such as resource constraints, scheduling issues, or network problems. This incident can impact the availability and performance of the application running on the Kubernetes cluster. It requires immediate attention to diagnose and resolve the underlying issue to ensure the pods are running as expected.


The Vault too many pending tokens incident refers to an issue where the Vault server has too many pending tokens. This can happen when the number of tokens generated exceeds the limit of what the server can handle, causing a backlog of requests. As a result, users may experience difficulty accessing certain resources or functions that require token authentication. This incident type typically requires the attention of a system administrator or engineer to investigate and resolve the underlying cause, such as increasing the token limit or optimizing the token generation process.


Vault too many pending tokens incident on kubernetes

The Vault cluster health incident is related to the health of a Vault cluster instance. This incident type is triggered when the cluster instance is not healthy and requires attention to ensure it is functioning properly. The incident typically involves evaluating the current state of the cluster instance, diagnosing the issue, and taking corrective action to restore the health of the instance.


Vault cluster health incident on kubernetes

This incident type involves an issue with Kubernetes deployments where the expected number of pods to run is not matching the actual number of pods running. This can lead to alerts being triggered and potential disruptions in the system.


Kubernetes pods not starting - Deployment issue

This incident type involves monitoring the replicas of a Kubernetes Statefulset, which is a type of workload in Kubernetes used for stateful applications. The incident is triggered when more than one replica's pods are down, creating an unsafe situation for manual operations. This incident is critical and requires immediate attention to resolve the issue and ensure the smooth functioning of the stateful applications.


Kubernetes Statefulset Replicas Monitoring Incident

A Kubernetes Pod Restarting Monitoring incident is triggered when a pod running on a Kubernetes cluster restarts multiple times within a certain time frame. This incident type is usually used to detect issues with the application or infrastructure running on the cluster, and can be caused by various factors such as resource constraints, misconfigurations, or bugs in the application code. The incident is typically resolved by identifying and addressing the underlying cause of the pod restarts.


Kubernetes Pod Restarting Monitoring

```shell
# Environment Variables

export NAMESPACE="PLACEHOLDER"

export POD_NAME="PLACEHOLDER"

export CONTAINER_NAME="PLACEHOLDER"

export RESOURCE_TYPE="PLACEHOLDER"

export RESOURCE_LIMIT="PLACEHOLDER"
```


### Get the list of namespaces in the Kubernetes cluster

```shell
kubectl get namespaces
```

### Get the list of pods in a specific namespace

```shell
kubectl get pods -n ${NAMESPACE}
```

### Get the details of a specific pod

```shell
kubectl describe pod ${POD_NAME} -n ${NAMESPACE}
```

### Get the list of events related to a specific pod

```shell
kubectl get events -n ${NAMESPACE} --field-selector involvedObject.name=${POD_NAME}
```

### Check the pod's YAML configuration file for any issues

```shell
kubectl get pod ${POD_NAME} -n ${NAMESPACE} -o yaml
```

### Check the status of the nodes in the Kubernetes cluster

```shell
kubectl get nodes
```

### Check the resource usage of the nodes in the Kubernetes cluster

```shell
kubectl top nodes
```

### Insufficient resources allocated to the Kubernetes cluster, such as CPU, memory, or storage.

```shell
#!/bin/bash

# Set variables

NAMESPACE=${NAMESPACE}

POD_NAME=${POD_NAME}

RESOURCE=${RESOURCE_TYPE} # e.g. cpu, memory, storage

# Get the resource limits and requests for the pod

LIMIT=$(kubectl describe pod $POD_NAME -n $NAMESPACE | grep "Limits" | awk '{print $2}')

REQUEST=$(kubectl describe pod $POD_NAME -n $NAMESPACE | grep "Requests" | awk '{print $2}')

# Remove the unit from the limit and request values

LIMIT=${LIMIT//Mi/}

LIMIT=${LIMIT//Gi/}

REQUEST=${REQUEST//Mi/}

REQUEST=${REQUEST//Gi/}

# Convert the limit and request values to integers

LIMIT=$(echo $LIMIT | awk '{print int($1)}')

REQUEST=$(echo $REQUEST | awk '{print int($1)}')

# Get the usage of the resource for the pod

USAGE=$(kubectl top pod $POD_NAME -n $NAMESPACE | awk '{print $3}')

USAGE=${USAGE//Mi/}

USAGE=${USAGE//Gi/}

USAGE=$(echo $USAGE | awk '{print int($1)}')

# Calculate the available resource for the pod

AVAILABLE=$((LIMIT - USAGE))

# Check if the available resource is less than the requested resource

if [ $AVAILABLE -lt $REQUEST ]; then

    echo "Insufficient $RESOURCE allocated to the Kubernetes cluster for pod $POD_NAME"

    echo "Limit: $LIMIT$RESOURCE, Request: $REQUEST$RESOURCE, Usage: $USAGE$RESOURCE, Available: $AVAILABLE$RESOURCE"

else

    echo "Sufficient $RESOURCE allocated to the Kubernetes cluster for pod $POD_NAME"

    echo "Limit: $LIMIT$RESOURCE, Request: $REQUEST$RESOURCE, Usage: $USAGE$RESOURCE, Available: $AVAILABLE$RESOURCE"

fi


```

### Misconfiguration of the pod specifications, such as incorrect image names or container ports, preventing the pods from starting.

```shell
#!/bin/bash

# Replace ${POD_NAME} and ${NAMESPACE} with the actual values

POD_NAME="${POD_NAME}"

NAMESPACE="${NAMESPACE}"

# Get the pod description and check the status

POD_DESC=$(kubectl describe pod $POD_NAME -n $NAMESPACE)

STATUS=$(echo "$POD_DESC" | awk '/Status:/{print $2}')

if [[ "$STATUS" == "Error" ]]; then

  # Check the container status and logs

  CONTAINER_STATUS=$(echo "$POD_DESC" | awk '/Container Status:/{print $3}')

  if [[ "$CONTAINER_STATUS" == "Error" ]]; then

    CONTAINER_NAME=$(echo "$POD_DESC" | awk '/Container ID:/{print $3}' | cut -d '/' -f 3)

    CONTAINER_LOGS=$(kubectl logs $POD_NAME -c $CONTAINER_NAME -n $NAMESPACE)

    echo "Container logs: $CONTAINER_LOGS"

  else

    echo "Container status: $CONTAINER_STATUS"

  fi

else

  echo "Pod status: $STATUS"

fi

```


### Check the resource allocation for the pods to ensure that they have sufficient resources to run. Increase the resource allocation if necessary.

```shell
#!/bin/bash

# Set parameters

NAMESPACE=${NAMESPACE}

POD_NAME=${POD_NAME}

RESOURCE=${RESOURCE_TYPE}

RESOURCE_LIMIT=${RESOURCE_LIMIT}

# Check current resource allocation

CURRENT_LIMIT=$(kubectl get pod $POD_NAME -n $NAMESPACE -o=jsonpath="{.spec.containers[0].resources.limits.$RESOURCE}")

echo "Current $RESOURCE limit for $POD_NAME is $CURRENT_LIMIT"

# Increase resource allocation if necessary

if [ "$CURRENT_LIMIT" -lt "$RESOURCE_LIMIT" ]; then

    kubectl patch pod $POD_NAME -n $NAMESPACE --type='json' -p='[{"op": "replace", "path": "/spec/containers/0/resources/limits/'$RESOURCE'", "value": "'$RESOURCE_LIMIT'"}]'

    echo "Increased $RESOURCE limit for $POD_NAME to $RESOURCE_LIMIT"

else

    echo "Current $RESOURCE limit for $POD_NAME is already greater than or equal to $RESOURCE_LIMIT"

fi

```


Kubernetes Pods Pending

Overview

Parameters

Debug

Get the list of namespaces in the Kubernetes cluster

Get the list of pods in a specific namespace

Get the details of a specific pod

Check the pod's YAML configuration file for any issues

Check the status of the nodes in the Kubernetes cluster

Check the resource usage of the nodes in the Kubernetes cluster

Insufficient resources allocated to the Kubernetes cluster, such as CPU, memory, or storage.

Misconfiguration of the pod specifications, such as incorrect image names or container ports, preventing the pods from starting.

Repair

Check the resource allocation for the pods to ensure that they have sufficient resources to run. Increase the resource allocation if necessary.

Learn more

Related Runbooks

Vault too many pending tokens incident on kubernetes

Vault cluster health incident on kubernetes

Kubernetes pods not starting - Deployment issue

Kubernetes Statefulset Replicas Monitoring Incident

Support