---
id: 7e6634d7-33bd-4ca6-b62b-a37fd1d048b8
---

# Kubernetes - Pod status not ready
---

This incident type occurs when a Kubernetes pod is not ready to accept traffic due to a configuration problem or a change in the pod specification. It can cause spikes in the number of unavailable pods and impact the performance of the system.

### Parameters
```shell
# Environment Variables
export POD_NAMESPACE="PLACEHOLDER"
export POD_NAME="PLACEHOLDER"
export CONTAINER_NAME="PLACEHOLDER"
export MAXIMUM_AMOUNT_OF_THE_RESOURCE="PLACEHOLDER"
export NAME_OF_THE_RESOURCE_TO_CHECK="PLACEHOLDER"
```

## Debug

### List all pods in namespace.
```shell
kubectl get pods -n ${POD_NAMESPACE}
```

### Check the status of a specific pod.
```shell
kubectl describe pod ${POD_NAME} -n ${POD_NAMESPACE}
```

### Get logs from a specific container  in pod.
```shell
kubectl logs ${POD_NAME} -c ${CONTAINER_NAME} -n ${POD_NAMESPACE}
```

### Check the events related to a specific pod.
```shell
kubectl get events --field-selector involvedObject.name=${POD_NAME} -n ${POD_NAMESPACE}
```
### Check the status of all persistent volume claims in namespace.

```shell
kubectl get pvc -n ${POD_NAMESPACE}
```

### Configuration errors or misconfigurations in the pod
```shell
#!/bin/bash

# Define variables
NAMESPACE=${POD_NAMESPACE}
POD=${POD_NAME}

# Check if kubectl is installed
if ! command -v kubectl &> /dev/null; then
    echo "kubectl could not be found. Please install kubectl and try again."
    exit 1
fi

# Check if the pod is running
POD_STATUS=$(kubectl get pods "$POD" -n "$NAMESPACE" -o jsonpath='{.status.phase}')
if [[ "$POD_STATUS" != "Running" ]]; then
    echo "The pod $POD is not running. Its status is $POD_STATUS."
    exit 1
fi

# Check if the pod is ready
POD_READY=$(kubectl get pods "$POD" -n "$NAMESPACE" -o jsonpath='{.status.containerStatuses[0].ready}')
if [[ "$POD_READY" != "true" ]]; then
    echo "The pod $POD is not ready. Its readiness status is $POD_READY."
    exit 1
fi

# Check if the pod configuration is valid
POD_CONFIG=$(kubectl validate pod "$POD" -n "$NAMESPACE" 2>&1)
if [[ "$POD_CONFIG" != "pod \"$POD\" successfully validated" ]]; then
    echo "The pod $POD has a configuration error: $POD_CONFIG"
    exit 1
fi

echo "The pod $POD in namespace $NAMESPACE is running and ready with a valid configuration."
exit 0

```
---

## Repair
---
### Increase the resources allocated to the pod if it is running out of memory or CPU.
```shell

#!/bin/bash

# Define variables
POD=${POD_NAME}
NAMESPACE=${POD_NAMESPACE}
RESOURCE_NAME=${NAME_OF_THE_RESOURCE_TO_CHECK} # e.g. "cpu", "memory"
LIMIT=${MAXIMUM_AMOUNT_OF_THE_RESOURCE} # e.g. "500Mi", "2Gi", "1.5"

# Check the resource usage of the pod
POD_USAGE=$(kubectl top pods $POD -n $NAMESPACE --containers | awk '/'$POD'/{getline; print $3}')

# Check if the pod is running out of the specified resource
if [[ $(echo "$POD_USAGE $LIMIT" | awk '{print ($1/$2)*100}') -ge 90 ]]; then
    # If the pod is running out of resources, increase the limit for the specified resource
    kubectl set resources pod $POD -n $NAMESPACE --containers=<> --${RESOURCE_NAME}=${LIMIT}
    echo "Increased the resources allocated to $POD in $NAMESPACE"
else
    echo "$POD is not running out of $RESOURCE_NAME"
fi

```
---


This incident type occurs when a Kubernetes pod is not ready to accept traffic due to a configuration problem or a change in the pod specification. It can cause spikes in the number of unavailable pods and impact the performance of the system.


The Vault cluster health incident is related to the health of a Vault cluster instance. This incident type is triggered when the cluster instance is not healthy and requires attention to ensure it is functioning properly. The incident typically involves evaluating the current state of the cluster instance, diagnosing the issue, and taking corrective action to restore the health of the instance.


Vault cluster health incident on kubernetes

This incident type refers to a situation where the MySQL service running on a pod becomes unresponsive and is unable to process new read and write requests. The incident could be caused by a database lock, which prevents other processes from accessing the database. The exact cause of the lock situation may need to be determined to resolve the issue.


Unresponsive MySQL service on pod with database lock.

This incident type involves an issue with Kubernetes deployments where the expected number of pods to run is not matching the actual number of pods running. This can lead to alerts being triggered and potential disruptions in the system.


Kubernetes pods not starting - Deployment issue

This incident type involves monitoring the replicas of a Kubernetes Statefulset, which is a type of workload in Kubernetes used for stateful applications. The incident is triggered when more than one replica's pods are down, creating an unsafe situation for manual operations. This incident is critical and requires immediate attention to resolve the issue and ensure the smooth functioning of the stateful applications.


Kubernetes Statefulset Replicas Monitoring Incident

A Kubernetes Replicaset Incomplete incident typically occurs when a specific number of pods that should be running are not, due to reasons such as failed pod initialization, unavailability of resources in the cluster, or inability to pull the image. This incident is usually triggered when the difference between desired and running pods is greater than zero, and it can be detected through monitoring tools like Datadog.


Kubernetes Replicaset Incomplete

```shell
# Environment Variables
export POD_NAMESPACE="PLACEHOLDER"
export POD_NAME="PLACEHOLDER"
export CONTAINER_NAME="PLACEHOLDER"
export MAXIMUM_AMOUNT_OF_THE_RESOURCE="PLACEHOLDER"
export NAME_OF_THE_RESOURCE_TO_CHECK="PLACEHOLDER"
```


### List all pods in namespace.

```shell
kubectl get pods -n ${POD_NAMESPACE}
```

### Check the status of a specific pod.

```shell
kubectl describe pod ${POD_NAME} -n ${POD_NAMESPACE}
```

### Get logs from a specific container  in pod.

```shell
kubectl logs ${POD_NAME} -c ${CONTAINER_NAME} -n ${POD_NAMESPACE}
```

### Check the events related to a specific pod.

```shell
kubectl get events --field-selector involvedObject.name=${POD_NAME} -n ${POD_NAMESPACE}
```

### Check the status of all persistent volume claims in namespace.

```shell
kubectl get pvc -n ${POD_NAMESPACE}
```

### Configuration errors or misconfigurations in the pod

```shell
#!/bin/bash

# Define variables
NAMESPACE=${POD_NAMESPACE}
POD=${POD_NAME}

# Check if kubectl is installed
if ! command -v kubectl &> /dev/null; then
    echo "kubectl could not be found. Please install kubectl and try again."
    exit 1
fi

# Check if the pod is running
POD_STATUS=$(kubectl get pods "$POD" -n "$NAMESPACE" -o jsonpath='{.status.phase}')
if [[ "$POD_STATUS" != "Running" ]]; then
    echo "The pod $POD is not running. Its status is $POD_STATUS."
    exit 1
fi

# Check if the pod is ready
POD_READY=$(kubectl get pods "$POD" -n "$NAMESPACE" -o jsonpath='{.status.containerStatuses[0].ready}')
if [[ "$POD_READY" != "true" ]]; then
    echo "The pod $POD is not ready. Its readiness status is $POD_READY."
    exit 1
fi

# Check if the pod configuration is valid
POD_CONFIG=$(kubectl validate pod "$POD" -n "$NAMESPACE" 2>&1)
if [[ "$POD_CONFIG" != "pod \"$POD\" successfully validated" ]]; then
    echo "The pod $POD has a configuration error: $POD_CONFIG"
    exit 1
fi

echo "The pod $POD in namespace $NAMESPACE is running and ready with a valid configuration."
exit 0

```


### Increase the resources allocated to the pod if it is running out of memory or CPU.

```shell

#!/bin/bash

# Define variables
POD=${POD_NAME}
NAMESPACE=${POD_NAMESPACE}
RESOURCE_NAME=${NAME_OF_THE_RESOURCE_TO_CHECK} # e.g. "cpu", "memory"
LIMIT=${MAXIMUM_AMOUNT_OF_THE_RESOURCE} # e.g. "500Mi", "2Gi", "1.5"

# Check the resource usage of the pod
POD_USAGE=$(kubectl top pods $POD -n $NAMESPACE --containers | awk '/'$POD'/{getline; print $3}')

# Check if the pod is running out of the specified resource
if [[ $(echo "$POD_USAGE $LIMIT" | awk '{print ($1/$2)*100}') -ge 90 ]]; then
    # If the pod is running out of resources, increase the limit for the specified resource
    kubectl set resources pod $POD -n $NAMESPACE --containers=<> --${RESOURCE_NAME}=${LIMIT}
    echo "Increased the resources allocated to $POD in $NAMESPACE"
else
    echo "$POD is not running out of $RESOURCE_NAME"
fi

```


Kubernetes - Pod status not ready

Overview

Parameters

Debug

List all pods in namespace.

Check the status of a specific pod.

Get logs from a specific container in pod.

Check the status of all persistent volume claims in namespace.

Configuration errors or misconfigurations in the pod

Repair

Increase the resources allocated to the pod if it is running out of memory or CPU.

Learn more

Related Runbooks

Vault cluster health incident on kubernetes

Unresponsive MySQL service on pod with database lock.

Kubernetes pods not starting - Deployment issue

Kubernetes Statefulset Replicas Monitoring Incident

Support