---
id: 3334d488-d597-4cee-b89f-0f61d93f0cfa
---

# Kubernetes Deployment Pod Discrepancy
---

This incident type refers to an issue where the number of desired pods in a Kubernetes deployment does not match the number of actual pods running. This can cause problems with the functioning of the application or service running in the pods and may require investigation and resolution by the DevOps or engineering team.

### Parameters
```shell
# Environment Variables

export NAMESPACE="PLACEHOLDER"

export POD_NAME="PLACEHOLDER"

export CONTAINER_NAME="PLACEHOLDER"

export DEPLOYMENT_NAME="PLACEHOLDER"

export EXPECTED_NUMBER_OF_PODS="PLACEHOLDER"

export POD_LABEL="PLACEHOLDER"

export LOG_FILE="PLACEHOLDER"
```

## Debug

### Check the status of all deployments
```shell
kubectl get deployments
```

### Check the status of all pods in a specific namespace
```shell
kubectl get pods --namespace=${NAMESPACE}
```

### Get detailed information about a specific pod
```shell
kubectl describe pod ${POD_NAME}
```

### Check the logs of a specific pod
```shell
kubectl logs ${POD_NAME}
```

### Check the logs of a specific container in a pod
```shell
kubectl logs ${POD_NAME} ${CONTAINER_NAME}
```

### Check the events related to a specific pod
```shell
kubectl get events --field-selector involvedObject.name=${POD_NAME}
```

### Configuration error: The configuration of the Kubernetes deployment may have been changed without proper testing or validation. This could lead to an incorrect number of desired pods specified in the deployment configuration.
```shell
bash

#!/bin/bash



# Set variables

NAMESPACE=${NAMESPACE}

DEPLOYMENT=${DEPLOYMENT_NAME}

EXPECTED_PODS=${EXPECTED_NUMBER_OF_PODS}



# Get the current number of pods in the deployment

CURRENT_PODS=$(kubectl get deployment $DEPLOYMENT -n $NAMESPACE -o=jsonpath='{.status.readyReplicas}')



# Check if the current and expected number of pods match

if [ $CURRENT_PODS -ne $EXPECTED_PODS ]; then

  echo "Error: The current number of pods ($CURRENT_PODS) in deployment $DEPLOYMENT does not match the expected number ($EXPECTED_PODS)."

  

  # Get the deployment configuration

  kubectl get deployment $DEPLOYMENT -n $NAMESPACE -o yaml > deployment.yaml

  

  echo "Deployment configuration:"

  cat deployment.yaml

  

  # Check for any recent changes to the deployment

  kubectl rollout history deployment/$DEPLOYMENT -n $NAMESPACE

  

  echo "Recent changes to the deployment:"

  kubectl rollout history deployment/$DEPLOYMENT -n $NAMESPACE

  

  # Check for any errors or warnings in the deployment events

  kubectl get events -n $NAMESPACE --field-selector involvedObject.name=$DEPLOYMENT | grep -E 'Warning|Error'

  

else

  echo "The current number of pods in deployment $DEPLOYMENT matches the expected number."

fi


```

### Application-specific issues: Issues with the application or service running in the pods can cause them to
```shell


#!/bin/bash



# Set variables

NAMESPACE=${NAMESPACE}

POD_LABEL=${POD_LABEL}

CONTAINER_NAME=${CONTAINER_NAME}

LOG_FILE=${LOG_FILE}



# Get pod name

POD_NAME=$(kubectl get pods -n $NAMESPACE -l $POD_LABEL -o jsonpath='{.items[0].metadata.name}')



# Get logs for container

kubectl logs -n $NAMESPACE $POD_NAME $CONTAINER_NAME > $LOG_FILE



# Check for errors

if grep -q "ERROR" $LOG_FILE; then

  echo "Application-specific issue detected: Errors found in container logs."

  echo "Please review the logs in $LOG_FILE for more details."

else

  echo "No application-specific issues detected."

fi



# Clean up log file

rm $LOG_FILE


```

## Repair

### Scale up the number of pods in the deployment to match the desired number.
```shell


#!/bin/bash



# Set variables for the deployment and desired number of pods

deployment=${DEPLOYMENT_NAME}

desired_pods=${EXPECTED_NUMBER_OF_PODS}



# Scale up the deployment to the desired number of pods

kubectl scale deployment $deployment --replicas=$desired_pods



# Verify that the deployment has been scaled up

kubectl get deployment $deployment


```

### Check the logs of the pods to see if any errors are occurring that might be preventing them from running.
```shell


#!/bin/bash



# Set environment variables

NAMESPACE=${NAMESPACE}

POD_NAME=${POD_NAME}



# Get logs from the pod

kubectl logs $POD_NAME -n $NAMESPACE



# Check for errors in the logs

if grep -q "error" $LOG_FILE; then

    echo "Errors detected in the logs."

    # Restart the pod to see if that resolves the issue

    kubectl delete pod $POD_NAME -n $NAMESPACE

else

    echo "No errors detected in the logs."

fi


```

This incident type refers to an issue where the number of desired pods in a Kubernetes deployment does not match the number of actual pods running. This can cause problems with the functioning of the application or service running in the pods and may require investigation and resolution by the DevOps or engineering team.


The Vault cluster health incident is related to the health of a Vault cluster instance. This incident type is triggered when the cluster instance is not healthy and requires attention to ensure it is functioning properly. The incident typically involves evaluating the current state of the cluster instance, diagnosing the issue, and taking corrective action to restore the health of the instance.


Vault cluster health incident on kubernetes

This incident type involves an issue with Kubernetes deployments where the expected number of pods to run is not matching the actual number of pods running. This can lead to alerts being triggered and potential disruptions in the system.


Kubernetes pods not starting - Deployment issue

This incident type involves monitoring the replicas of a Kubernetes Statefulset, which is a type of workload in Kubernetes used for stateful applications. The incident is triggered when more than one replica's pods are down, creating an unsafe situation for manual operations. This incident is critical and requires immediate attention to resolve the issue and ensure the smooth functioning of the stateful applications.


Kubernetes Statefulset Replicas Monitoring Incident

A Kubernetes Replicaset Incomplete incident typically occurs when a specific number of pods that should be running are not, due to reasons such as failed pod initialization, unavailability of resources in the cluster, or inability to pull the image. This incident is usually triggered when the difference between desired and running pods is greater than zero, and it can be detected through monitoring tools like Datadog.


Kubernetes Replicaset Incomplete

Kubernetes Pods Pending incident indicates that one or more pods in a Kubernetes cluster are not running as expected and are in a pending state. This can happen due to various reasons such as resource constraints, scheduling issues, or network problems. This incident can impact the availability and performance of the application running on the Kubernetes cluster. It requires immediate attention to diagnose and resolve the underlying issue to ensure the pods are running as expected.


Kubernetes Pods Pending

```shell
# Environment Variables

export NAMESPACE="PLACEHOLDER"

export POD_NAME="PLACEHOLDER"

export CONTAINER_NAME="PLACEHOLDER"

export DEPLOYMENT_NAME="PLACEHOLDER"

export EXPECTED_NUMBER_OF_PODS="PLACEHOLDER"

export POD_LABEL="PLACEHOLDER"

export LOG_FILE="PLACEHOLDER"
```


### Check the status of all deployments

```shell
kubectl get deployments
```

### Check the status of all pods in a specific namespace

```shell
kubectl get pods --namespace=${NAMESPACE}
```

### Get detailed information about a specific pod

```shell
kubectl describe pod ${POD_NAME}
```

### Check the logs of a specific pod

```shell
kubectl logs ${POD_NAME}
```

### Check the logs of a specific container in a pod

```shell
kubectl logs ${POD_NAME} ${CONTAINER_NAME}
```

### Check the events related to a specific pod

```shell
kubectl get events --field-selector involvedObject.name=${POD_NAME}
```

### Configuration error: The configuration of the Kubernetes deployment may have been changed without proper testing or validation. This could lead to an incorrect number of desired pods specified in the deployment configuration.

```shell
bash

#!/bin/bash



# Set variables

NAMESPACE=${NAMESPACE}

DEPLOYMENT=${DEPLOYMENT_NAME}

EXPECTED_PODS=${EXPECTED_NUMBER_OF_PODS}



# Get the current number of pods in the deployment

CURRENT_PODS=$(kubectl get deployment $DEPLOYMENT -n $NAMESPACE -o=jsonpath='{.status.readyReplicas}')



# Check if the current and expected number of pods match

if [ $CURRENT_PODS -ne $EXPECTED_PODS ]; then

  echo "Error: The current number of pods ($CURRENT_PODS) in deployment $DEPLOYMENT does not match the expected number ($EXPECTED_PODS)."

  

  # Get the deployment configuration

  kubectl get deployment $DEPLOYMENT -n $NAMESPACE -o yaml > deployment.yaml

  

  echo "Deployment configuration:"

  cat deployment.yaml

  

  # Check for any recent changes to the deployment

  kubectl rollout history deployment/$DEPLOYMENT -n $NAMESPACE

  

  echo "Recent changes to the deployment:"

  kubectl rollout history deployment/$DEPLOYMENT -n $NAMESPACE

  

  # Check for any errors or warnings in the deployment events

  kubectl get events -n $NAMESPACE --field-selector involvedObject.name=$DEPLOYMENT | grep -E 'Warning|Error'

  

else

  echo "The current number of pods in deployment $DEPLOYMENT matches the expected number."

fi


```

### Application-specific issues: Issues with the application or service running in the pods can cause them to

```shell


#!/bin/bash



# Set variables

NAMESPACE=${NAMESPACE}

POD_LABEL=${POD_LABEL}

CONTAINER_NAME=${CONTAINER_NAME}

LOG_FILE=${LOG_FILE}



# Get pod name

POD_NAME=$(kubectl get pods -n $NAMESPACE -l $POD_LABEL -o jsonpath='{.items[0].metadata.name}')



# Get logs for container

kubectl logs -n $NAMESPACE $POD_NAME $CONTAINER_NAME > $LOG_FILE



# Check for errors

if grep -q "ERROR" $LOG_FILE; then

  echo "Application-specific issue detected: Errors found in container logs."

  echo "Please review the logs in $LOG_FILE for more details."

else

  echo "No application-specific issues detected."

fi



# Clean up log file

rm $LOG_FILE


```


### Scale up the number of pods in the deployment to match the desired number.

```shell


#!/bin/bash



# Set variables for the deployment and desired number of pods

deployment=${DEPLOYMENT_NAME}

desired_pods=${EXPECTED_NUMBER_OF_PODS}



# Scale up the deployment to the desired number of pods

kubectl scale deployment $deployment --replicas=$desired_pods



# Verify that the deployment has been scaled up

kubectl get deployment $deployment


```

### Check the logs of the pods to see if any errors are occurring that might be preventing them from running.

```shell


#!/bin/bash



# Set environment variables

NAMESPACE=${NAMESPACE}

POD_NAME=${POD_NAME}



# Get logs from the pod

kubectl logs $POD_NAME -n $NAMESPACE



# Check for errors in the logs

if grep -q "error" $LOG_FILE; then

    echo "Errors detected in the logs."

    # Restart the pod to see if that resolves the issue

    kubectl delete pod $POD_NAME -n $NAMESPACE

else

    echo "No errors detected in the logs."

fi


```


Kubernetes Deployment Pod Discrepancy

Overview

Parameters

Debug

Check the status of all deployments

Check the status of all pods in a specific namespace

Get detailed information about a specific pod

Check the logs of a specific pod

Check the logs of a specific container in a pod

Configuration error: The configuration of the Kubernetes deployment may have been changed without proper testing or validation. This could lead to an incorrect number of desired pods specified in the deployment configuration.

Application-specific issues: Issues with the application or service running in the pods can cause them to

Repair

Scale up the number of pods in the deployment to match the desired number.

Check the logs of the pods to see if any errors are occurring that might be preventing them from running.

Learn more

Related Runbooks

Vault cluster health incident on kubernetes

Kubernetes pods not starting - Deployment issue

Kubernetes Statefulset Replicas Monitoring Incident

Kubernetes Replicaset Incomplete

Support