---
id: 5a54f4b1-1520-4fcb-bbed-4941c1dd0928
---

# Kubernetes Replicaset Incomplete
---

A Kubernetes Replicaset Incomplete incident typically occurs when a specific number of pods that should be running are not, due to reasons such as failed pod initialization, unavailability of resources in the cluster, or inability to pull the image. This incident is usually triggered when the difference between desired and running pods is greater than zero, and it can be detected through monitoring tools like Datadog.

### Parameters
```shell
# Environment Variables

export DEPLOYMENT_NAME="PLACEHOLDER"

export REPLICA_SET_NAME="PLACEHOLDER"

```

## Debug

### List all the deployments in the current namespace
```shell
kubectl get deployments
```

### Check the status of a specific deployment
```shell
kubectl describe deployment ${DEPLOYMENT_NAME}
```

### List all the replica sets in the current namespace
```shell
kubectl get rs
```

### Check the status of a specific replica set
```shell
kubectl describe rs ${REPLICA_SET_NAME}
```

### List all the events in the current namespace
```shell
kubectl get events
```
### Insufficient resources in the cluster, such as insufficient memory or CPU, can cause pods to fail to initialize or start.
```shell
#!/bin/bash

# 1. Get the list of nodes in the cluster

nodes=$(kubectl get nodes -ojsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}')

# 2. Loop through the nodes and check their resource allocation

for node in $nodes; do

  # Get the CPU and memory limits for the node

  cpu_limit=$(kubectl describe node $node | awk '/cpu:/ {print $3}')

  memory_limit=$(kubectl describe node $node | awk '/memory:/ {print $3}')

  # Get the total CPU and memory available for the node

  cpu_available=$(kubectl describe node $node | awk '/Allocatable.*cpu/ {print $2}')

  memory_available=$(kubectl describe node $node | awk '/Allocatable.*memory/ {print $2}')

  # Calculate the percentage of CPU and memory being used

  cpu_percent=$(echo "scale=2; (1 - ($cpu_available/$cpu_limit)) * 100" | bc)

  memory_percent=$(echo "scale=2; (1 - ($memory_available/$memory_limit)) * 100" | bc)

  # Check if the percentage of CPU or memory being used is too high

  if (( $(echo "$cpu_percent > 90" | bc -l) )) || (( $(echo "$memory_percent > 90" | bc -l) )); then

    echo "Node $node is running out of resources."

  fi

done
```
### Incorrect configuration of the Kubernetes Replicaset and its associated pods can cause the desired number of pods to be different from the actual number of running pods.
```shell
bash
#!/bin/bash
# Define the name of the replicaset to check

REPLICASET_NAME=${REPLICA_SET_NAME}

# Get the number of desired replicas for the replicaset

DESIRED_REPLICAS=$(kubectl get replicaset $REPLICASET_NAME -o=jsonpath='{.spec.replicas}')

# Get the number of current replicas for the replicaset

CURRENT_REPLICAS=$(kubectl get replicaset $REPLICASET_NAME -o=jsonpath='{.status.replicas}')

# Compare the number of desired and current replicas

if [ "$DESIRED_REPLICAS" -ne "$CURRENT_REPLICAS" ]; then

  echo "There is a mismatch between the desired ($DESIRED_REPLICAS) and current ($CURRENT_REPLICAS) number of replicas for the $REPLICASET_NAME replicaset."

  # Additional actions to take in case of a mismatch

else

  echo "The desired and current number of replicas for the $REPLICASET_NAME replicaset match."

fi

```

## Repair
### Adjusts the number of replicas in a deployment.
```shell
#!/bin/bash

# Set the deployment name and desired number of replicas
deployment_name=$DEPLOYMENT_NAME
desired_replicas="PLACEHOLDER"

# Scale the deployment to the desired number of replicas
kubectl scale deployment "$deployment_name" --replicas="$desired_replicas"

```
### Rolling restart deployment.
```shell
kubectl rollout restart deployment $DEPLOYMENT_NAME

```

A Kubernetes Replicaset Incomplete incident typically occurs when a specific number of pods that should be running are not, due to reasons such as failed pod initialization, unavailability of resources in the cluster, or inability to pull the image. This incident is usually triggered when the difference between desired and running pods is greater than zero, and it can be detected through monitoring tools like Datadog.


This incident type involves an issue with Kubernetes deployments where the expected number of pods to run is not matching the actual number of pods running. This can lead to alerts being triggered and potential disruptions in the system.


Kubernetes pods not starting - Deployment issue

This incident type involves monitoring the replicas of a Kubernetes Statefulset, which is a type of workload in Kubernetes used for stateful applications. The incident is triggered when more than one replica's pods are down, creating an unsafe situation for manual operations. This incident is critical and requires immediate attention to resolve the issue and ensure the smooth functioning of the stateful applications.


Kubernetes Statefulset Replicas Monitoring Incident

Kubernetes Pods Pending incident indicates that one or more pods in a Kubernetes cluster are not running as expected and are in a pending state. This can happen due to various reasons such as resource constraints, scheduling issues, or network problems. This incident can impact the availability and performance of the application running on the Kubernetes cluster. It requires immediate attention to diagnose and resolve the underlying issue to ensure the pods are running as expected.


Kubernetes Pods Pending

A Kubernetes Pod Restarting Monitoring incident is triggered when a pod running on a Kubernetes cluster restarts multiple times within a certain time frame. This incident type is usually used to detect issues with the application or infrastructure running on the cluster, and can be caused by various factors such as resource constraints, misconfigurations, or bugs in the application code. The incident is typically resolved by identifying and addressing the underlying cause of the pod restarts.


Kubernetes Pod Restarting Monitoring

A Kubernetes Pod ImagePullBackOff incident occurs when a pod in a Kubernetes cluster is unable to pull its container image. This can happen due to various reasons, such as incorrect image path or tag, or misconfigured image pulling credentials. This incident can cause the pod to fail to start and impact the availability of the application running in the pod. It requires investigation and resolution to ensure the pod can pull its container image and restart successfully.


Kubernetes Pod ImagePullBackOff Incident

```shell
# Environment Variables

export DEPLOYMENT_NAME="PLACEHOLDER"

export REPLICA_SET_NAME="PLACEHOLDER"

```


### List all the deployments in the current namespace

```shell
kubectl get deployments
```

### Check the status of a specific deployment

```shell
kubectl describe deployment ${DEPLOYMENT_NAME}
```

### List all the replica sets in the current namespace

```shell
kubectl get rs
```

### Check the status of a specific replica set

```shell
kubectl describe rs ${REPLICA_SET_NAME}
```

### List all the events in the current namespace

```shell
kubectl get events
```

### Insufficient resources in the cluster, such as insufficient memory or CPU, can cause pods to fail to initialize or start.

```shell
#!/bin/bash

# 1. Get the list of nodes in the cluster

nodes=$(kubectl get nodes -ojsonpath='{range .items[*]}{.metadata.name}{"\n"}{end}')

# 2. Loop through the nodes and check their resource allocation

for node in $nodes; do

  # Get the CPU and memory limits for the node

  cpu_limit=$(kubectl describe node $node | awk '/cpu:/ {print $3}')

  memory_limit=$(kubectl describe node $node | awk '/memory:/ {print $3}')

  # Get the total CPU and memory available for the node

  cpu_available=$(kubectl describe node $node | awk '/Allocatable.*cpu/ {print $2}')

  memory_available=$(kubectl describe node $node | awk '/Allocatable.*memory/ {print $2}')

  # Calculate the percentage of CPU and memory being used

  cpu_percent=$(echo "scale=2; (1 - ($cpu_available/$cpu_limit)) * 100" | bc)

  memory_percent=$(echo "scale=2; (1 - ($memory_available/$memory_limit)) * 100" | bc)

  # Check if the percentage of CPU or memory being used is too high

  if (( $(echo "$cpu_percent > 90" | bc -l) )) || (( $(echo "$memory_percent > 90" | bc -l) )); then

    echo "Node $node is running out of resources."

  fi

done
```

### Incorrect configuration of the Kubernetes Replicaset and its associated pods can cause the desired number of pods to be different from the actual number of running pods.

```shell
bash
#!/bin/bash
# Define the name of the replicaset to check

REPLICASET_NAME=${REPLICA_SET_NAME}

# Get the number of desired replicas for the replicaset

DESIRED_REPLICAS=$(kubectl get replicaset $REPLICASET_NAME -o=jsonpath='{.spec.replicas}')

# Get the number of current replicas for the replicaset

CURRENT_REPLICAS=$(kubectl get replicaset $REPLICASET_NAME -o=jsonpath='{.status.replicas}')

# Compare the number of desired and current replicas

if [ "$DESIRED_REPLICAS" -ne "$CURRENT_REPLICAS" ]; then

  echo "There is a mismatch between the desired ($DESIRED_REPLICAS) and current ($CURRENT_REPLICAS) number of replicas for the $REPLICASET_NAME replicaset."

  # Additional actions to take in case of a mismatch

else

  echo "The desired and current number of replicas for the $REPLICASET_NAME replicaset match."

fi

```


### Adjusts the number of replicas in a deployment.

```shell
#!/bin/bash

# Set the deployment name and desired number of replicas
deployment_name=$DEPLOYMENT_NAME
desired_replicas="PLACEHOLDER"

# Scale the deployment to the desired number of replicas
kubectl scale deployment "$deployment_name" --replicas="$desired_replicas"

```

### Rolling restart deployment.

```shell
kubectl rollout restart deployment $DEPLOYMENT_NAME

```


Kubernetes Replicaset Incomplete

Overview

Parameters

Debug

List all the deployments in the current namespace

Check the status of a specific deployment

List all the replica sets in the current namespace

Check the status of a specific replica set

List all the events in the current namespace

Insufficient resources in the cluster, such as insufficient memory or CPU, can cause pods to fail to initialize or start.

Incorrect configuration of the Kubernetes Replicaset and its associated pods can cause the desired number of pods to be different from the actual number of running pods.

Repair

Adjusts the number of replicas in a deployment.

Rolling restart deployment.

Learn more

Related Runbooks

Kubernetes pods not starting - Deployment issue

Kubernetes Statefulset Replicas Monitoring Incident

Kubernetes Pods Pending

Kubernetes Pod Restarting Monitoring

Support