---
id: cf7ecd9b-a1b3-4423-a8f5-aa3cd85e8aed
---

# Elasticsearch Initializing Shards Failure on Kubernetes.
---

This incident type refers to an issue with initializing shards in Elasticsearch, which can cause problems with accessing and managing data in the system. When this incident occurs, it can impact the overall performance and availability of Elasticsearch, potentially causing downtime or data loss. Immediate action is needed to resolve the issue and prevent any further impact on the system.

### Parameters
```shell
export ELASTICSEARCH_POD_NAME="PLACEHOLDER"

export ELASTICSEARCH_SERVICE="PLACEHOLDER"

export DEPLOYMENT_NAME="PLACEHOLDER"

export MEMORY_THRESHOLD="PLACEHOLDER"

export NEW_REPLICA_COUNT="PLACEHOLDER"

export CPU_THRESHOLD="PLACEHOLDER"
```

## Debug

### List all Elasticsearch pods in the current namespace
```shell
kubectl get pods -l app=elasticsearch
```

### Check the logs of a specific Elasticsearch pod for any errors related to shards initialization
```shell
kubectl logs ${ELASTICSEARCH_POD_NAME} | grep "initialization failed"
```

### Check the status of Elasticsearch shards initialization using the Elasticsearch API
```shell
curl -sS ${ELASTICSEARCH_SERVICE}:9200/_cluster/health?pretty=true | grep "initializing_shards"
```

### Check the Elasticsearch cluster status to see if any nodes have been removed or are not responding
```shell
curl -sS ${ELASTICSEARCH_SERVICE}:9200/_cluster/health?pretty=true | grep "status"
```

### Check the Elasticsearch logs for any errors related to shards initialization or cluster status
```shell
kubectl logs ${ELASTICSEARCH_POD_NAME} | grep "shards" && kubectl logs ${ELASTICSEARCH_POD_NAME} | grep "cluster"
```

### Check the Kubernetes events related to Elasticsearch pods and services to see if any errors have been reported
```shell
kubectl get events --sort-by=.metadata.creationTimestamp | grep "Elasticsearch"
```

### Describe the Elasticsearch service to see if it's pointing to the correct pods and endpoints
```shell
kubectl describe svc ${ELASTICSEARCH_SERVICE}
```

### Check the Elasticsearch configuration to see if any changes have been made recently that could affect shards initialization
```shell
kubectl exec ${ELASTICSEARCH_POD_NAME} -- cat /usr/share/elasticsearch/config/elasticsearch.yml
```

## Repair

### Check system resources: Elasticsearch relies heavily on system resources such as memory and CPU. If the system is running low on resources, it can impact the initialization of shards. Check the system resource utilization and allocate additional resources as needed.
```shell


#!/bin/bash



# Set variables

NAMESPACE=${ELASTICSEARCH_NAMESPACE}

POD=${ELASTICSEARCH_POD_NAME}

CONTAINER=${CONTAINER_NAME}



# Check system resource utilization

MEMORY_THRESHOLD=${MEMORY_THRESHOLD}

CPU_THRESHOLD=${CPU_THRESHOLD}



MEMORY_USAGE=$(kubectl exec -n $NAMESPACE $POD -c $CONTAINER -- bash -c "free | awk '/Mem:/ {print $3/$2*100}'")

CPU_USAGE=$(kubectl exec -n $NAMESPACE $POD -c $CONTAINER -- top -bn1 | awk '/Cpu$s$:/ {print $2}')



if (( $(echo "$MEMORY_USAGE > $MEMORY_THRESHOLD" | bc -l) )) || (( $(echo "$CPU_USAGE > $CPU_THRESHOLD" | bc -l) )); then

    # Allocate additional resources

    kubectl scale deployment ${DEPLOYMENT_NAME} --replicas=${NEW_REPLICA_COUNT} -n ${NAMESPACE}

fi


```

This incident type refers to an issue with initializing shards in Elasticsearch, which can cause problems with accessing and managing data in the system. When this incident occurs, it can impact the overall performance and availability of Elasticsearch, potentially causing downtime or data loss. Immediate action is needed to resolve the issue and prevent any further impact on the system.


This incident type involves an issue with Kubernetes deployments where the expected number of pods to run is not matching the actual number of pods running. This can lead to alerts being triggered and potential disruptions in the system.


Kubernetes pods not starting - Deployment issue

The incident type of "Kubernetes deployment with multiple restarts" indicates that a Kubernetes deployment has experienced multiple restarts within a certain timeframe, which is usually indicative of a problem. Kubernetes is a popular container orchestration platform that automates the deployment, scaling, and management of containerized applications. When a deployment experiences multiple restarts, it can impact the availability and performance of the application, and can be a sign of underlying issues that need to be addressed. This incident type is typically monitored and managed by DevOps teams responsible for ensuring the health and reliability of Kubernetes-based applications.


Kubernetes deployment with multiple restarts

This incident type relates to the monitoring of Kubernetes deployments replica pods. It implies that there is an issue with the number of replica pods available as compared to the desired number. The incident might be triggered by a query alert monitor and might require immediate action to resolve the issue. The incident could impact the deployment of applications hosted on Kubernetes and might require troubleshooting and fixing the underlying issue.


Kubernetes Deployments Replica Pods Monitoring Incident

This incident type occurs when the pods in a Kubernetes cluster fail to schedule, which prevents the application from scaling. Kubernetes is a container orchestration platform that automates the deployment, scaling, and management of containerized applications. Pods in Kubernetes are the smallest deployable units that can be created and managed. When pods do not schedule, it means that the Kubernetes scheduler is unable to find a node with enough resources to run the pod. This can cause issues with application performance and scalability.


Kubernetes - Pods not scheduled preventing application scaling.

This incident type relates to a failure of Kubernetes pods, which are a fundamental unit of deployment for containerized applications in Kubernetes. The failure could be related to any number of issues, such as a misconfiguration, a software bug, or a resource constraint. The incident may require immediate attention from a software engineer to resolve the issue and prevent further disruption to the application.


Kubernetes - Pods Failed

```shell
export ELASTICSEARCH_POD_NAME="PLACEHOLDER"

export ELASTICSEARCH_SERVICE="PLACEHOLDER"

export DEPLOYMENT_NAME="PLACEHOLDER"

export MEMORY_THRESHOLD="PLACEHOLDER"

export NEW_REPLICA_COUNT="PLACEHOLDER"

export CPU_THRESHOLD="PLACEHOLDER"
```


### List all Elasticsearch pods in the current namespace

```shell
kubectl get pods -l app=elasticsearch
```

### Check the logs of a specific Elasticsearch pod for any errors related to shards initialization

```shell
kubectl logs ${ELASTICSEARCH_POD_NAME} | grep "initialization failed"
```

### Check the status of Elasticsearch shards initialization using the Elasticsearch API

```shell
curl -sS ${ELASTICSEARCH_SERVICE}:9200/_cluster/health?pretty=true | grep "initializing_shards"
```

### Check the Elasticsearch cluster status to see if any nodes have been removed or are not responding

```shell
curl -sS ${ELASTICSEARCH_SERVICE}:9200/_cluster/health?pretty=true | grep "status"
```

### Check the Elasticsearch logs for any errors related to shards initialization or cluster status

```shell
kubectl logs ${ELASTICSEARCH_POD_NAME} | grep "shards" && kubectl logs ${ELASTICSEARCH_POD_NAME} | grep "cluster"
```

### Check the Kubernetes events related to Elasticsearch pods and services to see if any errors have been reported

```shell
kubectl get events --sort-by=.metadata.creationTimestamp | grep "Elasticsearch"
```

### Describe the Elasticsearch service to see if it's pointing to the correct pods and endpoints

```shell
kubectl describe svc ${ELASTICSEARCH_SERVICE}
```

### Check the Elasticsearch configuration to see if any changes have been made recently that could affect shards initialization

```shell
kubectl exec ${ELASTICSEARCH_POD_NAME} -- cat /usr/share/elasticsearch/config/elasticsearch.yml
```


### Check system resources: Elasticsearch relies heavily on system resources such as memory and CPU. If the system is running low on resources, it can impact the initialization of shards. Check the system resource utilization and allocate additional resources as needed.

```shell


#!/bin/bash



# Set variables

NAMESPACE=${ELASTICSEARCH_NAMESPACE}

POD=${ELASTICSEARCH_POD_NAME}

CONTAINER=${CONTAINER_NAME}



# Check system resource utilization

MEMORY_THRESHOLD=${MEMORY_THRESHOLD}

CPU_THRESHOLD=${CPU_THRESHOLD}



MEMORY_USAGE=$(kubectl exec -n $NAMESPACE $POD -c $CONTAINER -- bash -c "free | awk '/Mem:/ {print $3/$2*100}'")

CPU_USAGE=$(kubectl exec -n $NAMESPACE $POD -c $CONTAINER -- top -bn1 | awk '/Cpu$s$:/ {print $2}')



if (( $(echo "$MEMORY_USAGE > $MEMORY_THRESHOLD" | bc -l) )) || (( $(echo "$CPU_USAGE > $CPU_THRESHOLD" | bc -l) )); then

    # Allocate additional resources

    kubectl scale deployment ${DEPLOYMENT_NAME} --replicas=${NEW_REPLICA_COUNT} -n ${NAMESPACE}

fi


```


Elasticsearch Initializing Shards Failure on Kubernetes.

Overview

Parameters

Debug

List all Elasticsearch pods in the current namespace

Check the status of Elasticsearch shards initialization using the Elasticsearch API

Check the Elasticsearch cluster status to see if any nodes have been removed or are not responding

Describe the Elasticsearch service to see if it's pointing to the correct pods and endpoints

Check the Elasticsearch configuration to see if any changes have been made recently that could affect shards initialization

Repair

Check system resources: Elasticsearch relies heavily on system resources such as memory and CPU. If the system is running low on resources, it can impact the initialization of shards. Check the system resource utilization and allocate additional resources as needed.

Learn more

Related Runbooks

Kubernetes pods not starting - Deployment issue

Kubernetes deployment with multiple restarts

Kubernetes Deployments Replica Pods Monitoring Incident

Kubernetes - Pods not scheduled preventing application scaling.

Support