---
id: 7e19ca51-a220-422c-a5e8-6a29ad3c0193
---
# Gray Disk on Kafka Node.
---

This incident type refers to the need to check for gray disks on a Kafka node. A gray disk is a disk that has failed or is failing, and it can cause data loss or interruption in the Kafka cluster. This check is important to ensure that the Kafka cluster is running smoothly and that data is being properly replicated across all nodes.

### Parameters
```shell
export DISK_NAME="PLACEHOLDER"

export TOPIC_NAME="PLACEHOLDER"

export ZOOKEEPER_HOST="PLACEHOLDER"

export PATH_TO_GRAY_DISK="PLACEHOLDER"

export PATH_TO_NEW_DISK="PLACEHOLDER"
```

## Debug

### Check if any disk is at critical threshold
```shell
df -h
```

### Check if there are any disk errors
```shell
dmesg | grep -i error
```

### Check if there are any disk failures
```shell
sudo smartctl -a /dev/${DISK_NAME}
```

### Check if there are any Kafka log errors
```shell
grep -i "error" /var/log/kafka/server.log
```

### Check the replication status of topics
```shell
kafka-topics.sh --describe --zookeeper ${ZOOKEEPER_HOST}:2181 --topic ${TOPIC_NAME}
```

### Check the health of the Kafka cluster
```shell
kafka-topics.sh --list --zookeeper ${ZOOKEEPER_HOST}:2181
```

## Repair

### Replace the gray disk with a new one.
```shell


#!/bin/bash



# Define variables

GRAY_DISK=${PATH_TO_GRAY_DISK}

NEW_DISK=${PATH_TO_NEW_DISK}



# Stop Kafka service

sudo systemctl stop kafka



# Copy data from gray disk to new disk

sudo rsync -avzh $GRAY_DISK/ $NEW_DISK/



# Update Kafka configuration files to use new disk

sudo sed -i "s|$GRAY_DISK|$NEW_DISK|g" /etc/kafka/server.properties



# Start Kafka service

sudo systemctl start kafka


```


This incident type refers to the need to check for gray disks on a Kafka node. A gray disk is a disk that has failed or is failing, and it can cause data loss or interruption in the Kafka cluster. This check is important to ensure that the Kafka cluster is running smoothly and that data is being properly replicated across all nodes.


This incident type typically occurs in distributed computing systems, where Spark tasks are experiencing high disk I/O and shuffle spills. Spark is a popular distributed computing engine that uses shuffle operations to move data between nodes in a cluster, which can sometimes result in performance issues due to spills. The spills occur when the data being shuffled exceeds the memory capacity allocated for the shuffle operations. This incident requires optimization of the shuffle operations to reduce spills and improve overall performance.


Spark tasks experiencing shuffle spills and high disk I/O.

A Host CPU high iowait incident occurs when the CPU iowait is greater than 10%, indicating that the system is disk or network bound. This can cause performance issues and may require investigation to identify the root cause and resolve the issue.


Host CPU high iowait incident.

Kubernetes is a popular container orchestration system used to manage and deploy containerized applications. Kubernetes nodes are the individual servers in a Kubernetes cluster that run the containers. Disk pressure is a condition where a node is using too much disk space or is using disk space too fast according to the thresholds set in the Kubernetes configuration. This condition can be caused by applications legitimately needing more space or an application misbehaving and filling up the disk prematurely in an unanticipated manner. It is important to monitor disk pressure as it can lead to performance issues, instability, or even downtime. When a Kubernetes node experiences disk pressure, it can trigger an incident that needs to be addressed to ensure the stability of the cluster.


Kubernetes Nodes with Disk Pressure

This incident type refers to a situation where there is an issue with the communication between Kafka brokers due to network partitioning. A network partition is a situation where a network is split into two or more parts, and communication between them is disrupted. This issue can lead to data inconsistency, loss of messages, delayed message delivery, and other related problems. As a result, it is important to quickly detect and resolve these incidents to prevent any downstream impact on the system.


Parameters

```shell
export DISK_NAME="PLACEHOLDER"

export TOPIC_NAME="PLACEHOLDER"

export ZOOKEEPER_HOST="PLACEHOLDER"

export PATH_TO_GRAY_DISK="PLACEHOLDER"

export PATH_TO_NEW_DISK="PLACEHOLDER"
```


### Check if any disk is at critical threshold

```shell
df -h
```

### Check if there are any disk errors

```shell
dmesg | grep -i error
```

### Check if there are any disk failures

```shell
sudo smartctl -a /dev/${DISK_NAME}
```

### Check if there are any Kafka log errors

```shell
grep -i "error" /var/log/kafka/server.log
```

### Check the replication status of topics

```shell
kafka-topics.sh --describe --zookeeper ${ZOOKEEPER_HOST}:2181 --topic ${TOPIC_NAME}
```

### Check the health of the Kafka cluster

```shell
kafka-topics.sh --list --zookeeper ${ZOOKEEPER_HOST}:2181
```


### Replace the gray disk with a new one.

```shell


#!/bin/bash



# Define variables

GRAY_DISK=${PATH_TO_GRAY_DISK}

NEW_DISK=${PATH_TO_NEW_DISK}



# Stop Kafka service

sudo systemctl stop kafka



# Copy data from gray disk to new disk

sudo rsync -avzh $GRAY_DISK/ $NEW_DISK/



# Update Kafka configuration files to use new disk

sudo sed -i "s|$GRAY_DISK|$NEW_DISK|g" /etc/kafka/server.properties



# Start Kafka service

sudo systemctl start kafka


```


Gray Disk on Kafka Node.

Overview

Parameters

Debug

Check if any disk is at critical threshold

Check if there are any disk errors

Check if there are any disk failures

Check if there are any Kafka log errors

Check the replication status of topics

Check the health of the Kafka cluster

Repair

Replace the gray disk with a new one.

Learn more

Related Runbooks

Spark tasks experiencing shuffle spills and high disk I/O.

Host CPU high iowait incident.

Kubernetes Nodes with Disk Pressure

Network Partitioning Between Kafka Brokers.

Support