---
id: ad417674-4639-4938-8964-ea083a1b33ad
---
# Kafka node underutilization check incident.
---

This type of incident occurs when a node in the Kafka cluster is not being fully utilized. Kafka is a distributed streaming platform, and each node in the cluster is responsible for processing and storing a portion of the data. If a node is underutilized, it means that it is not processing as much data as it should be, which can lead to performance issues and potential data loss. Therefore, it is important to identify and address underutilized nodes in a Kafka cluster to ensure optimal performance and data reliability.

### Parameters
```shell
export KAFKA_MOUNT_POINT="PLACEHOLDER"

export KAFKA_NODE_IP="PLACEHOLDER"

export TOPIC_NAME="PLACEHOLDER"

export PORT="PLACEHOLDER"

export NEW_PARTITION_COUNT="PLACEHOLDER"
```

## Debug

### Check the status of Kafka node
```shell
systemctl status kafka
```

### Check Kafka node logs for any errors
```shell
journalctl -u kafka
```

### Check disk usage of Kafka node
```shell
df -hT ${KAFKA_MOUNT_POINT}
```

### Check CPU usage of Kafka node
```shell
top -b -n 1 | grep kafka
```

### Check memory usage of Kafka node
```shell
free -m
```

### Check network connectivity between Kafka nodes
```shell
ping ${KAFKA_NODE_IP}
```

### Check the replication factor of Kafka topic
```shell
kafka-topics --describe --topic ${TOPIC_NAME} --bootstrap-server ${KAFKA_NODE_IP}:${PORT}
```

### Check the number of partitions of Kafka topic
```shell
kafka-topics --describe --topic ${TOPIC_NAME} --bootstrap-server ${KAFKA_NODE_IP}:${PORT}
```

### Check the size of Kafka topic
```shell
kafka-run-class kafka.tools.GetOffsetShell --broker-list ${KAFKA_NODE_IP}:${PORT} --topic ${TOPIC_NAME} --time -1 --offsets 1 | awk -F ":" '{sum += $3} END {print sum}'
```

## Repair

### Rebalance the partitions across the Kafka nodes to ensure even distribution of the workload.
```shell


#!/bin/bash



# Rebalance the partitions across the Kafka nodes

kafka-topics.sh --bootstrap-server ${KAFKA_NODE_IP}:${PORT} --alter --topic ${TOPIC_NAME} --partitions ${NEW_PARTITION_COUNT}



echo "Partition rebalancing complete."


```


This type of incident occurs when a node in the Kafka cluster is not being fully utilized. Kafka is a distributed streaming platform, and each node in the cluster is responsible for processing and storing a portion of the data. If a node is underutilized, it means that it is not processing as much data as it should be, which can lead to performance issues and potential data loss. Therefore, it is important to identify and address underutilized nodes in a Kafka cluster to ensure optimal performance and data reliability.


This incident type refers to a situation where Spark jobs are failing due to resource contentions in the cluster. When multiple Spark jobs are trying to access the same resources or data at the same time, it can cause a bottleneck that leads to job failures. This can happen when the resources in the cluster are not properly allocated or when the number of jobs running simultaneously exceeds the cluster's capacity to handle them. The result is that Spark jobs fail, leading to disruptions in data processing and analysis.


Spark job failures due to cluster resource contentions.

This incident type refers to the detection of an unused topic partition in Kafka Platform, a popular data streaming platform. Unused partitions can increase broker overhead and reduce efficiency, so it is recommended to delete them. The incident is triggered by a test notification or a monitoring query that checks the value of relevant metrics. The incident is resolved once the partition is deleted or confirmed to be intentional.


Unused Topic Partition in Kafka

This incident type refers to a situation where there is an issue with the communication between Kafka brokers due to network partitioning. A network partition is a situation where a network is split into two or more parts, and communication between them is disrupted. This issue can lead to data inconsistency, loss of messages, delayed message delivery, and other related problems. As a result, it is important to quickly detect and resolve these incidents to prevent any downstream impact on the system.


Network Partitioning Between Kafka Brokers.

This incident type refers to an issue where the network bandwidth on one or more Kafka nodes becomes fully saturated, causing the node(s) to experience performance degradation or even failure. This can result in message delivery delays or loss, and can also impact other nodes that rely on the affected node(s) for message replication. The saturation can be caused by a variety of factors, such as increased message traffic, misconfiguration, or hardware issues.


Network Saturation Check for Kafka Nodes

This incident type refers to when a Kafka node runs out of storage space to store data. This can cause issues with data processing and storage, potentially leading to data loss and system downtime.


Kafka node insufficient storage check

```shell
export KAFKA_MOUNT_POINT="PLACEHOLDER"

export KAFKA_NODE_IP="PLACEHOLDER"

export TOPIC_NAME="PLACEHOLDER"

export PORT="PLACEHOLDER"

export NEW_PARTITION_COUNT="PLACEHOLDER"
```


### Check the status of Kafka node

```shell
systemctl status kafka
```

### Check Kafka node logs for any errors

```shell
journalctl -u kafka
```

### Check disk usage of Kafka node

```shell
df -hT ${KAFKA_MOUNT_POINT}
```

### Check CPU usage of Kafka node

```shell
top -b -n 1 | grep kafka
```

### Check memory usage of Kafka node

```shell
free -m
```

### Check network connectivity between Kafka nodes

```shell
ping ${KAFKA_NODE_IP}
```

### Check the replication factor of Kafka topic

```shell
kafka-topics --describe --topic ${TOPIC_NAME} --bootstrap-server ${KAFKA_NODE_IP}:${PORT}
```

### Check the number of partitions of Kafka topic

```shell
kafka-topics --describe --topic ${TOPIC_NAME} --bootstrap-server ${KAFKA_NODE_IP}:${PORT}
```

### Check the size of Kafka topic

```shell
kafka-run-class kafka.tools.GetOffsetShell --broker-list ${KAFKA_NODE_IP}:${PORT} --topic ${TOPIC_NAME} --time -1 --offsets 1 | awk -F ":" '{sum += $3} END {print sum}'
```


### Rebalance the partitions across the Kafka nodes to ensure even distribution of the workload.

```shell


#!/bin/bash



# Rebalance the partitions across the Kafka nodes

kafka-topics.sh --bootstrap-server ${KAFKA_NODE_IP}:${PORT} --alter --topic ${TOPIC_NAME} --partitions ${NEW_PARTITION_COUNT}



echo "Partition rebalancing complete."


```


Kafka node underutilization check incident.

Overview

Parameters

Debug

Check the status of Kafka node

Check Kafka node logs for any errors

Check disk usage of Kafka node

Check CPU usage of Kafka node

Check memory usage of Kafka node

Check network connectivity between Kafka nodes

Check the replication factor of Kafka topic

Check the number of partitions of Kafka topic

Check the size of Kafka topic

Repair

Rebalance the partitions across the Kafka nodes to ensure even distribution of the workload.

Learn more

Related Runbooks

Spark job failures due to cluster resource contentions.

Unused Topic Partition in Kafka

Network Partitioning Between Kafka Brokers.

Network Saturation Check for Kafka Nodes

Support