---
id: 53d83ae4-12a9-4ec9-9b4e-21a9bb55190b
---

# Kafka Broker Failure Causing Partition Unavailability
---

This incident type refers to a situation where the Kafka broker, which is responsible for managing and storing messages in a Kafka cluster, has failed. This failure results in the unavailability of one or more partitions, which are used to distribute messages across the cluster. As a result, messages cannot be sent or received, leading to disruptions in the system's operations. This type of incident requires immediate attention to restore the Kafka broker and ensure that messages can be processed as expected.

### Parameters
```shell
export KAFKA_BROKER="PLACEHOLDER"

export PORT="PLACEHOLDER"

export ZOOKEEPER_HOST="PLACEHOLDER"

export PATH_TO_BROKER_LOGS_DIRECTORY="PLACEHOLDER"

export NUMBER_OF_ADDITIONAL_BROKERS_TO_ADD="PLACEHOLDER"

export PATH_TO_KAFKA_INSTALLATION="PLACEHOLDER"
```

## Debug

### Check if Kafka broker is running
```shell
systemctl status kafka
```

### Check if all Kafka partitions are available
```shell
kafka-topics --bootstrap-server ${KAFKA_BROKER}:${PORT} --describe
```

### Check if there are any disk space issues
```shell
df -h
```

### Check if there are any issues with Zookeeper
```shell
zkCli.sh -server ${ZOOKEEPER_HOST}:${PORT} ls /brokers/ids
```

### Check if there are any errors in Kafka logs
```shell
journalctl -u kafka.service
```

### Check if there are any network issues between Kafka brokers
```shell
ping ${KAFKA_BROKER}
```

### Resource exhaustion due to high traffic or large message sizes
```shell


#!/bin/bash



# Define variables

KAFKA_HOME=${PATH_TO_KAFKA_INSTALLATION}

BROKER_LOG_DIR=${PATH_TO_BROKER_LOGS_DIRECTORY}



# Check CPU and memory usage

top -b -n1 | head -n 5



# Check disk usage

df -h



# Check Kafka broker logs for errors

tail -n 100 $BROKER_LOG_DIR/server.log | grep ERROR



# Check Kafka broker metrics for resource usage

$KAFKA_HOME/bin/kafka-run-class.sh kafka.tools.JmxTool --jmx-url service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi --object-name kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec --attributes Count,MeanRate,OneMinuteRate,FiveMinuteRate,FifteenMinuteRate


```

## Repair

### Increase the number of Kafka brokers in the cluster to ensure that there are enough replicas and that the cluster can handle the required load.
```shell


#!/bin/bash



# Set variables

KAFKA_HOME=${PATH_TO_KAFKA_INSTALLATION}

NUM_BROKERS=${NUMBER_OF_ADDITIONAL_BROKERS_TO_ADD}



# Stop Kafka server

${KAFKA_HOME}/bin/kafka-server-stop.sh



# Update server properties file to add new brokers

echo "num.network.threads=3" >> ${KAFKA_HOME}/config/server.properties

echo "num.io.threads=8" >> ${KAFKA_HOME}/config/server.properties

echo "socket.send.buffer.bytes=102400" >> ${KAFKA_HOME}/config/server.properties

echo "socket.receive.buffer.bytes=102400" >> ${KAFKA_HOME}/config/server.properties

echo "socket.request.max.bytes=104857600" >> ${KAFKA_HOME}/config/server.properties



# Add new brokers to the cluster

for i in $(seq 1 ${NUM_BROKERS}); do

    ${KAFKA_HOME}/bin/kafka-server-start.sh ${KAFKA_HOME}/config/server-${i}.properties

done



# Restart Kafka server

${KAFKA_HOME}/bin/kafka-server-start.sh ${KAFKA_HOME}/config/server.properties


```

This incident type refers to a situation where the Kafka broker, which is responsible for managing and storing messages in a Kafka cluster, has failed. This failure results in the unavailability of one or more partitions, which are used to distribute messages across the cluster. As a result, messages cannot be sent or received, leading to disruptions in the system's operations. This type of incident requires immediate attention to restore the Kafka broker and ensure that messages can be processed as expected.


This incident type refers to the detection of an unused topic partition in Kafka Platform, a popular data streaming platform. Unused partitions can increase broker overhead and reduce efficiency, so it is recommended to delete them. The incident is triggered by a test notification or a monitoring query that checks the value of relevant metrics. The incident is resolved once the partition is deleted or confirmed to be intentional.


Unused Topic Partition in Kafka

This incident type refers to a situation where there is an issue with the communication between Kafka brokers due to network partitioning. A network partition is a situation where a network is split into two or more parts, and communication between them is disrupted. This issue can lead to data inconsistency, loss of messages, delayed message delivery, and other related problems. As a result, it is important to quickly detect and resolve these incidents to prevent any downstream impact on the system.


Parameters

This type of incident occurs when the offset of a Kafka topic decreases unexpectedly. In Kafka, an offset is a unique identifier that represents the position of a message within a partition. When the offset decreases, it means that some messages have been lost or deleted, which can cause data inconsistencies or other issues within the system. This incident type is critical for Kafka-based systems as it affects data integrity and can lead to data loss.


Kafka topic offset decreased incident.

This incident type refers to when a Kafka node runs out of storage space to store data. This can cause issues with data processing and storage, potentially leading to data loss and system downtime.


Kafka node insufficient storage check

```shell
export KAFKA_BROKER="PLACEHOLDER"

export PORT="PLACEHOLDER"

export ZOOKEEPER_HOST="PLACEHOLDER"

export PATH_TO_BROKER_LOGS_DIRECTORY="PLACEHOLDER"

export NUMBER_OF_ADDITIONAL_BROKERS_TO_ADD="PLACEHOLDER"

export PATH_TO_KAFKA_INSTALLATION="PLACEHOLDER"
```


### Check if Kafka broker is running

```shell
systemctl status kafka
```

### Check if all Kafka partitions are available

```shell
kafka-topics --bootstrap-server ${KAFKA_BROKER}:${PORT} --describe
```

### Check if there are any disk space issues

```shell
df -h
```

### Check if there are any issues with Zookeeper

```shell
zkCli.sh -server ${ZOOKEEPER_HOST}:${PORT} ls /brokers/ids
```

### Check if there are any errors in Kafka logs

```shell
journalctl -u kafka.service
```

### Check if there are any network issues between Kafka brokers

```shell
ping ${KAFKA_BROKER}
```

### Resource exhaustion due to high traffic or large message sizes

```shell


#!/bin/bash



# Define variables

KAFKA_HOME=${PATH_TO_KAFKA_INSTALLATION}

BROKER_LOG_DIR=${PATH_TO_BROKER_LOGS_DIRECTORY}



# Check CPU and memory usage

top -b -n1 | head -n 5



# Check disk usage

df -h



# Check Kafka broker logs for errors

tail -n 100 $BROKER_LOG_DIR/server.log | grep ERROR



# Check Kafka broker metrics for resource usage

$KAFKA_HOME/bin/kafka-run-class.sh kafka.tools.JmxTool --jmx-url service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi --object-name kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec --attributes Count,MeanRate,OneMinuteRate,FiveMinuteRate,FifteenMinuteRate


```


### Increase the number of Kafka brokers in the cluster to ensure that there are enough replicas and that the cluster can handle the required load.

```shell


#!/bin/bash



# Set variables

KAFKA_HOME=${PATH_TO_KAFKA_INSTALLATION}

NUM_BROKERS=${NUMBER_OF_ADDITIONAL_BROKERS_TO_ADD}



# Stop Kafka server

${KAFKA_HOME}/bin/kafka-server-stop.sh



# Update server properties file to add new brokers

echo "num.network.threads=3" >> ${KAFKA_HOME}/config/server.properties

echo "num.io.threads=8" >> ${KAFKA_HOME}/config/server.properties

echo "socket.send.buffer.bytes=102400" >> ${KAFKA_HOME}/config/server.properties

echo "socket.receive.buffer.bytes=102400" >> ${KAFKA_HOME}/config/server.properties

echo "socket.request.max.bytes=104857600" >> ${KAFKA_HOME}/config/server.properties



# Add new brokers to the cluster

for i in $(seq 1 ${NUM_BROKERS}); do

    ${KAFKA_HOME}/bin/kafka-server-start.sh ${KAFKA_HOME}/config/server-${i}.properties

done



# Restart Kafka server

${KAFKA_HOME}/bin/kafka-server-start.sh ${KAFKA_HOME}/config/server.properties


```


Kafka Broker Failure Causing Partition Unavailability

Overview

Parameters

Debug

Check if Kafka broker is running

Check if all Kafka partitions are available

Check if there are any disk space issues

Check if there are any issues with Zookeeper

Check if there are any errors in Kafka logs

Check if there are any network issues between Kafka brokers

Resource exhaustion due to high traffic or large message sizes

Repair

Increase the number of Kafka brokers in the cluster to ensure that there are enough replicas and that the cluster can handle the required load.

Learn more

Related Runbooks

Unused Topic Partition in Kafka

Network Partitioning Between Kafka Brokers.

Parameters

Kafka topic offset decreased incident.

Support