---
id: 4cbc7672-ea55-4923-97f3-d03fe092f0f2
---
# Inconsistent Kafka Broker Settings.
---

This incident type refers to the situation where there are inconsistencies in the settings of brokers across a Kafka cluster. Kafka is a distributed streaming platform that operates in a cluster of servers called brokers. These brokers work together to maintain and replicate the data stored in Kafka topics. Inconsistencies in broker settings can lead to data loss, reduced performance, and other operational issues. Therefore, it is crucial to ensure that all Kafka brokers have consistent settings.

### Parameters
```shell
export PATH_TO_KAFKA_BIN="PLACEHOLDER"
export KAFKA_HOME="PLACEHOLDER"
export BROKER_CONFIG_FILE="PLACEHOLDER"
export ZK_PORT="PLACEHOLDER"
export ZK_HOST="PLACEHOLDER"
export BROKER_PORT="PLACEHOLDER"
export BROKER_HOST="PLACEHOLDER"
```

## Debug

### Check the number of brokers on the Kafka cluster
```shell
${PATH_TO_KAFKA_BIN}/kafka-topics.sh --zookeeper ${ZK_HOST}:${ZK_PORT} --list
```

### Check the status of each broker in the Kafka cluster
```shell
${PATH_TO_KAFKA_BIN}/kafka-broker-api-versions.sh --bootstrap-server ${BROKER_HOST}:${BROKER_PORT}
```

### Check the setting values on each broker in the Kafka cluster
```shell
${PATH_TO_KAFKA_BIN}/kafka-configs.sh --bootstrap-server ${BROKER_HOST}:${BROKER_PORT} --entity-type brokers --describe
```

### Check the consistency of the broker settings across the Kafka cluster
```shell
${PATH_TO_KAFKA_BIN}/kafka-configs.sh --bootstrap-server ${BROKER_HOST}:${BROKER_PORT} --entity-type brokers --describe | awk '{print $2}' | uniq -c
```
## Repair

### Update the configuration settings for affected Kafka brokers to ensure consistency across the cluster.
```shell
bash

#!/bin/bash



# Set variables for Kafka cluster configuration

KAFKA_HOME=${PATH_TO_KAFKA_HOME_DIRECTORY}

BROKER_CONFIG_FILE=${PATH_TO_KAFKA_BROKER_CONFIGURATION_FILE}



# Loop through the list of Kafka brokers

for BROKER in ${LIST_OF_AFFECTED_KAFKA_BROKER_IDS}

do

  # Update the broker configuration to match the rest of the cluster

  sed -i 's/${INCONSISTENT_SETTING}/${CONSISTENT_SETTING}/g' $KAFKA_HOME/config/$BROKER_CONFIG_FILE

done


```

This incident type refers to the situation where there are inconsistencies in the settings of brokers across a Kafka cluster. Kafka is a distributed streaming platform that operates in a cluster of servers called brokers. These brokers work together to maintain and replicate the data stored in Kafka topics. Inconsistencies in broker settings can lead to data loss, reduced performance, and other operational issues. Therefore, it is crucial to ensure that all Kafka brokers have consistent settings.


This incident type refers to a situation where there is an issue with the communication between Kafka brokers due to network partitioning. A network partition is a situation where a network is split into two or more parts, and communication between them is disrupted. This issue can lead to data inconsistency, loss of messages, delayed message delivery, and other related problems. As a result, it is important to quickly detect and resolve these incidents to prevent any downstream impact on the system.


Network Partitioning Between Kafka Brokers.

This type of incident occurs when the offset of a Kafka topic decreases unexpectedly. In Kafka, an offset is a unique identifier that represents the position of a message within a partition. When the offset decreases, it means that some messages have been lost or deleted, which can cause data inconsistencies or other issues within the system. This incident type is critical for Kafka-based systems as it affects data integrity and can lead to data loss.


Kafka topic offset decreased incident.

This incident type refers to when a Kafka node runs out of storage space to store data. This can cause issues with data processing and storage, potentially leading to data loss and system downtime.


Kafka node insufficient storage check

This incident type refers to a scenario where the liveness check for Kafka cannot reach/communicate with the host where the broker is running. This issue can arise due to various reasons such as network issues, broker failure, or configuration errors. This can result in the Kafka broker being unavailable or not responding to requests, leading to potential service disruptions or downtime.


Kafka liveness check failure.

In a Kafka cluster, replication factor is the number of copies of a topic's partitions across different brokers. The replication factor is set at the time of topic creation and cannot be changed once the topic is created. In case the replication factor is less than the minimum required number, there is a risk of data loss if one or more brokers go down. This incident type involves checking the replication factor of Kafka clusters to ensure that the number of replicas is sufficient, and that data is safe in case of a broker failure.


Kafka Cluster Replication Factor Check

```shell
export PATH_TO_KAFKA_BIN="PLACEHOLDER"
export KAFKA_HOME="PLACEHOLDER"
export BROKER_CONFIG_FILE="PLACEHOLDER"
export ZK_PORT="PLACEHOLDER"
export ZK_HOST="PLACEHOLDER"
export BROKER_PORT="PLACEHOLDER"
export BROKER_HOST="PLACEHOLDER"
```


### Check the number of brokers on the Kafka cluster

```shell
${PATH_TO_KAFKA_BIN}/kafka-topics.sh --zookeeper ${ZK_HOST}:${ZK_PORT} --list
```

### Check the status of each broker in the Kafka cluster

```shell
${PATH_TO_KAFKA_BIN}/kafka-broker-api-versions.sh --bootstrap-server ${BROKER_HOST}:${BROKER_PORT}
```

### Check the setting values on each broker in the Kafka cluster

```shell
${PATH_TO_KAFKA_BIN}/kafka-configs.sh --bootstrap-server ${BROKER_HOST}:${BROKER_PORT} --entity-type brokers --describe
```

### Check the consistency of the broker settings across the Kafka cluster

```shell
${PATH_TO_KAFKA_BIN}/kafka-configs.sh --bootstrap-server ${BROKER_HOST}:${BROKER_PORT} --entity-type brokers --describe | awk '{print $2}' | uniq -c
```


### Update the configuration settings for affected Kafka brokers to ensure consistency across the cluster.

```shell
bash

#!/bin/bash



# Set variables for Kafka cluster configuration

KAFKA_HOME=${PATH_TO_KAFKA_HOME_DIRECTORY}

BROKER_CONFIG_FILE=${PATH_TO_KAFKA_BROKER_CONFIGURATION_FILE}



# Loop through the list of Kafka brokers

for BROKER in ${LIST_OF_AFFECTED_KAFKA_BROKER_IDS}

do

  # Update the broker configuration to match the rest of the cluster

  sed -i 's/${INCONSISTENT_SETTING}/${CONSISTENT_SETTING}/g' $KAFKA_HOME/config/$BROKER_CONFIG_FILE

done


```


Inconsistent Kafka Broker Settings.

Overview

Parameters

Debug

Check the number of brokers on the Kafka cluster

Check the status of each broker in the Kafka cluster

Check the setting values on each broker in the Kafka cluster

Check the consistency of the broker settings across the Kafka cluster

Repair

Update the configuration settings for affected Kafka brokers to ensure consistency across the cluster.

Learn more

Related Runbooks

Network Partitioning Between Kafka Brokers.

Kafka topic offset decreased incident.

Kafka node insufficient storage check

Kafka liveness check failure.

Support