---
id: 315dd128-1585-4bbf-9045-dde468b9dbdd
---

# Kafka In Sync Replica Alert for Under-Replicated Topics
---

This incident type refers to an alert triggered by the Kafka system when one or more topics have under-replicated data due to an insufficient number of in-sync replicas. In-sync replicas are replicas that are fully caught up with the leader replica and are considered reliable for serving data to consumers. When the number of in-sync replicas drops below a certain threshold, the Kafka system raises an alert to notify administrators to investigate and take corrective action to ensure data consistency and availability.

### Parameters
```shell
export TOPIC_NAME="PLACEHOLDER"

export ZOOKEEPER_URL="PLACEHOLDER"

export KAFKA_LOG_FILE="PLACEHOLDER"

export NEW_PARTITION_COUNT="PLACEHOLDER"

export KAFKA_HOME_DIRECTORY="PLACEHOLDER"

export NEW_REPLICATION_FACTOR="PLACEHOLDER"

export BROKER_HOST_PORT="PLACEHOLDER"
```

## Debug

### 1. Check the status of the Kafka brokers
```shell
systemctl status kafka
```

### 2. Verify that all Kafka brokers are up and running
```shell
kafka-topics.sh --describe --zookeeper ${ZOOKEEPER_URL} --topic ${TOPIC_NAME}
```

### 3. Check the in-sync replicas for each partition of the under-replicated topic
```shell
kafka-topics.sh --describe --zookeeper ${ZOOKEEPER_URL} --topic ${TOPIC_NAME} --under-replicated-partitions
```

### 4. Check the replication factor for the under-replicated topic
```shell
kafka-topics.sh --describe --zookeeper ${ZOOKEEPER_URL} --topic ${TOPIC_NAME} | grep ReplicationFactor
```

### 5. Check the ISR (in-sync replica) count for each partition of the under-replicated topic
```shell
kafka-topics.sh --describe --zookeeper ${ZOOKEEPER_URL} --topic ${TOPIC_NAME} | grep Isr
```

### 6. Check the logs for any errors or warnings related to the under-replicated topic
```shell
tail -f ${KAFKA_LOG_FILE}
```

### 7. Check the network connectivity between the Kafka brokers and Zookeeper
```shell
ping ${ZOOKEEPER_URL}
```

### The replication factor for the affected topics may have been set too low, increasing the likelihood of under-replication.
```shell


#!/bin/bash



# Set the broker host and port

broker=${BROKER_HOST_PORT}



# Set the topic name

topic=${TOPIC_NAME}



# Get the replication factor for the topic

replication_factor=$(kafka-topics --bootstrap-server $broker --describe --topic $topic | awk '/ReplicationFactor/ {print $3}')



# Compare the replication factor with a threshold value

threshold=2

if [ $replication_factor -lt $threshold ]; then

    echo "Replication factor for topic $topic is too low: $replication_factor (threshold is $threshold)"

    # Perform additional diagnostic checks or actions here

else

    echo "Replication factor for topic $topic is sufficient: $replication_factor"

fi


```

## Repair

### Increase the replication factor for the affected topic to ensure that there are more replicas of the partitions available.
```shell


#!/bin/bash



# Set the variables

KAFKA_HOME=${KAFKA_HOME_DIRECTORY}

TOPIC=${TOPIC_NAME}

REPLICATION_FACTOR=${NEW_REPLICATION_FACTOR}



# Increase the replication factor for the topic

$KAFKA_HOME/bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic $TOPIC --partitions ${NEW_PARTITION_COUNT} --replication-factor $REPLICATION_FACTOR



# Check if the replication factor has been updated

$KAFKA_HOME/bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic $TOPIC


```

This incident type refers to an alert triggered by the Kafka system when one or more topics have under-replicated data due to an insufficient number of in-sync replicas. In-sync replicas are replicas that are fully caught up with the leader replica and are considered reliable for serving data to consumers. When the number of in-sync replicas drops below a certain threshold, the Kafka system raises an alert to notify administrators to investigate and take corrective action to ensure data consistency and availability.


This incident type relates to the monitoring of Kubernetes deployments replica pods. It implies that there is an issue with the number of replica pods available as compared to the desired number. The incident might be triggered by a query alert monitor and might require immediate action to resolve the issue. The incident could impact the deployment of applications hosted on Kubernetes and might require troubleshooting and fixing the underlying issue.


Kubernetes Deployments Replica Pods Monitoring Incident

```shell
export KAFKA_BROKER_IP="PLACEHOLDER"

export PORT="PLACEHOLDER"

export TOPIC_NAME="PLACEHOLDER"

export ZOOKEEPER_IP="PLACEHOLDER"

export NEW_RETENTION_TIME="PLACEHOLDER"

export GROUP_NAME="PLACEHOLDER"

export PATH_TO_KAFKA_CONFIG_FILE="PLACEHOLDER"
```


Parameters

This type of incident occurs when the offset of a Kafka topic decreases unexpectedly. In Kafka, an offset is a unique identifier that represents the position of a message within a partition. When the offset decreases, it means that some messages have been lost or deleted, which can cause data inconsistencies or other issues within the system. This incident type is critical for Kafka-based systems as it affects data integrity and can lead to data loss.


Kafka topic offset decreased incident.

This type of incident occurs when a node in the Kafka cluster is not being fully utilized. Kafka is a distributed streaming platform, and each node in the cluster is responsible for processing and storing a portion of the data. If a node is underutilized, it means that it is not processing as much data as it should be, which can lead to performance issues and potential data loss. Therefore, it is important to identify and address underutilized nodes in a Kafka cluster to ensure optimal performance and data reliability.


Kafka node underutilization check incident.

This incident type refers to when a Kafka node runs out of storage space to store data. This can cause issues with data processing and storage, potentially leading to data loss and system downtime.


Kafka node insufficient storage check

```shell
export TOPIC_NAME="PLACEHOLDER"

export ZOOKEEPER_URL="PLACEHOLDER"

export KAFKA_LOG_FILE="PLACEHOLDER"

export NEW_PARTITION_COUNT="PLACEHOLDER"

export KAFKA_HOME_DIRECTORY="PLACEHOLDER"

export NEW_REPLICATION_FACTOR="PLACEHOLDER"

export BROKER_HOST_PORT="PLACEHOLDER"
```


### 1. Check the status of the Kafka brokers

```shell
systemctl status kafka
```

### 2. Verify that all Kafka brokers are up and running

```shell
kafka-topics.sh --describe --zookeeper ${ZOOKEEPER_URL} --topic ${TOPIC_NAME}
```

### 3. Check the in-sync replicas for each partition of the under-replicated topic

```shell
kafka-topics.sh --describe --zookeeper ${ZOOKEEPER_URL} --topic ${TOPIC_NAME} --under-replicated-partitions
```

### 4. Check the replication factor for the under-replicated topic

```shell
kafka-topics.sh --describe --zookeeper ${ZOOKEEPER_URL} --topic ${TOPIC_NAME} | grep ReplicationFactor
```

### 5. Check the ISR (in-sync replica) count for each partition of the under-replicated topic

```shell
kafka-topics.sh --describe --zookeeper ${ZOOKEEPER_URL} --topic ${TOPIC_NAME} | grep Isr
```

### 6. Check the logs for any errors or warnings related to the under-replicated topic

```shell
tail -f ${KAFKA_LOG_FILE}
```

### 7. Check the network connectivity between the Kafka brokers and Zookeeper

```shell
ping ${ZOOKEEPER_URL}
```

### The replication factor for the affected topics may have been set too low, increasing the likelihood of under-replication.

```shell


#!/bin/bash



# Set the broker host and port

broker=${BROKER_HOST_PORT}



# Set the topic name

topic=${TOPIC_NAME}



# Get the replication factor for the topic

replication_factor=$(kafka-topics --bootstrap-server $broker --describe --topic $topic | awk '/ReplicationFactor/ {print $3}')



# Compare the replication factor with a threshold value

threshold=2

if [ $replication_factor -lt $threshold ]; then

    echo "Replication factor for topic $topic is too low: $replication_factor (threshold is $threshold)"

    # Perform additional diagnostic checks or actions here

else

    echo "Replication factor for topic $topic is sufficient: $replication_factor"

fi


```


### Increase the replication factor for the affected topic to ensure that there are more replicas of the partitions available.

```shell


#!/bin/bash



# Set the variables

KAFKA_HOME=${KAFKA_HOME_DIRECTORY}

TOPIC=${TOPIC_NAME}

REPLICATION_FACTOR=${NEW_REPLICATION_FACTOR}



# Increase the replication factor for the topic

$KAFKA_HOME/bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic $TOPIC --partitions ${NEW_PARTITION_COUNT} --replication-factor $REPLICATION_FACTOR



# Check if the replication factor has been updated

$KAFKA_HOME/bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic $TOPIC


```


Kafka In Sync Replica Alert for Under-Replicated Topics

Overview

Parameters

Debug

1. Check the status of the Kafka brokers

2. Verify that all Kafka brokers are up and running

3. Check the in-sync replicas for each partition of the under-replicated topic

4. Check the replication factor for the under-replicated topic

5. Check the ISR (in-sync replica) count for each partition of the under-replicated topic

7. Check the network connectivity between the Kafka brokers and Zookeeper

The replication factor for the affected topics may have been set too low, increasing the likelihood of under-replication.

Repair

Increase the replication factor for the affected topic to ensure that there are more replicas of the partitions available.

Learn more

Related Runbooks

Kubernetes Deployments Replica Pods Monitoring Incident

Parameters

Kafka topic offset decreased incident.

Kafka node underutilization check incident.

Support