---
id: b8e9b934-b6a3-494c-9826-7d2aa019a11b
---

# Kafka Consumer Group Lag Incident
---

The Kafka Consumer Group Lag incident refers to a situation where the lag time for a Kafka consumer group exceeds the expected threshold. This delay can result in delayed or lost data processing, leading to service degradation or failure.

### Parameters
```shell
export BROKER_HOSTNAME_PORT="PLACEHOLDER"

export CONSUMER_GROUP_NAME="PLACEHOLDER"

export KAFKA_TOPIC="PLACEHOLDER"

export PATH_TO_KAFKA_BIN="PLACEHOLDER"

export ZOOKEEPER_HOST="PLACEHOLDER"

export NUMBER_OF_NEW_CONSUMERS="PLACEHOLDER"
```

## Debug

### Find out the brokers in the Kafka cluster
```shell
kafka-topics.sh --bootstrap-server ${BROKER_HOSTNAME_PORT} --list
```

### Check the status of Kafka brokers and zookeeper
```shell
systemctl status kafka zookeeper
```

### Check the consumer group status
```shell
kafka-consumer-groups.sh --bootstrap-server ${BROKER_HOSTNAME_PORT} --describe --group ${CONSUMER_GROUP_NAME}
```

### Check the partition lag for the consumer group
```shell
kafka-consumer-groups.sh --bootstrap-server ${BROKER_HOSTNAME_PORT} --describe --group ${CONSUMER_GROUP_NAME} | awk 'NR > 1 {print $1"\t"$6}'
```

### Check topics metadata to see if any topics are unbalanced
```shell
kafka-topics.sh --bootstrap-server ${BROKER_HOSTNAME_PORT} --describe
```

### Check the disk space usage on the Kafka brokers
```shell
df -h
```

### Check the network connectivity between the brokers and the consumer group
```shell
ping ${BROKER_HOSTNAME}
```

### Check the Zookeeper logs for any errors
```shell
tail -f /var/log/zookeeper/zookeeper.log
```

### Check the Kafka broker logs for any errors
```shell
tail -f /var/log/kafka/server.log
```

### One or more Kafka brokers in the cluster are down or experiencing high latency, causing the consumer group to fall behind in processing messages.
```shell
bash

#!/bin/bash



# Set variables

ZK_HOST=${ZOOKEEPER_HOST}

KAFKA_BIN=${PATH_TO_KAFKA_BIN}

KAFKA_TOPIC=${KAFKA_TOPIC}

CONSUMER_GROUP=${CONSUMER_GROUP_NAME}



# Check zookeeper status

zk_status=$(echo ruok | nc "$ZK_HOST" 2181)

if [ "$zk_status" != "imok" ]; then

  echo "Zookeeper is not running."

  exit 1

fi



# Check Kafka brokers status

broker_list=$("$KAFKA_BIN"/zookeeper-shell.sh "$ZK_HOST":2181 <<< "ls /brokers/ids" | tail -n 1 | tr -d '[]' | tr ',' '\n')

for broker in $broker_list; do

  broker_status=$("$KAFKA_BIN"/kafka-broker-api-versions.sh --bootstrap-server "$ZK_HOST":9092 --broker-id "$broker" | grep -c "Metadata response received")

  if [ "$broker_status" -eq 0 ]; then

    echo "Broker $broker is not running or experiencing high latency."

    exit 1

  fi

done



# Check consumer group lag

consumer_group_lag=$("$KAFKA_BIN"/kafka-consumer-groups.sh --bootstrap-server "$ZK_HOST":9092 --describe --group "$CONSUMER_GROUP" | awk 'BEGIN {sum=0} {sum+=$3} END {print sum}')

if [ "$consumer_group_lag" -gt 0 ]; then

  echo "Consumer group $CONSUMER_GROUP is falling behind in processing messages. Total lag is $consumer_group_lag."

  exit 1

fi



echo "No issues found."

exit 0


```

## Repair

### Increase the number of consumers to handle the load and reduce the lag.
```shell


#!/bin/bash



# Set the Kafka topic and consumer group name

KAFKA_TOPIC=${KAFKA_TOPIC}

CONSUMER_GROUP=${CONSUMER_GROUP_NAME}



# Set the number of consumers to add

NEW_CONSUMERS=${NUMBER_OF_NEW_CONSUMERS}



# Get the current number of consumers in the group

CURRENT_CONSUMERS=$(kafka-consumer-groups.sh --bootstrap-server ${BROKER_HOSTNAME_PORT} --describe --group $CONSUMER_GROUP | grep "CONSUMER_COUNT" | awk '{print $2}')



# Calculate the new number of consumers after adding the new ones

TOTAL_CONSUMERS=$((CURRENT_CONSUMERS+NEW_CONSUMERS))



# Scale the Kafka consumer group to the new number of consumers

kafka-consumer-groups.sh --bootstrap-server ${BROKER_HOSTNAME_PORT} --group $CONSUMER_GROUP --reset-offsets --to-earliest --all-topics --execute

kafka-consumer-groups.sh --bootstrap-server ${BROKER_HOSTNAME_PORT} --group $CONSUMER_GROUP --topic $KAFKA_TOPIC --reset-offsets --shift-by -1 --execute

kafka-consumer-groups.sh --bootstrap-server ${BROKER_HOSTNAME_PORT} --group $CONSUMER_GROUP --describe

kafka-consumer-groups.sh --bootstrap-server ${BROKER_HOSTNAME_PORT} --group $CONSUMER_GROUP --scale-out $TOTAL_CONSUMERS



# Verify that the number of consumers has been increased

kafka-consumer-groups.sh --bootstrap-server ${BROKER_HOSTNAME_PORT} --describe --group $CONSUMER_GROUP


```

The Kafka Consumer Group Lag incident refers to a situation where the lag time for a Kafka consumer group exceeds the expected threshold. This delay can result in delayed or lost data processing, leading to service degradation or failure.


A MongoDB Replication Lag Incident occurs when there is a delay in the replication of data from the primary MongoDB instance to its secondary instances. This delay can cause data inconsistencies and affect the performance of the application. It is important to identify and resolve this incident as soon as possible to ensure that the application is functioning optimally.


MongoDB Replication Lag Incident

This incident type refers to a situation where there is an issue with the communication between Kafka brokers due to network partitioning. A network partition is a situation where a network is split into two or more parts, and communication between them is disrupted. This issue can lead to data inconsistency, loss of messages, delayed message delivery, and other related problems. As a result, it is important to quickly detect and resolve these incidents to prevent any downstream impact on the system.


Parameters

This incident type refers to an issue where the retention size for a Kafka topic is set too low. The retention size is the amount of time that Kafka retains messages in a topic before they are deleted. When the retention size is set too low, messages may be deleted before they can be consumed or processed, which can result in data loss or other issues. This can be caused by misconfiguration or changes in the data ingestion rate. It is important to monitor retention sizes and adjust them as necessary to prevent data loss and ensure smooth data processing.


Low Retention Size for Kafka Topic.

This type of incident occurs when the offset of a Kafka topic decreases unexpectedly. In Kafka, an offset is a unique identifier that represents the position of a message within a partition. When the offset decreases, it means that some messages have been lost or deleted, which can cause data inconsistencies or other issues within the system. This incident type is critical for Kafka-based systems as it affects data integrity and can lead to data loss.


Kafka topic offset decreased incident.

```shell
export BROKER_HOSTNAME_PORT="PLACEHOLDER"

export CONSUMER_GROUP_NAME="PLACEHOLDER"

export KAFKA_TOPIC="PLACEHOLDER"

export PATH_TO_KAFKA_BIN="PLACEHOLDER"

export ZOOKEEPER_HOST="PLACEHOLDER"

export NUMBER_OF_NEW_CONSUMERS="PLACEHOLDER"
```


### Find out the brokers in the Kafka cluster

```shell
kafka-topics.sh --bootstrap-server ${BROKER_HOSTNAME_PORT} --list
```

### Check the status of Kafka brokers and zookeeper

```shell
systemctl status kafka zookeeper
```

### Check the consumer group status

```shell
kafka-consumer-groups.sh --bootstrap-server ${BROKER_HOSTNAME_PORT} --describe --group ${CONSUMER_GROUP_NAME}
```

### Check the partition lag for the consumer group

```shell
kafka-consumer-groups.sh --bootstrap-server ${BROKER_HOSTNAME_PORT} --describe --group ${CONSUMER_GROUP_NAME} | awk 'NR > 1 {print $1"\t"$6}'
```

### Check topics metadata to see if any topics are unbalanced

```shell
kafka-topics.sh --bootstrap-server ${BROKER_HOSTNAME_PORT} --describe
```

### Check the disk space usage on the Kafka brokers

```shell
df -h
```

### Check the network connectivity between the brokers and the consumer group

```shell
ping ${BROKER_HOSTNAME}
```

### Check the Zookeeper logs for any errors

```shell
tail -f /var/log/zookeeper/zookeeper.log
```

### Check the Kafka broker logs for any errors

```shell
tail -f /var/log/kafka/server.log
```

### One or more Kafka brokers in the cluster are down or experiencing high latency, causing the consumer group to fall behind in processing messages.

```shell
bash

#!/bin/bash



# Set variables

ZK_HOST=${ZOOKEEPER_HOST}

KAFKA_BIN=${PATH_TO_KAFKA_BIN}

KAFKA_TOPIC=${KAFKA_TOPIC}

CONSUMER_GROUP=${CONSUMER_GROUP_NAME}



# Check zookeeper status

zk_status=$(echo ruok | nc "$ZK_HOST" 2181)

if [ "$zk_status" != "imok" ]; then

  echo "Zookeeper is not running."

  exit 1

fi



# Check Kafka brokers status

broker_list=$("$KAFKA_BIN"/zookeeper-shell.sh "$ZK_HOST":2181 <<< "ls /brokers/ids" | tail -n 1 | tr -d '[]' | tr ',' '\n')

for broker in $broker_list; do

  broker_status=$("$KAFKA_BIN"/kafka-broker-api-versions.sh --bootstrap-server "$ZK_HOST":9092 --broker-id "$broker" | grep -c "Metadata response received")

  if [ "$broker_status" -eq 0 ]; then

    echo "Broker $broker is not running or experiencing high latency."

    exit 1

  fi

done



# Check consumer group lag

consumer_group_lag=$("$KAFKA_BIN"/kafka-consumer-groups.sh --bootstrap-server "$ZK_HOST":9092 --describe --group "$CONSUMER_GROUP" | awk 'BEGIN {sum=0} {sum+=$3} END {print sum}')

if [ "$consumer_group_lag" -gt 0 ]; then

  echo "Consumer group $CONSUMER_GROUP is falling behind in processing messages. Total lag is $consumer_group_lag."

  exit 1

fi



echo "No issues found."

exit 0


```


### Increase the number of consumers to handle the load and reduce the lag.

```shell


#!/bin/bash



# Set the Kafka topic and consumer group name

KAFKA_TOPIC=${KAFKA_TOPIC}

CONSUMER_GROUP=${CONSUMER_GROUP_NAME}



# Set the number of consumers to add

NEW_CONSUMERS=${NUMBER_OF_NEW_CONSUMERS}



# Get the current number of consumers in the group

CURRENT_CONSUMERS=$(kafka-consumer-groups.sh --bootstrap-server ${BROKER_HOSTNAME_PORT} --describe --group $CONSUMER_GROUP | grep "CONSUMER_COUNT" | awk '{print $2}')



# Calculate the new number of consumers after adding the new ones

TOTAL_CONSUMERS=$((CURRENT_CONSUMERS+NEW_CONSUMERS))



# Scale the Kafka consumer group to the new number of consumers

kafka-consumer-groups.sh --bootstrap-server ${BROKER_HOSTNAME_PORT} --group $CONSUMER_GROUP --reset-offsets --to-earliest --all-topics --execute

kafka-consumer-groups.sh --bootstrap-server ${BROKER_HOSTNAME_PORT} --group $CONSUMER_GROUP --topic $KAFKA_TOPIC --reset-offsets --shift-by -1 --execute

kafka-consumer-groups.sh --bootstrap-server ${BROKER_HOSTNAME_PORT} --group $CONSUMER_GROUP --describe

kafka-consumer-groups.sh --bootstrap-server ${BROKER_HOSTNAME_PORT} --group $CONSUMER_GROUP --scale-out $TOTAL_CONSUMERS



# Verify that the number of consumers has been increased

kafka-consumer-groups.sh --bootstrap-server ${BROKER_HOSTNAME_PORT} --describe --group $CONSUMER_GROUP


```


Kafka Consumer Group Lag Incident

Overview

Parameters

Debug

Find out the brokers in the Kafka cluster

Check the status of Kafka brokers and zookeeper

Check the consumer group status

Check the partition lag for the consumer group

Check topics metadata to see if any topics are unbalanced

Check the disk space usage on the Kafka brokers

Check the network connectivity between the brokers and the consumer group

Check the Zookeeper logs for any errors

Check the Kafka broker logs for any errors

One or more Kafka brokers in the cluster are down or experiencing high latency, causing the consumer group to fall behind in processing messages.

Repair

Increase the number of consumers to handle the load and reduce the lag.

Learn more

Related Runbooks

MongoDB Replication Lag Incident

Network Partitioning Between Kafka Brokers.

Parameters

Low Retention Size for Kafka Topic.

Support