---
id: e75acf67-d9a7-492f-9008-245af316cadb
---

# Duplicate Messages in Kafka Topics
---

Duplicate messages in Kafka topics occur when one or more identical messages are written to the same Kafka topic. This can happen due to various reasons such as incorrect configuration of producers and consumers, network issues, or software bugs. Duplicate messages can cause issues in the downstream systems that consume data from Kafka topics, leading to data inconsistencies and other problems. It is important to monitor Kafka topics for duplicate messages and take appropriate measures to prevent them.

### Parameters
```shell
export ZOOKEEPER_PORT="PLACEHOLDER"

export ZOOKEEPER_HOST="PLACEHOLDER"

export TOPIC_NAME="PLACEHOLDER"

export CONSUMER_GROUP="PLACEHOLDER"

export KAFKA_BROKER_PORT="PLACEHOLDER"

export KAFKA_BROKER_HOST="PLACEHOLDER"

export KAFKA_LOG_DIRECTORY="PLACEHOLDER"
```

## Debug

### Check if Kafka is running
```shell
systemctl status kafka.service
```

### List topics in Kafka
```shell
kafka-topics.sh --list --zookeeper ${ZOOKEEPER_HOST}:${ZOOKEEPER_PORT}
```

### Check the number of replicas for a specific topic
```shell
kafka-topics.sh --describe --zookeeper ${ZOOKEEPER_HOST}:${ZOOKEEPER_PORT} --topic ${TOPIC_NAME}
```

### Check the number of partitions for a specific topic
```shell
kafka-topics.sh --describe --zookeeper ${ZOOKEEPER_HOST}:${ZOOKEEPER_PORT} --topic ${TOPIC_NAME}
```

### Check the consumer groups and their offsets for a specific topic
```shell
kafka-consumer-groups.sh --bootstrap-server ${KAFKA_BROKER_HOST}:${KAFKA_BROKER_PORT} --describe --group ${CONSUMER_GROUP} --topic ${TOPIC_NAME}
```

### Check the Kafka logs for errors or warnings
```shell
tail -f /var/log/kafka/kafka.log
```

### Multiple producers are publishing the same message to the same topic, resulting in duplicates.
```shell


#!/bin/bash



# Set the Kafka topic name

KAFKA_TOPIC=${TOPIC_NAME}



# Get the list of producers publishing to the topic

PRODUCERS=$(kafka-topics.sh --zookeeper ${ZOOKEEPER_HOST}:${ZOOKEEPER_PORT} --describe --topic $KAFKA_TOPIC | grep -oP 'ProducerId-\K\d+' | sort | uniq)



# Loop through each producer and check for duplicates

for PRODUCER_ID in $PRODUCERS

do

    DUPLICATE_MESSAGES=$(kafka-run-class.sh kafka.tools.DumpLogSegments --files ${KAFKA_LOG_DIRECTORY}/*.log --producer-threads 1 --deep-iteration --print-data-log --decoder avro --key-decoder avro --value-decoder avro --max-messages 10000000 --message-match "ProducerId=${PRODUCER_ID}" | grep -oP 'offset=\K\d+' | sort | uniq -d)

    

    if [ -z "$DUPLICATE_MESSAGES" ]

    then

        echo "No duplicates found for producer $PRODUCER_ID."

    else

        echo "Duplicates found for producer $PRODUCER_ID: $DUPLICATE_MESSAGES"

    fi

done


```

## Repair

### Implement message deduplication at the consumer end to prevent duplicate messages from being processed.
```shell


#!/bin/bash



# Define Kafka consumer properties

GROUP_ID="${CONSUMER_GROUP}"

TOPIC="${TOPIC_NAME}"



# Start consumer with deduplication

kafka-console-consumer.sh --bootstrap-server ${KAFKA_BROKER_HOST}:${KAFKA_BROKER_PORT} --group $GROUP_ID --topic $TOPIC --property print.key=true --property key.separator=: | awk '!x[$0]++'


```

Duplicate messages in Kafka topics occur when one or more identical messages are written to the same Kafka topic. This can happen due to various reasons such as incorrect configuration of producers and consumers, network issues, or software bugs. Duplicate messages can cause issues in the downstream systems that consume data from Kafka topics, leading to data inconsistencies and other problems. It is important to monitor Kafka topics for duplicate messages and take appropriate measures to prevent them.


This incident type refers to the detection of an unused topic partition in Kafka Platform, a popular data streaming platform. Unused partitions can increase broker overhead and reduce efficiency, so it is recommended to delete them. The incident is triggered by a test notification or a monitoring query that checks the value of relevant metrics. The incident is resolved once the partition is deleted or confirmed to be intentional.


Unused Topic Partition in Kafka

```shell
export KAFKA_BROKER_IP="PLACEHOLDER"

export PORT="PLACEHOLDER"

export TOPIC_NAME="PLACEHOLDER"

export ZOOKEEPER_IP="PLACEHOLDER"

export NEW_RETENTION_TIME="PLACEHOLDER"

export GROUP_NAME="PLACEHOLDER"

export PATH_TO_KAFKA_CONFIG_FILE="PLACEHOLDER"
```


Parameters

This type of incident occurs when the offset of a Kafka topic decreases unexpectedly. In Kafka, an offset is a unique identifier that represents the position of a message within a partition. When the offset decreases, it means that some messages have been lost or deleted, which can cause data inconsistencies or other issues within the system. This incident type is critical for Kafka-based systems as it affects data integrity and can lead to data loss.


Kafka topic offset decreased incident.

This incident type refers to when a Kafka node runs out of storage space to store data. This can cause issues with data processing and storage, potentially leading to data loss and system downtime.


Kafka node insufficient storage check

This incident type refers to a scenario where the liveness check for Kafka cannot reach/communicate with the host where the broker is running. This issue can arise due to various reasons such as network issues, broker failure, or configuration errors. This can result in the Kafka broker being unavailable or not responding to requests, leading to potential service disruptions or downtime.


Kafka liveness check failure.

```shell
export ZOOKEEPER_PORT="PLACEHOLDER"

export ZOOKEEPER_HOST="PLACEHOLDER"

export TOPIC_NAME="PLACEHOLDER"

export CONSUMER_GROUP="PLACEHOLDER"

export KAFKA_BROKER_PORT="PLACEHOLDER"

export KAFKA_BROKER_HOST="PLACEHOLDER"

export KAFKA_LOG_DIRECTORY="PLACEHOLDER"
```


### Check if Kafka is running

```shell
systemctl status kafka.service
```

### List topics in Kafka

```shell
kafka-topics.sh --list --zookeeper ${ZOOKEEPER_HOST}:${ZOOKEEPER_PORT}
```

### Check the number of replicas for a specific topic

```shell
kafka-topics.sh --describe --zookeeper ${ZOOKEEPER_HOST}:${ZOOKEEPER_PORT} --topic ${TOPIC_NAME}
```

### Check the number of partitions for a specific topic

```shell
kafka-topics.sh --describe --zookeeper ${ZOOKEEPER_HOST}:${ZOOKEEPER_PORT} --topic ${TOPIC_NAME}
```

### Check the consumer groups and their offsets for a specific topic

```shell
kafka-consumer-groups.sh --bootstrap-server ${KAFKA_BROKER_HOST}:${KAFKA_BROKER_PORT} --describe --group ${CONSUMER_GROUP} --topic ${TOPIC_NAME}
```

### Check the Kafka logs for errors or warnings

```shell
tail -f /var/log/kafka/kafka.log
```

### Multiple producers are publishing the same message to the same topic, resulting in duplicates.

```shell


#!/bin/bash



# Set the Kafka topic name

KAFKA_TOPIC=${TOPIC_NAME}



# Get the list of producers publishing to the topic

PRODUCERS=$(kafka-topics.sh --zookeeper ${ZOOKEEPER_HOST}:${ZOOKEEPER_PORT} --describe --topic $KAFKA_TOPIC | grep -oP 'ProducerId-\K\d+' | sort | uniq)



# Loop through each producer and check for duplicates

for PRODUCER_ID in $PRODUCERS

do

    DUPLICATE_MESSAGES=$(kafka-run-class.sh kafka.tools.DumpLogSegments --files ${KAFKA_LOG_DIRECTORY}/*.log --producer-threads 1 --deep-iteration --print-data-log --decoder avro --key-decoder avro --value-decoder avro --max-messages 10000000 --message-match "ProducerId=${PRODUCER_ID}" | grep -oP 'offset=\K\d+' | sort | uniq -d)

    

    if [ -z "$DUPLICATE_MESSAGES" ]

    then

        echo "No duplicates found for producer $PRODUCER_ID."

    else

        echo "Duplicates found for producer $PRODUCER_ID: $DUPLICATE_MESSAGES"

    fi

done


```


### Implement message deduplication at the consumer end to prevent duplicate messages from being processed.

```shell


#!/bin/bash



# Define Kafka consumer properties

GROUP_ID="${CONSUMER_GROUP}"

TOPIC="${TOPIC_NAME}"



# Start consumer with deduplication

kafka-console-consumer.sh --bootstrap-server ${KAFKA_BROKER_HOST}:${KAFKA_BROKER_PORT} --group $GROUP_ID --topic $TOPIC --property print.key=true --property key.separator=: | awk '!x[$0]++'


```


Duplicate Messages in Kafka Topics

Overview

Parameters

Debug

Check if Kafka is running

List topics in Kafka

Check the number of replicas for a specific topic

Check the number of partitions for a specific topic

Check the consumer groups and their offsets for a specific topic

Check the Kafka logs for errors or warnings

Multiple producers are publishing the same message to the same topic, resulting in duplicates.

Repair

Implement message deduplication at the consumer end to prevent duplicate messages from being processed.

Learn more

Related Runbooks

Unused Topic Partition in Kafka

Parameters

Kafka topic offset decreased incident.

Kafka node insufficient storage check

Support