---
id: 17ae76d5-d7f1-4206-939e-ada4b002c7df
---
# Low Retention Size for Kafka Topic.
---

This incident type refers to an issue where the retention size for a Kafka topic is set too low. The retention size is the amount of time that Kafka retains messages in a topic before they are deleted. When the retention size is set too low, messages may be deleted before they can be consumed or processed, which can result in data loss or other issues. This can be caused by misconfiguration or changes in the data ingestion rate. It is important to monitor retention sizes and adjust them as necessary to prevent data loss and ensure smooth data processing.

### Parameters
```shell
export KAFKA_BROKER_IP="PLACEHOLDER"

export PORT="PLACEHOLDER"

export TOPIC_NAME="PLACEHOLDER"

export ZOOKEEPER_IP="PLACEHOLDER"

export NEW_RETENTION_SIZE="PLACEHOLDER"

export GROUP_NAME="PLACEHOLDER"

export PATH_TO_KAFKA_CONFIG_FILE="PLACEHOLDER"
```

## Debug

### Connect to the Kafka broker
```shell
kafka-console-consumer.sh --bootstrap-server ${KAFKA_BROKER_IP}:${PORT} --topic ${TOPIC_NAME} --from-beginning
```

### View the current retention size for the topic
```shell
kafka-topics.sh --describe --zookeeper ${ZOOKEEPER_IP}:${PORT} --topic ${TOPIC_NAME}
```

### Check if there are any consumers for the topic
```shell
kafka-consumer-groups.sh --bootstrap-server ${KAFKA_BROKER_IP}:${PORT} --describe --group ${GROUP_NAME}
```

### View the current disk usage for the Kafka broker
```shell
df -h /path/to/kafka/dir
```

## Repair

### Increase the retention size of the Kafka topic by modifying the configuration file.
```shell


#!/bin/bash



# Set variables

CONFIG_FILE="${PATH_TO_KAFKA_CONFIG_FILE}"

NEW_RETENTION_SIZE="${NEW_RETENTION_SIZE}"



# Backup the original configuration file

cp $CONFIG_FILE $CONFIG_FILE.bak



# Modify the retention size for the specified topic

sed -i "s/^retention..bytes=.*/retention..bytes=$NEW_RETENTION_SIZE/g" $CONFIG_FILE



# Restart Kafka to apply the changes

systemctl restart kafka



echo "Retention size for Kafka topic $TOPIC_NAME has been increased to $NEW_RETENTION_SIZE."


```

### Increase the retention size for the topic
```shell
kafka-configs.sh --zookeeper ${ZOOKEEPER_IP}:${PORT} --alter --entity-type topics --entity-name ${TOPIC_NAME} --add-config retention.bytes=${NEW_RETENTION_SIZE}
```


This incident type refers to an issue where the retention size for a Kafka topic is set too low. The retention size is the amount of time that Kafka retains messages in a topic before they are deleted. When the retention size is set too low, messages may be deleted before they can be consumed or processed, which can result in data loss or other issues. This can be caused by misconfiguration or changes in the data ingestion rate. It is important to monitor retention sizes and adjust them as necessary to prevent data loss and ensure smooth data processing.


This incident type refers to the detection of an unused topic partition in Kafka Platform, a popular data streaming platform. Unused partitions can increase broker overhead and reduce efficiency, so it is recommended to delete them. The incident is triggered by a test notification or a monitoring query that checks the value of relevant metrics. The incident is resolved once the partition is deleted or confirmed to be intentional.


Unused Topic Partition in Kafka

This incident type refers to a situation where there is an issue with the communication between Kafka brokers due to network partitioning. A network partition is a situation where a network is split into two or more parts, and communication between them is disrupted. This issue can lead to data inconsistency, loss of messages, delayed message delivery, and other related problems. As a result, it is important to quickly detect and resolve these incidents to prevent any downstream impact on the system.


Parameters

This type of incident occurs when the offset of a Kafka topic decreases unexpectedly. In Kafka, an offset is a unique identifier that represents the position of a message within a partition. When the offset decreases, it means that some messages have been lost or deleted, which can cause data inconsistencies or other issues within the system. This incident type is critical for Kafka-based systems as it affects data integrity and can lead to data loss.


Kafka topic offset decreased incident.

This incident type refers to when a Kafka node runs out of storage space to store data. This can cause issues with data processing and storage, potentially leading to data loss and system downtime.


Kafka node insufficient storage check

```shell
export KAFKA_BROKER_IP="PLACEHOLDER"

export PORT="PLACEHOLDER"

export TOPIC_NAME="PLACEHOLDER"

export ZOOKEEPER_IP="PLACEHOLDER"

export NEW_RETENTION_SIZE="PLACEHOLDER"

export GROUP_NAME="PLACEHOLDER"

export PATH_TO_KAFKA_CONFIG_FILE="PLACEHOLDER"
```


### Connect to the Kafka broker

```shell
kafka-console-consumer.sh --bootstrap-server ${KAFKA_BROKER_IP}:${PORT} --topic ${TOPIC_NAME} --from-beginning
```

### View the current retention size for the topic

```shell
kafka-topics.sh --describe --zookeeper ${ZOOKEEPER_IP}:${PORT} --topic ${TOPIC_NAME}
```

### Check if there are any consumers for the topic

```shell
kafka-consumer-groups.sh --bootstrap-server ${KAFKA_BROKER_IP}:${PORT} --describe --group ${GROUP_NAME}
```

### View the current disk usage for the Kafka broker

```shell
df -h /path/to/kafka/dir
```


### Increase the retention size of the Kafka topic by modifying the configuration file.

```shell


#!/bin/bash



# Set variables

CONFIG_FILE="${PATH_TO_KAFKA_CONFIG_FILE}"

NEW_RETENTION_SIZE="${NEW_RETENTION_SIZE}"



# Backup the original configuration file

cp $CONFIG_FILE $CONFIG_FILE.bak



# Modify the retention size for the specified topic

sed -i "s/^retention..bytes=.*/retention..bytes=$NEW_RETENTION_SIZE/g" $CONFIG_FILE



# Restart Kafka to apply the changes

systemctl restart kafka



echo "Retention size for Kafka topic $TOPIC_NAME has been increased to $NEW_RETENTION_SIZE."


```

### Increase the retention size for the topic

```shell
kafka-configs.sh --zookeeper ${ZOOKEEPER_IP}:${PORT} --alter --entity-type topics --entity-name ${TOPIC_NAME} --add-config retention.bytes=${NEW_RETENTION_SIZE}
```


Low Retention Size for Kafka Topic.

Overview

Parameters

Debug

Connect to the Kafka broker

View the current retention size for the topic

Check if there are any consumers for the topic

View the current disk usage for the Kafka broker

Repair

Increase the retention size of the Kafka topic by modifying the configuration file.

Increase the retention size for the topic

Learn more

Related Runbooks

Unused Topic Partition in Kafka

Network Partitioning Between Kafka Brokers.

Parameters

Kafka topic offset decreased incident.

Support