---
id: 6f6ef2de-0e57-4d23-9f72-cb905b6b6b12
---
# Kafka Data Loss Incident
---

The Kafka Data Loss Incident refers to an incident where there is a misconfiguration of the data retention policy in Kafka, which can result in data loss. Kafka is a distributed streaming platform that is commonly used in big data environments. When data retention policies are misconfigured, data that should be retained is deleted, or data that should be deleted is retained. This can lead to significant data loss and can have serious consequences for businesses that rely on the data stored in Kafka.

### Parameters
```shell
export PATH_TO_KAFKA_CONFIG="PLACEHOLDER"

export PATH_TO_KAFKA_LOGS="PLACEHOLDER"

export ZOOKEEPER_HOST="PLACEHOLDER"

export ZOOKEEPER_PORT="PLACEHOLDER"

export KAFKA_BROKER_HOST="PLACEHOLDER"

export KAFKA_BROKER_PORT="PLACEHOLDER"

export PATH_TO_KAFKA_CONFIG_FILE="PLACEHOLDER"

export DESIRED_DATA_RETENTION_POLICY="PLACEHOLDER"
```

## Debug

### Check if Kafka is running
```shell
systemctl status kafka
```

### Check the Kafka logs for any errors
```shell
journalctl -u kafka --since "10 minutes ago"
```

### Check if the Kafka retention policy is configured correctly
```shell
cat ${PATH_TO_KAFKA_CONFIG}/server.properties | grep log.retention.bytes
```

### Check the disk usage of the Kafka logs
```shell
du -sh ${PATH_TO_KAFKA_LOGS}
```

### Check if the topic in question is still available
```shell
kafka-topics --list --zookeeper ${ZOOKEEPER_HOST}:${ZOOKEEPER_PORT}
```

### Check the status of the Kafka brokers
```shell
kafka-broker-api-versions --bootstrap-server ${KAFKA_BROKER_HOST}:${KAFKA_BROKER_PORT}
```

## Repair

### Review and update Kafka's data retention policy to ensure that it is correctly configured and aligned with your data retention requirements.
```shell


#!/bin/bash



# Set variables

KAFKA_CONFIG_FILE=${PATH_TO_KAFKA_CONFIG_FILE}

DATA_RETENTION_POLICY=${DESIRED_DATA_RETENTION_POLICY}



# Stop Kafka server

sudo systemctl stop kafka



# Update retention policy in Kafka config file

sudo sed -i "s/^log.retention.hours=.*/log.retention.hours=$DATA_RETENTION_POLICY/" $KAFKA_CONFIG_FILE



# Start Kafka server

sudo systemctl start kafka



# Verify retention policy update

echo "Current data retention policy:"

grep log.retention.hours $KAFKA_CONFIG_FILE


```


The Kafka Data Loss Incident refers to an incident where there is a misconfiguration of the data retention policy in Kafka, which can result in data loss. Kafka is a distributed streaming platform that is commonly used in big data environments. When data retention policies are misconfigured, data that should be retained is deleted, or data that should be deleted is retained. This can lead to significant data loss and can have serious consequences for businesses that rely on the data stored in Kafka.


This incident type refers to a situation where there is an issue with the communication between Kafka brokers due to network partitioning. A network partition is a situation where a network is split into two or more parts, and communication between them is disrupted. This issue can lead to data inconsistency, loss of messages, delayed message delivery, and other related problems. As a result, it is important to quickly detect and resolve these incidents to prevent any downstream impact on the system.


Parameters

This incident type refers to an issue where the retention size for a Kafka topic is set too low. The retention size is the amount of time that Kafka retains messages in a topic before they are deleted. When the retention size is set too low, messages may be deleted before they can be consumed or processed, which can result in data loss or other issues. This can be caused by misconfiguration or changes in the data ingestion rate. It is important to monitor retention sizes and adjust them as necessary to prevent data loss and ensure smooth data processing.


Low Retention Size for Kafka Topic.

This type of incident occurs when the offset of a Kafka topic decreases unexpectedly. In Kafka, an offset is a unique identifier that represents the position of a message within a partition. When the offset decreases, it means that some messages have been lost or deleted, which can cause data inconsistencies or other issues within the system. This incident type is critical for Kafka-based systems as it affects data integrity and can lead to data loss.


Kafka topic offset decreased incident.

This incident type refers to when a Kafka node runs out of storage space to store data. This can cause issues with data processing and storage, potentially leading to data loss and system downtime.


Kafka node insufficient storage check

```shell
export PATH_TO_KAFKA_CONFIG="PLACEHOLDER"

export PATH_TO_KAFKA_LOGS="PLACEHOLDER"

export ZOOKEEPER_HOST="PLACEHOLDER"

export ZOOKEEPER_PORT="PLACEHOLDER"

export KAFKA_BROKER_HOST="PLACEHOLDER"

export KAFKA_BROKER_PORT="PLACEHOLDER"

export PATH_TO_KAFKA_CONFIG_FILE="PLACEHOLDER"

export DESIRED_DATA_RETENTION_POLICY="PLACEHOLDER"
```


### Check if Kafka is running

```shell
systemctl status kafka
```

### Check the Kafka logs for any errors

```shell
journalctl -u kafka --since "10 minutes ago"
```

### Check if the Kafka retention policy is configured correctly

```shell
cat ${PATH_TO_KAFKA_CONFIG}/server.properties | grep log.retention.bytes
```

### Check the disk usage of the Kafka logs

```shell
du -sh ${PATH_TO_KAFKA_LOGS}
```

### Check if the topic in question is still available

```shell
kafka-topics --list --zookeeper ${ZOOKEEPER_HOST}:${ZOOKEEPER_PORT}
```

### Check the status of the Kafka brokers

```shell
kafka-broker-api-versions --bootstrap-server ${KAFKA_BROKER_HOST}:${KAFKA_BROKER_PORT}
```


### Review and update Kafka's data retention policy to ensure that it is correctly configured and aligned with your data retention requirements.

```shell


#!/bin/bash



# Set variables

KAFKA_CONFIG_FILE=${PATH_TO_KAFKA_CONFIG_FILE}

DATA_RETENTION_POLICY=${DESIRED_DATA_RETENTION_POLICY}



# Stop Kafka server

sudo systemctl stop kafka



# Update retention policy in Kafka config file

sudo sed -i "s/^log.retention.hours=.*/log.retention.hours=$DATA_RETENTION_POLICY/" $KAFKA_CONFIG_FILE



# Start Kafka server

sudo systemctl start kafka



# Verify retention policy update

echo "Current data retention policy:"

grep log.retention.hours $KAFKA_CONFIG_FILE


```


Kafka Data Loss Incident

Overview

Parameters

Debug

Check if Kafka is running

Check the Kafka logs for any errors

Check if the Kafka retention policy is configured correctly

Check the disk usage of the Kafka logs

Check if the topic in question is still available

Check the status of the Kafka brokers

Repair

Review and update Kafka's data retention policy to ensure that it is correctly configured and aligned with your data retention requirements.

Learn more

Related Runbooks

Network Partitioning Between Kafka Brokers.

Parameters

Low Retention Size for Kafka Topic.

Kafka topic offset decreased incident.

Support