---
id: 55c91f9a-44e6-438d-bc35-dbc947cd86e0
---
# Kafka node insufficient storage check
---

This incident type refers to when a Kafka node runs out of storage space to store data. This can cause issues with data processing and storage, potentially leading to data loss and system downtime.

### Parameters
```shell
export MOUNT_POINT="PLACEHOLDER"

export KAFKA_LOGS_DIRECTORY="PLACEHOLDER"

export KAFKA_LOG_FILE="PLACEHOLDER"

export KAFKA_CONFIG_FILE="PLACEHOLDER"

export PATH_TO_KAFKA_DATA_DIRECTORY="PLACEHOLDER"

export NUMBER_OF_DAYS="PLACEHOLDER"

export PATH_TO_KAFKA_CONFIG_FILE="PLACEHOLDER"
```

## Debug

### Check available storage space on the Kafka node
```shell
df -h ${MOUNT_POINT}
```

### Check the size of the Kafka logs directory
```shell
du -sh ${KAFKA_LOGS_DIRECTORY}
```

### Check the amount of free memory on the Kafka node
```shell
free -h
```

### Check the Kafka server logs for any errors related to storage space
```shell
grep "insufficient storage" ${KAFKA_LOG_FILE}
```

### Check the Kafka server configuration for the maximum storage size
```shell
grep "max.message.bytes" ${KAFKA_CONFIG_FILE}
```

## Repair

### Reduce the amount of data being stored on the Kafka node by deleting old data or implementing data retention policies.
```shell


#!/bin/bash



# Set the path to the Kafka data directory

KAFKA_DATA_DIR=${PATH_TO_KAFKA_DATA_DIRECTORY}



# Set the number of days of data to retain

DAYS_TO_RETAIN=${NUMBER_OF_DAYS}



# Find all Kafka topic directories older than DAYS_TO_RETAIN

OLD_TOPIC_DIRS=$(find $KAFKA_DATA_DIR -maxdepth 1 -type d -mtime +$DAYS_TO_RETAIN -name "kafka-*")



# Delete the old topic directories

for DIR in $OLD_TOPIC_DIRS

do

    echo "Deleting Kafka topic directory: $DIR"

    rm -rf $DIR

done


```

### Optimize the Kafka node's configuration to use storage more efficiently, such as compressing data or using a more effective storage engine.
```shell
bash

#!/bin/bash



# Set Kafka configuration file path

KAFKA_CONFIG_PATH=${PATH_TO_KAFKA_CONFIG_FILE}



# Enable data compression in the Kafka configuration file

sed -i 's/^#compression.type=.*/compression.type=snappy/' $KAFKA_CONFIG_PATH



# Restart the Kafka service to apply the configuration changes

systemctl restart kafka.service


```


This incident type refers to when a Kafka node runs out of storage space to store data. This can cause issues with data processing and storage, potentially leading to data loss and system downtime.


This incident type refers to a situation where the persistent volumes used in a software application are running out of storage space. Persistent volumes are used to store data in a containerized environment, and if the storage space is limited, it can lead to errors and service disruptions. This issue can occur due to various reasons such as a sudden increase in data volume, incorrect configuration, or inefficient utilization of storage resources. It is crucial to monitor and manage the available storage space to ensure the smooth functioning of the software application.


Kubernetes - Persistent Volumes Running Out of Storage Space.

This incident type refers to a situation where there is an issue with the communication between Kafka brokers due to network partitioning. A network partition is a situation where a network is split into two or more parts, and communication between them is disrupted. This issue can lead to data inconsistency, loss of messages, delayed message delivery, and other related problems. As a result, it is important to quickly detect and resolve these incidents to prevent any downstream impact on the system.


Parameters

This incident type refers to an issue where the retention size for a Kafka topic is set too low. The retention size is the amount of time that Kafka retains messages in a topic before they are deleted. When the retention size is set too low, messages may be deleted before they can be consumed or processed, which can result in data loss or other issues. This can be caused by misconfiguration or changes in the data ingestion rate. It is important to monitor retention sizes and adjust them as necessary to prevent data loss and ensure smooth data processing.


Low Retention Size for Kafka Topic.

This type of incident occurs when the offset of a Kafka topic decreases unexpectedly. In Kafka, an offset is a unique identifier that represents the position of a message within a partition. When the offset decreases, it means that some messages have been lost or deleted, which can cause data inconsistencies or other issues within the system. This incident type is critical for Kafka-based systems as it affects data integrity and can lead to data loss.


Kafka topic offset decreased incident.

```shell
export MOUNT_POINT="PLACEHOLDER"

export KAFKA_LOGS_DIRECTORY="PLACEHOLDER"

export KAFKA_LOG_FILE="PLACEHOLDER"

export KAFKA_CONFIG_FILE="PLACEHOLDER"

export PATH_TO_KAFKA_DATA_DIRECTORY="PLACEHOLDER"

export NUMBER_OF_DAYS="PLACEHOLDER"

export PATH_TO_KAFKA_CONFIG_FILE="PLACEHOLDER"
```


### Check available storage space on the Kafka node

```shell
df -h ${MOUNT_POINT}
```

### Check the size of the Kafka logs directory

```shell
du -sh ${KAFKA_LOGS_DIRECTORY}
```

### Check the amount of free memory on the Kafka node

```shell
free -h
```

### Check the Kafka server logs for any errors related to storage space

```shell
grep "insufficient storage" ${KAFKA_LOG_FILE}
```

### Check the Kafka server configuration for the maximum storage size

```shell
grep "max.message.bytes" ${KAFKA_CONFIG_FILE}
```


### Reduce the amount of data being stored on the Kafka node by deleting old data or implementing data retention policies.

```shell


#!/bin/bash



# Set the path to the Kafka data directory

KAFKA_DATA_DIR=${PATH_TO_KAFKA_DATA_DIRECTORY}



# Set the number of days of data to retain

DAYS_TO_RETAIN=${NUMBER_OF_DAYS}



# Find all Kafka topic directories older than DAYS_TO_RETAIN

OLD_TOPIC_DIRS=$(find $KAFKA_DATA_DIR -maxdepth 1 -type d -mtime +$DAYS_TO_RETAIN -name "kafka-*")



# Delete the old topic directories

for DIR in $OLD_TOPIC_DIRS

do

    echo "Deleting Kafka topic directory: $DIR"

    rm -rf $DIR

done


```

### Optimize the Kafka node's configuration to use storage more efficiently, such as compressing data or using a more effective storage engine.

```shell
bash

#!/bin/bash



# Set Kafka configuration file path

KAFKA_CONFIG_PATH=${PATH_TO_KAFKA_CONFIG_FILE}



# Enable data compression in the Kafka configuration file

sed -i 's/^#compression.type=.*/compression.type=snappy/' $KAFKA_CONFIG_PATH



# Restart the Kafka service to apply the configuration changes

systemctl restart kafka.service


```


Kafka node insufficient storage check

Overview

Parameters

Debug

Check available storage space on the Kafka node

Check the size of the Kafka logs directory

Check the amount of free memory on the Kafka node

Check the Kafka server configuration for the maximum storage size

Repair

Reduce the amount of data being stored on the Kafka node by deleting old data or implementing data retention policies.

Optimize the Kafka node's configuration to use storage more efficiently, such as compressing data or using a more effective storage engine.

Learn more

Related Runbooks

Kubernetes - Persistent Volumes Running Out of Storage Space.

Network Partitioning Between Kafka Brokers.

Parameters

Low Retention Size for Kafka Topic.

Support