---
id: 6d5d8d47-1a05-4917-ac72-68ba937d8958
---
# Kafka Resource Exhaustion in Kafka Broker Cluster.
---

This incident type refers to a situation where the Kafka broker cluster has exhausted its resources. This could happen due to several reasons such as an increase in traffic, inadequate resources, or inefficient configurations. As a result, the Kafka brokers may stop responding or experience performance issues, leading to a disruption in the overall Kafka cluster.

### Parameters
```shell
export KAFKA_BROKER_PID="PLACEHOLDER"

export KAFKA_LOG_FILE="PLACEHOLDER"

export KAFKA_CONFIG_FILE="PLACEHOLDER"

export BROKER_ID="PLACEHOLDER"

export NEW_STORAGE_SIZE="PLACEHOLDER"

export NEW_MEMORY_SIZE="PLACEHOLDER"

export KAFKA_HOME_DIRECTORY="PLACEHOLDER"
```

## Debug

### Check CPU and memory usage of Kafka broker processes
```shell
top -p ${KAFKA_BROKER_PID}
```

### Check disk space usage on Kafka brokers
```shell
df -h
```

### Check network traffic to and from Kafka brokers
```shell
iftop
```

### Check Kafka logs for errors and warnings
```shell
tail -f ${KAFKA_LOG_FILE}
```

### Check Kafka broker configurations for potential resource limitations
```shell
cat ${KAFKA_CONFIG_FILE}
```

## Repair

### Increase the memory and storage capacity of the Kafka broker cluster to prevent resource exhaustion.
```shell


#!/bin/bash



# Define variables

KAFKA_HOME=${KAFKA_HOME_DIRECTORY}

BROKER_ID=${BROKER_ID}

MEMORY_SIZE=${NEW_MEMORY_SIZE}

STORAGE_SIZE=${NEW_STORAGE_SIZE}



# Update the Kafka broker configuration file with new memory and storage sizes

sed -i "s/^broker.id=$BROKER_ID/broker.id=$BROKER_ID\nmessage.max.bytes=$MEMORY_SIZE\nlog.retention.bytes=$STORAGE_SIZE/g" $KAFKA_HOME/config/server.properties



# Restart the Kafka broker service

systemctl restart kafka


```


This incident type refers to a situation where the Kafka broker cluster has exhausted its resources. This could happen due to several reasons such as an increase in traffic, inadequate resources, or inefficient configurations. As a result, the Kafka brokers may stop responding or experience performance issues, leading to a disruption in the overall Kafka cluster.


This incident type refers to a situation where Spark jobs are failing due to resource contentions in the cluster. When multiple Spark jobs are trying to access the same resources or data at the same time, it can cause a bottleneck that leads to job failures. This can happen when the resources in the cluster are not properly allocated or when the number of jobs running simultaneously exceeds the cluster's capacity to handle them. The result is that Spark jobs fail, leading to disruptions in data processing and analysis.


Spark job failures due to cluster resource contentions.

This incident type refers to a situation where a Spark cluster experiences performance bottlenecks when it is subjected to peak loads. In other words, the Spark cluster struggles to handle the high volume of requests it receives during times of heavy traffic or increased demand. This can lead to slower processing times, delays, or even system crashes. Identifying and resolving the root cause of the bottlenecks is crucial to ensure the smooth functioning of the Spark cluster during peak loads.


Spark cluster bottlenecks during peak loads.

This incident type involves monitoring the replicas of a Kubernetes Statefulset, which is a type of workload in Kubernetes used for stateful applications. The incident is triggered when more than one replica's pods are down, creating an unsafe situation for manual operations. This incident is critical and requires immediate attention to resolve the issue and ensure the smooth functioning of the stateful applications.


Kubernetes Statefulset Replicas Monitoring Incident

This incident type refers to a situation where there is an issue with the communication between Kafka brokers due to network partitioning. A network partition is a situation where a network is split into two or more parts, and communication between them is disrupted. This issue can lead to data inconsistency, loss of messages, delayed message delivery, and other related problems. As a result, it is important to quickly detect and resolve these incidents to prevent any downstream impact on the system.


Network Partitioning Between Kafka Brokers.

This type of incident occurs when a node in the Kafka cluster is not being fully utilized. Kafka is a distributed streaming platform, and each node in the cluster is responsible for processing and storing a portion of the data. If a node is underutilized, it means that it is not processing as much data as it should be, which can lead to performance issues and potential data loss. Therefore, it is important to identify and address underutilized nodes in a Kafka cluster to ensure optimal performance and data reliability.


Kafka node underutilization check incident.

```shell
export KAFKA_BROKER_PID="PLACEHOLDER"

export KAFKA_LOG_FILE="PLACEHOLDER"

export KAFKA_CONFIG_FILE="PLACEHOLDER"

export BROKER_ID="PLACEHOLDER"

export NEW_STORAGE_SIZE="PLACEHOLDER"

export NEW_MEMORY_SIZE="PLACEHOLDER"

export KAFKA_HOME_DIRECTORY="PLACEHOLDER"
```


### Check CPU and memory usage of Kafka broker processes

```shell
top -p ${KAFKA_BROKER_PID}
```

### Check disk space usage on Kafka brokers

```shell
df -h
```

### Check network traffic to and from Kafka brokers

```shell
iftop
```

### Check Kafka logs for errors and warnings

```shell
tail -f ${KAFKA_LOG_FILE}
```

### Check Kafka broker configurations for potential resource limitations

```shell
cat ${KAFKA_CONFIG_FILE}
```


### Increase the memory and storage capacity of the Kafka broker cluster to prevent resource exhaustion.

```shell


#!/bin/bash



# Define variables

KAFKA_HOME=${KAFKA_HOME_DIRECTORY}

BROKER_ID=${BROKER_ID}

MEMORY_SIZE=${NEW_MEMORY_SIZE}

STORAGE_SIZE=${NEW_STORAGE_SIZE}



# Update the Kafka broker configuration file with new memory and storage sizes

sed -i "s/^broker.id=$BROKER_ID/broker.id=$BROKER_ID\nmessage.max.bytes=$MEMORY_SIZE\nlog.retention.bytes=$STORAGE_SIZE/g" $KAFKA_HOME/config/server.properties



# Restart the Kafka broker service

systemctl restart kafka


```


Kafka Resource Exhaustion in Kafka Broker Cluster.

Overview

Parameters

Debug

Check CPU and memory usage of Kafka broker processes

Check disk space usage on Kafka brokers

Check network traffic to and from Kafka brokers

Check Kafka logs for errors and warnings

Check Kafka broker configurations for potential resource limitations

Repair

Increase the memory and storage capacity of the Kafka broker cluster to prevent resource exhaustion.

Learn more

Related Runbooks

Spark job failures due to cluster resource contentions.

Spark cluster bottlenecks during peak loads.

Kubernetes Statefulset Replicas Monitoring Incident

Network Partitioning Between Kafka Brokers.

Support