---
id: d2dc25b9-86e7-421a-bb18-70f31281b9d3
---

# Kafka liveness check failure.
---

This incident type refers to a scenario where the liveness check for Kafka cannot reach/communicate with the host where the broker is running. This issue can arise due to various reasons such as network issues, broker failure, or configuration errors. This can result in the Kafka broker being unavailable or not responding to requests, leading to potential service disruptions or downtime.

### Parameters
```shell
export ZK_HOST="PLACEHOLDER"

export ZK_PORT="PLACEHOLDER"

export BROKER_PORT="PLACEHOLDER"

export BROKER_HOST="PLACEHOLDER"

export TOPIC_NAME="PLACEHOLDER"
```

## Debug

### Check if Zookeeper is running on the specified <zk_host> and <zk_port>
```shell
echo ruok | nc ${ZK_HOST} ${ZK_PORT}
```

### Check the status of all Kafka brokers
```shell
bin/kafka-broker-api-versions.sh --bootstrap-server ${BROKER_HOST}:${BROKER_PORT}
```

### Check if the Kafka broker is running on the specified <broker_host> and <broker_port>
```shell
echo "stats" | nc ${BROKER_HOST} ${BROKER_PORT}
```

### Check if the Kafka broker is registered with Zookeeper
```shell
bin/kafka-topics.sh --list --zookeeper ${ZK_HOST}:${ZK_PORT}
```

### Check if the Kafka topics are being replicated correctly
```shell
bin/kafka-replica-verification.sh --broker-list ${BROKER_HOST}:${BROKER_PORT} --topic ${TOPIC_NAME}
```

### Check if the Kafka producer is able to send messages to the broker
```shell
bin/kafka-console-producer.sh --broker-list ${BROKER_HOST}:${BROKER_PORT} --topic ${TOPIC_NAME}
```

### Check if the Kafka consumer is able to receive messages from the broker
```shell
bin/kafka-console-consumer.sh --bootstrap-server ${BROKER_HOST}:${BROKER_PORT} --topic ${TOPIC_NAME}
```

### Network connectivity issues between the liveness check service and the broker host.
```shell


#!/bin/bash



# Set the Kafka broker host and port

KAFKA_BROKER_HOST=${BROKER_HOST}

KAFKA_BROKER_PORT=${BROKER_PORT}



# Check if the Kafka broker host is reachable via ping

ping -c 4 $KAFKA_BROKER_HOST



# Check if the Kafka broker port is open and listening

nc -zv $KAFKA_BROKER_HOST $KAFKA_BROKER_PORT



# Check if the liveness check service can connect to the Kafka broker host and port

telnet $KAFKA_BROKER_HOST $KAFKA_BROKER_PORT


```

### The broker host is experiencing high CPU or memory utilization that is causing it to be unresponsive.
```shell


#!/bin/bash



#Set variables

BROKER_HOST=${BROKER_HOST} 

BROKER_PORT=${BROKER_PORT}



#Check CPU utilization

CPU_UTIL=$(ssh $BROKER_HOST "top -bn1 | grep 'Cpu(s)' | awk '{print $2+$4}'")



if (( $(echo "$CPU_UTIL > 80" | bc -l) )); then

    echo "WARNING: High CPU utilization on $BROKER_HOST. Current CPU utilization is $CPU_UTIL%"

fi



#Check memory utilization

MEMORY_UTIL=$(ssh $BROKER_HOST "free | awk '/Mem:/ {print $3/$2 * 100.0}'")



if (( $(echo "$MEMORY_UTIL > 80" | bc -l) )); then

    echo "WARNING: High memory utilization on $BROKER_HOST. Current memory utilization is $MEMORY_UTIL%"

fi



#Check Kafka broker status

KAFKA_STATUS=$(ssh $BROKER_HOST "systemctl status kafka")



if [[ "$KAFKA_STATUS" != *"active (running)"* ]]; then

    echo "ERROR: Kafka broker is not running on $BROKER_HOST. Status: $KAFKA_STATUS"

fi



#Check Kafka broker port

KAFKA_PORT_STATUS=$(nc -zv $BROKER_HOST $BROKER_PORT 2>&1)



if [[ "$KAFKA_PORT_STATUS" == *"Connection refused"* ]]; then

    echo "ERROR: Kafka broker port $BROKER_PORT is not open on $BROKER_HOST"

fi


```

## Repair

### Check if the broker is running on the host specified in the configuration file. If not, start the broker on the specified host.
```shell
bash

#!/bin/bash

# Define variables

BROKER_HOST=${BROKER_HOST}

BROKER_PORT=${BROKER_PORT}



# Check if broker is running

nc -z -v $BROKER_HOST $BROKER_PORT

if [ $? -eq 0 ]; then

    echo "Broker is running on $BROKER_HOST"

else

    echo "Broker is not running on $BROKER_HOST"

    # Start the broker

    sudo systemctl start kafka

    echo "Started Kafka broker on $BROKER_HOST"

fi


```

This incident type refers to a scenario where the liveness check for Kafka cannot reach/communicate with the host where the broker is running. This issue can arise due to various reasons such as network issues, broker failure, or configuration errors. This can result in the Kafka broker being unavailable or not responding to requests, leading to potential service disruptions or downtime.


This incident type refers to a situation where there is an issue with the communication between Kafka brokers due to network partitioning. A network partition is a situation where a network is split into two or more parts, and communication between them is disrupted. This issue can lead to data inconsistency, loss of messages, delayed message delivery, and other related problems. As a result, it is important to quickly detect and resolve these incidents to prevent any downstream impact on the system.


Network Partitioning Between Kafka Brokers.

This incident type refers to an issue where the network bandwidth on one or more Kafka nodes becomes fully saturated, causing the node(s) to experience performance degradation or even failure. This can result in message delivery delays or loss, and can also impact other nodes that rely on the affected node(s) for message replication. The saturation can be caused by a variety of factors, such as increased message traffic, misconfiguration, or hardware issues.


Network Saturation Check for Kafka Nodes

This incident type refers to when a Kafka node runs out of storage space to store data. This can cause issues with data processing and storage, potentially leading to data loss and system downtime.


Kafka node insufficient storage check

This incident type refers to a situation where an unauthorized user gains access to a Kafka cluster due to misconfigured Access Control Lists (ACLs) in Kafka. This can result in potential data breaches or other security issues, as the unauthorized user may be able to read, write or modify data within the Kafka cluster. The incident typically requires immediate attention and remediation to prevent further unauthorized access.


Kafka Unauthorized Access

This incident type relates to an issue with time synchronization that affects the proper functioning of a Kafka cluster. As Kafka relies on accurate time synchronization to maintain data consistency, any failure in this process can cause serious issues. The incident might cause data loss or duplication, service disruptions, or other problems that can impact the overall performance of the system. It requires immediate attention from the engineering team to investigate and resolve the root cause of the issue.


Kafka Time synchronization issue impacting Kafka cluster.

```shell
export ZK_HOST="PLACEHOLDER"

export ZK_PORT="PLACEHOLDER"

export BROKER_PORT="PLACEHOLDER"

export BROKER_HOST="PLACEHOLDER"

export TOPIC_NAME="PLACEHOLDER"
```


### Check if Zookeeper is running on the specified \<zk\_host> and \<zk\_port>

```shell
echo ruok | nc ${ZK_HOST} ${ZK_PORT}
```

### Check the status of all Kafka brokers

```shell
bin/kafka-broker-api-versions.sh --bootstrap-server ${BROKER_HOST}:${BROKER_PORT}
```

### Check if the Kafka broker is running on the specified \<broker\_host> and \<broker\_port>

```shell
echo "stats" | nc ${BROKER_HOST} ${BROKER_PORT}
```

### Check if the Kafka broker is registered with Zookeeper

```shell
bin/kafka-topics.sh --list --zookeeper ${ZK_HOST}:${ZK_PORT}
```

### Check if the Kafka topics are being replicated correctly

```shell
bin/kafka-replica-verification.sh --broker-list ${BROKER_HOST}:${BROKER_PORT} --topic ${TOPIC_NAME}
```

### Check if the Kafka producer is able to send messages to the broker

```shell
bin/kafka-console-producer.sh --broker-list ${BROKER_HOST}:${BROKER_PORT} --topic ${TOPIC_NAME}
```

### Check if the Kafka consumer is able to receive messages from the broker

```shell
bin/kafka-console-consumer.sh --bootstrap-server ${BROKER_HOST}:${BROKER_PORT} --topic ${TOPIC_NAME}
```

### Network connectivity issues between the liveness check service and the broker host.

```shell


#!/bin/bash



# Set the Kafka broker host and port

KAFKA_BROKER_HOST=${BROKER_HOST}

KAFKA_BROKER_PORT=${BROKER_PORT}



# Check if the Kafka broker host is reachable via ping

ping -c 4 $KAFKA_BROKER_HOST



# Check if the Kafka broker port is open and listening

nc -zv $KAFKA_BROKER_HOST $KAFKA_BROKER_PORT



# Check if the liveness check service can connect to the Kafka broker host and port

telnet $KAFKA_BROKER_HOST $KAFKA_BROKER_PORT


```

### The broker host is experiencing high CPU or memory utilization that is causing it to be unresponsive.

```shell


#!/bin/bash



#Set variables

BROKER_HOST=${BROKER_HOST} 

BROKER_PORT=${BROKER_PORT}



#Check CPU utilization

CPU_UTIL=$(ssh $BROKER_HOST "top -bn1 | grep 'Cpu(s)' | awk '{print $2+$4}'")



if (( $(echo "$CPU_UTIL > 80" | bc -l) )); then

    echo "WARNING: High CPU utilization on $BROKER_HOST. Current CPU utilization is $CPU_UTIL%"

fi



#Check memory utilization

MEMORY_UTIL=$(ssh $BROKER_HOST "free | awk '/Mem:/ {print $3/$2 * 100.0}'")



if (( $(echo "$MEMORY_UTIL > 80" | bc -l) )); then

    echo "WARNING: High memory utilization on $BROKER_HOST. Current memory utilization is $MEMORY_UTIL%"

fi



#Check Kafka broker status

KAFKA_STATUS=$(ssh $BROKER_HOST "systemctl status kafka")



if [[ "$KAFKA_STATUS" != *"active (running)"* ]]; then

    echo "ERROR: Kafka broker is not running on $BROKER_HOST. Status: $KAFKA_STATUS"

fi



#Check Kafka broker port

KAFKA_PORT_STATUS=$(nc -zv $BROKER_HOST $BROKER_PORT 2>&1)



if [[ "$KAFKA_PORT_STATUS" == *"Connection refused"* ]]; then

    echo "ERROR: Kafka broker port $BROKER_PORT is not open on $BROKER_HOST"

fi


```


### Check if the broker is running on the host specified in the configuration file. If not, start the broker on the specified host.

```shell
bash

#!/bin/bash

# Define variables

BROKER_HOST=${BROKER_HOST}

BROKER_PORT=${BROKER_PORT}



# Check if broker is running

nc -z -v $BROKER_HOST $BROKER_PORT

if [ $? -eq 0 ]; then

    echo "Broker is running on $BROKER_HOST"

else

    echo "Broker is not running on $BROKER_HOST"

    # Start the broker

    sudo systemctl start kafka

    echo "Started Kafka broker on $BROKER_HOST"

fi


```


Kafka liveness check failure.

Overview

Parameters

Debug

Check if Zookeeper is running on the specified <zk_host> and <zk_port>

Check the status of all Kafka brokers

Check if the Kafka broker is running on the specified <broker_host> and <broker_port>

Check if the Kafka broker is registered with Zookeeper

Check if the Kafka topics are being replicated correctly

Check if the Kafka producer is able to send messages to the broker

Check if the Kafka consumer is able to receive messages from the broker

Network connectivity issues between the liveness check service and the broker host.

The broker host is experiencing high CPU or memory utilization that is causing it to be unresponsive.

Repair

Check if the broker is running on the host specified in the configuration file. If not, start the broker on the specified host.

Learn more

Related Runbooks

Network Partitioning Between Kafka Brokers.

Network Saturation Check for Kafka Nodes

Kafka node insufficient storage check

Kafka Unauthorized Access

Support