---
id: e66d93db-3549-4bb6-9317-37da3583d48a
---
# Network Partitioning Between Kafka Brokers.
---

This incident type refers to a situation where there is an issue with the communication between Kafka brokers due to network partitioning. A network partition is a situation where a network is split into two or more parts, and communication between them is disrupted. This issue can lead to data inconsistency, loss of messages, delayed message delivery, and other related problems. As a result, it is important to quickly detect and resolve these incidents to prevent any downstream impact on the system.

### Parameters
```shell
export KAFKA_BROKER_IP="PLACEHOLDER"

export BROKER_1_IP="PLACEHOLDER"

export BROKER_2_IP="PLACEHOLDER"
```

## Debug

### Check the status of Kafka brokers
```shell
systemctl status kafka
```

### Check the logs of Kafka brokers for any errors related to network partitioning
```shell
journalctl -u kafka | grep "network partitioning"
```

### Check the network configuration of Kafka brokers
```shell
ifconfig
```

### Check the network routing tables of Kafka brokers
```shell
route -n
```

### Verify the network connectivity between Kafka brokers
```shell
ping ${KAFKA_BROKER_IP}
```

### Check the firewall settings to ensure that the necessary ports are open for Kafka communication
```shell
iptables -L
```

### Check the disk usage of Kafka brokers to ensure that there is enough space for message storage
```shell
df -h
```

## Repair

### Check the network connectivity between Kafka brokers to identify any issues with the network configuration.
```shell


#!/bin/bash



# Define the Kafka brokers to check

BROKER_1=${BROKER_1_IP}

BROKER_2=${BROKER_2_IP}



# Ping the first broker

ping -c 3 $BROKER_1



# Check the exit status of the ping command

if [ $? -eq 0 ]; then

  echo "The network connection to $BROKER_1 is working."

else

  echo "The network connection to $BROKER_1 is not working."

fi



# Ping the second broker

ping -c 3 $BROKER_2



# Check the exit status of the ping command

if [ $? -eq 0 ]; then

  echo "The network connection to $BROKER_2 is working."

else

  echo "The network connection to $BROKER_2 is not working."

fi


```


This incident type refers to a situation where there is an issue with the communication between Kafka brokers due to network partitioning. A network partition is a situation where a network is split into two or more parts, and communication between them is disrupted. This issue can lead to data inconsistency, loss of messages, delayed message delivery, and other related problems. As a result, it is important to quickly detect and resolve these incidents to prevent any downstream impact on the system.


This incident type refers to the detection of an unused topic partition in Kafka Platform, a popular data streaming platform. Unused partitions can increase broker overhead and reduce efficiency, so it is recommended to delete them. The incident is triggered by a test notification or a monitoring query that checks the value of relevant metrics. The incident is resolved once the partition is deleted or confirmed to be intentional.


Unused Topic Partition in Kafka

This type of incident occurs when the offset of a Kafka topic decreases unexpectedly. In Kafka, an offset is a unique identifier that represents the position of a message within a partition. When the offset decreases, it means that some messages have been lost or deleted, which can cause data inconsistencies or other issues within the system. This incident type is critical for Kafka-based systems as it affects data integrity and can lead to data loss.


Kafka topic offset decreased incident.

This incident type refers to an issue where the network bandwidth on one or more Kafka nodes becomes fully saturated, causing the node(s) to experience performance degradation or even failure. This can result in message delivery delays or loss, and can also impact other nodes that rely on the affected node(s) for message replication. The saturation can be caused by a variety of factors, such as increased message traffic, misconfiguration, or hardware issues.


Network Saturation Check for Kafka Nodes

This incident type refers to a situation where an unauthorized user gains access to a Kafka cluster due to misconfigured Access Control Lists (ACLs) in Kafka. This can result in potential data breaches or other security issues, as the unauthorized user may be able to read, write or modify data within the Kafka cluster. The incident typically requires immediate attention and remediation to prevent further unauthorized access.


Kafka Unauthorized Access

This incident type relates to an issue with time synchronization that affects the proper functioning of a Kafka cluster. As Kafka relies on accurate time synchronization to maintain data consistency, any failure in this process can cause serious issues. The incident might cause data loss or duplication, service disruptions, or other problems that can impact the overall performance of the system. It requires immediate attention from the engineering team to investigate and resolve the root cause of the issue.


Kafka Time synchronization issue impacting Kafka cluster.

```shell
export KAFKA_BROKER_IP="PLACEHOLDER"

export BROKER_1_IP="PLACEHOLDER"

export BROKER_2_IP="PLACEHOLDER"
```


### Check the status of Kafka brokers

```shell
systemctl status kafka
```

### Check the logs of Kafka brokers for any errors related to network partitioning

```shell
journalctl -u kafka | grep "network partitioning"
```

### Check the network configuration of Kafka brokers

```shell
ifconfig
```

### Check the network routing tables of Kafka brokers

```shell
route -n
```

### Verify the network connectivity between Kafka brokers

```shell
ping ${KAFKA_BROKER_IP}
```

### Check the firewall settings to ensure that the necessary ports are open for Kafka communication

```shell
iptables -L
```

### Check the disk usage of Kafka brokers to ensure that there is enough space for message storage

```shell
df -h
```


### Check the network connectivity between Kafka brokers to identify any issues with the network configuration.

```shell


#!/bin/bash



# Define the Kafka brokers to check

BROKER_1=${BROKER_1_IP}

BROKER_2=${BROKER_2_IP}



# Ping the first broker

ping -c 3 $BROKER_1



# Check the exit status of the ping command

if [ $? -eq 0 ]; then

  echo "The network connection to $BROKER_1 is working."

else

  echo "The network connection to $BROKER_1 is not working."

fi



# Ping the second broker

ping -c 3 $BROKER_2



# Check the exit status of the ping command

if [ $? -eq 0 ]; then

  echo "The network connection to $BROKER_2 is working."

else

  echo "The network connection to $BROKER_2 is not working."

fi


```


Network Partitioning Between Kafka Brokers.

Overview

Parameters

Debug

Check the status of Kafka brokers

Check the network configuration of Kafka brokers

Check the network routing tables of Kafka brokers

Verify the network connectivity between Kafka brokers

Check the firewall settings to ensure that the necessary ports are open for Kafka communication

Check the disk usage of Kafka brokers to ensure that there is enough space for message storage

Repair

Check the network connectivity between Kafka brokers to identify any issues with the network configuration.

Learn more

Related Runbooks

Unused Topic Partition in Kafka

Kafka topic offset decreased incident.

Network Saturation Check for Kafka Nodes

Kafka Unauthorized Access

Support