---
id: 85dc6c99-f2fa-4090-81fa-cbb998189403
---
# Network Saturation Check for Kafka Nodes
---

This incident type refers to an issue where the network bandwidth on one or more Kafka nodes becomes fully saturated, causing the node(s) to experience performance degradation or even failure. This can result in message delivery delays or loss, and can also impact other nodes that rely on the affected node(s) for message replication. The saturation can be caused by a variety of factors, such as increased message traffic, misconfiguration, or hardware issues.

### Parameters
```shell
export NODE_IP_ADDRESS="PLACEHOLDER"

export FIN_TIMEOUT="PLACEHOLDER"

export BACKLOG_SIZE="PLACEHOLDER"

export NUM_CONNECTIONS="PLACEHOLDER"

export SYN_BACKLOG_SIZE="PLACEHOLDER"
```

## Debug

### Check network interface statistics
```shell
ifconfig -a
```

### Check network connections
```shell
netstat -an
```

### Check network bandwidth usage for each process
```shell
iftop
```

### Check network latency between nodes
```shell
ping ${NODE_IP_ADDRESS}
```

### Check network throughput between nodes
```shell
iperf -c ${NODE_IP_ADDRESS}
```

### Check Kafka node status
```shell
systemctl status kafka
```

### Check Kafka logs
```shell
journalctl -u kafka
```

### Check Kafka node configuration
```shell
cat /etc/kafka/server.properties
```

## Repair

### Optimize network configuration: This could involve tuning network settings on the nodes to better handle the traffic or changing the network topology to reduce congestion.
```shell


#!/bin/bash

# Script to optimize network configuration



# Stop Kafka services

sudo systemctl stop kafka.service



# Adjust network settings

sudo sysctl -w net.core.somaxconn=${NUM_CONNECTIONS}

sudo sysctl -w net.core.netdev_max_backlog=${BACKLOG_SIZE}

sudo sysctl -w net.ipv4.tcp_max_syn_backlog=${SYN_BACKLOG_SIZE}

sudo sysctl -w net.ipv4.tcp_syncookies=1

sudo sysctl -w net.ipv4.tcp_tw_reuse=1

sudo sysctl -w net.ipv4.tcp_fin_timeout=${FIN_TIMEOUT}



# Restart Kafka services

sudo systemctl start kafka.service


```


This incident type refers to an issue where the network bandwidth on one or more Kafka nodes becomes fully saturated, causing the node(s) to experience performance degradation or even failure. This can result in message delivery delays or loss, and can also impact other nodes that rely on the affected node(s) for message replication. The saturation can be caused by a variety of factors, such as increased message traffic, misconfiguration, or hardware issues.


This incident type refers to a situation where Spark jobs are failing due to resource contentions in the cluster. When multiple Spark jobs are trying to access the same resources or data at the same time, it can cause a bottleneck that leads to job failures. This can happen when the resources in the cluster are not properly allocated or when the number of jobs running simultaneously exceeds the cluster's capacity to handle them. The result is that Spark jobs fail, leading to disruptions in data processing and analysis.


Spark job failures due to cluster resource contentions.

This incident type refers to a situation where there is an issue with the communication between Kafka brokers due to network partitioning. A network partition is a situation where a network is split into two or more parts, and communication between them is disrupted. This issue can lead to data inconsistency, loss of messages, delayed message delivery, and other related problems. As a result, it is important to quickly detect and resolve these incidents to prevent any downstream impact on the system.


Network Partitioning Between Kafka Brokers.

This type of incident occurs when a node in the Kafka cluster is not being fully utilized. Kafka is a distributed streaming platform, and each node in the cluster is responsible for processing and storing a portion of the data. If a node is underutilized, it means that it is not processing as much data as it should be, which can lead to performance issues and potential data loss. Therefore, it is important to identify and address underutilized nodes in a Kafka cluster to ensure optimal performance and data reliability.


Kafka node underutilization check incident.

This incident type refers to when a Kafka node runs out of storage space to store data. This can cause issues with data processing and storage, potentially leading to data loss and system downtime.


Kafka node insufficient storage check

This incident type refers to a scenario where the liveness check for Kafka cannot reach/communicate with the host where the broker is running. This issue can arise due to various reasons such as network issues, broker failure, or configuration errors. This can result in the Kafka broker being unavailable or not responding to requests, leading to potential service disruptions or downtime.


Kafka liveness check failure.

```shell
export NODE_IP_ADDRESS="PLACEHOLDER"

export FIN_TIMEOUT="PLACEHOLDER"

export BACKLOG_SIZE="PLACEHOLDER"

export NUM_CONNECTIONS="PLACEHOLDER"

export SYN_BACKLOG_SIZE="PLACEHOLDER"
```


### Check network interface statistics

```shell
ifconfig -a
```

### Check network connections

```shell
netstat -an
```

### Check network bandwidth usage for each process

```shell
iftop
```

### Check network latency between nodes

```shell
ping ${NODE_IP_ADDRESS}
```

### Check network throughput between nodes

```shell
iperf -c ${NODE_IP_ADDRESS}
```

### Check Kafka node status

```shell
systemctl status kafka
```

### Check Kafka logs

```shell
journalctl -u kafka
```

### Check Kafka node configuration

```shell
cat /etc/kafka/server.properties
```


### Optimize network configuration: This could involve tuning network settings on the nodes to better handle the traffic or changing the network topology to reduce congestion.

```shell


#!/bin/bash

# Script to optimize network configuration



# Stop Kafka services

sudo systemctl stop kafka.service



# Adjust network settings

sudo sysctl -w net.core.somaxconn=${NUM_CONNECTIONS}

sudo sysctl -w net.core.netdev_max_backlog=${BACKLOG_SIZE}

sudo sysctl -w net.ipv4.tcp_max_syn_backlog=${SYN_BACKLOG_SIZE}

sudo sysctl -w net.ipv4.tcp_syncookies=1

sudo sysctl -w net.ipv4.tcp_tw_reuse=1

sudo sysctl -w net.ipv4.tcp_fin_timeout=${FIN_TIMEOUT}



# Restart Kafka services

sudo systemctl start kafka.service


```


Network Saturation Check for Kafka Nodes

Overview

Parameters

Debug

Check network interface statistics

Check network connections

Check network bandwidth usage for each process

Check network latency between nodes

Check network throughput between nodes

Check Kafka node status

Check Kafka logs

Check Kafka node configuration

Repair

Optimize network configuration: This could involve tuning network settings on the nodes to better handle the traffic or changing the network topology to reduce congestion.

Learn more

Related Runbooks

Spark job failures due to cluster resource contentions.

Network Partitioning Between Kafka Brokers.

Kafka node underutilization check incident.

Kafka node insufficient storage check

Support