---
id: 901d8feb-3867-44b3-8fba-7d8cfe2cb586
---

# Kubernetes Nodes with Network Unavailable
---

This incident type involves nodes in a Kubernetes cluster that are experiencing network unavailability, meaning they are not accessible. This could be due to a misconfiguration, route exhaustion, or a physical problem with the network connection to the hardware. It is a high urgency incident that requires immediate attention to restore network connectivity to the affected nodes.

### Parameters
```shell
# Environment Variables

export SERVICE_NAME="PLACEHOLDER"

export NODE_NAME="PLACEHOLDER"

export POD_NAME="PLACEHOLDER"

export DESTINATION_NETWORK="PLACEHOLDER"

export GATEWAY_IP="PLACEHOLDER"

export NAMESPACE="PLACEHOLDER"

export POLICY_FILE="PLACEHOLDER"
```

## Debug

### Check if Kubernetes nodes are available
```shell
kubectl get nodes
```

### Check the status of each Kubernetes node
```shell
kubectl describe nodes
```

### Check the network configuration of each Kubernetes node
```shell
kubectl describe nodes | grep -A 5 Addresses
```

### Check if there are any pods that are failing due to network issues
```shell
kubectl get pods --all-namespaces | grep -i error
```

### Check the status of the Kubernetes network components
```shell
kubectl get pod -n kube-system | grep kube-proxy

kubectl get pod -n kube-system | grep kube-dns
```

### Check if there are any network policies that could be blocking traffic
```shell
kubectl get networkpolicy --all-namespaces
```

### Check if there are any issues with the Kubernetes service
```shell
kubectl get service
```

### Check if there are any issues with the Kubernetes endpoint
```shell
kubectl get endpoints ${SERVICE_NAME}
```

### Check if there are any issues with the Kubernetes ingress
```shell
kubectl get ingress
```

### Firewall or security group settings blocking network traffic on the affected nodes
```shell
#!/bin/bash

# Define variables

NODE_NAME=${NODE_NAME}

# Check firewall settings

echo "Checking firewall settings on node $NODE_NAME in the current cluster..."

kubectl exec $NODE_NAME -n kube-system -- sh -c "iptables -L -n" | grep "Chain INPUT (policy DROP)"

if [ $? -eq 0 ]; then

    echo "Firewall settings are blocking incoming network traffic on node $NODE_NAME in the current cluster."

else

    echo "Firewall settings are not blocking incoming network traffic on node $NODE_NAME in the current cluster."

fi

# Check security group settings

echo "Checking security group settings on node $NODE_NAME in the current cluster..."

kubectl exec $NODE_NAME -n kube-system -- sh -c "ss -lntu" | grep "LISTEN"

if [ $? -eq 0 ]; then

    echo "Security group settings are allowing incoming network traffic on node $NODE_NAME in the current cluster."

else

    echo "Security group settings are not allowing incoming network traffic on node $NODE_NAME in the current cluster."

fi

```

### Routing issues in the cluster
```shell
#!/bin/bash

# Define Kubernetes namespace and pod name

K8S_NAMESPACE=${NAMESPACE}

K8S_POD_NAME=${POD_NAME}

# Check if kubectl is installed

if ! command -v kubectl &> /dev/null

then

    echo "kubectl command could not be found"

    exit

fi

# Verify network connectivity to Kubernetes API server

echo "Testing network connectivity to Kubernetes API server..."

kubectl --namespace=$K8S_NAMESPACE exec $K8S_POD_NAME -- curl -s -I -m 5 https://kubernetes.default.svc.cluster.local >/dev/null 2>&1

if [ $? -ne 0 ]; then

    echo "Unable to connect to Kubernetes API server"

    exit

fi

# Verify network connectivity to other pods in the same namespace

echo "Testing network connectivity to other pods in the same namespace..."

POD_IP=$(kubectl --namespace=$K8S_NAMESPACE exec $K8S_POD_NAME -- ip route get 8.8.8.8 | awk '{print $7}')

for pod in $(kubectl --namespace=$K8S_NAMESPACE get pods -o jsonpath='{range .items[*]}{@.metadata.name}{"\n"}{end}' | grep -v $K8S_POD_NAME); do

    kubectl --namespace=$K8S_NAMESPACE exec $K8S_POD_NAME -- curl -s -I -m 5 http://$pod.$K8S_NAMESPACE.svc.cluster.local >/dev/null 2>&1

    if [ $? -eq 0 ]; then

        echo "Successfully connected to $pod"

    else

        echo "Unable to connect to $pod"

    fi

done

```

## Repair

### Check if the routing tables are correctly configured to ensure that the nodes can communicate with each other.
```shell
bash
#!/bin/bash

# Set the namespace and pod name of the affected node

NAMESPACE=${NAMESPACE}

POD_NAME=${POD_NAME}

# Check the routing tables of the affected node

kubectl exec -n $NAMESPACE $POD_NAME -- sh -c "ip route show"

# If the routing tables are incorrect, update them to enable communication between nodes

kubectl exec -n $NAMESPACE $POD_NAME -- sh -c "ip route add ${DESTINATION_NETWORK} via ${GATEWAY_IP}"


```

### Check for any network security policies that may be blocking traffic between nodes and adjust them accordingly.
```shell
#!/bin/bash

# Set variables

NAMESPACE=${NAMESPACE}

# Check for network security policies

kubectl get networkpolicies -n $NAMESPACE

# Adjust network security policies

kubectl apply -f ${POLICY_FILE}.yaml -n $NAMESPACE


```

This incident type involves nodes in a Kubernetes cluster that are experiencing network unavailability, meaning they are not accessible. This could be due to a misconfiguration, route exhaustion, or a physical problem with the network connection to the hardware. It is a high urgency incident that requires immediate attention to restore network connectivity to the affected nodes.


This incident type is characterized by the detection of unauthorized access to the Kubernetes API server. This unauthorized access potentially enables attackers to manipulate cluster resources. Such incidents can lead to serious consequences, as unauthorized access allows attackers to make changes to the Kubernetes cluster, which may compromise the security and integrity of the entire system. Therefore, it is crucial to promptly detect and address such incidents to ensure the security and proper functioning of the Kubernetes cluster.


Unauthorized Access to Kubernetes API Server Detected

This incident type involves monitoring the replicas of a Kubernetes Statefulset, which is a type of workload in Kubernetes used for stateful applications. The incident is triggered when more than one replica's pods are down, creating an unsafe situation for manual operations. This incident is critical and requires immediate attention to resolve the issue and ensure the smooth functioning of the stateful applications.


Kubernetes Statefulset Replicas Monitoring Incident

Kubernetes Pods Pending incident indicates that one or more pods in a Kubernetes cluster are not running as expected and are in a pending state. This can happen due to various reasons such as resource constraints, scheduling issues, or network problems. This incident can impact the availability and performance of the application running on the Kubernetes cluster. It requires immediate attention to diagnose and resolve the underlying issue to ensure the pods are running as expected.


Kubernetes Pods Pending

The Kubernetes Nodes with Memorypressure incident type occurs when a Kubernetes cluster node is running low on memory, which can be caused by a memory leak in an application. This incident type requires immediate attention to prevent any downtime and ensure the proper functioning of the Kubernetes cluster. Typically, this incident type is monitored by DevOps teams using various monitoring tools, including PagerDuty, to identify and address memory pressure issues quickly.


Kubernetes Nodes with Memorypressure incident

This incident type occurs when the Kubernetes node status is not OK. It means that the scheduler cannot place pods on the node due to some underlying issue with the node's health. This incident can impact the availability and performance of the applications running on the Kubernetes cluster. Immediate attention is required to resolve this incident to restore the normal functioning of the Kubernetes cluster.


Kubernetes Node Status Not OK

```shell
# Environment Variables

export SERVICE_NAME="PLACEHOLDER"

export NODE_NAME="PLACEHOLDER"

export POD_NAME="PLACEHOLDER"

export DESTINATION_NETWORK="PLACEHOLDER"

export GATEWAY_IP="PLACEHOLDER"

export NAMESPACE="PLACEHOLDER"

export POLICY_FILE="PLACEHOLDER"
```


### Check if Kubernetes nodes are available

```shell
kubectl get nodes
```

### Check the status of each Kubernetes node

```shell
kubectl describe nodes
```

### Check the network configuration of each Kubernetes node

```shell
kubectl describe nodes | grep -A 5 Addresses
```

### Check if there are any pods that are failing due to network issues

```shell
kubectl get pods --all-namespaces | grep -i error
```

### Check the status of the Kubernetes network components

```shell
kubectl get pod -n kube-system | grep kube-proxy

kubectl get pod -n kube-system | grep kube-dns
```

### Check if there are any network policies that could be blocking traffic

```shell
kubectl get networkpolicy --all-namespaces
```

### Check if there are any issues with the Kubernetes service

```shell
kubectl get service
```

### Check if there are any issues with the Kubernetes endpoint

```shell
kubectl get endpoints ${SERVICE_NAME}
```

### Check if there are any issues with the Kubernetes ingress

```shell
kubectl get ingress
```

### Firewall or security group settings blocking network traffic on the affected nodes

```shell
#!/bin/bash

# Define variables

NODE_NAME=${NODE_NAME}

# Check firewall settings

echo "Checking firewall settings on node $NODE_NAME in the current cluster..."

kubectl exec $NODE_NAME -n kube-system -- sh -c "iptables -L -n" | grep "Chain INPUT (policy DROP)"

if [ $? -eq 0 ]; then

    echo "Firewall settings are blocking incoming network traffic on node $NODE_NAME in the current cluster."

else

    echo "Firewall settings are not blocking incoming network traffic on node $NODE_NAME in the current cluster."

fi

# Check security group settings

echo "Checking security group settings on node $NODE_NAME in the current cluster..."

kubectl exec $NODE_NAME -n kube-system -- sh -c "ss -lntu" | grep "LISTEN"

if [ $? -eq 0 ]; then

    echo "Security group settings are allowing incoming network traffic on node $NODE_NAME in the current cluster."

else

    echo "Security group settings are not allowing incoming network traffic on node $NODE_NAME in the current cluster."

fi

```

### Routing issues in the cluster

```shell
#!/bin/bash

# Define Kubernetes namespace and pod name

K8S_NAMESPACE=${NAMESPACE}

K8S_POD_NAME=${POD_NAME}

# Check if kubectl is installed

if ! command -v kubectl &> /dev/null

then

    echo "kubectl command could not be found"

    exit

fi

# Verify network connectivity to Kubernetes API server

echo "Testing network connectivity to Kubernetes API server..."

kubectl --namespace=$K8S_NAMESPACE exec $K8S_POD_NAME -- curl -s -I -m 5 https://kubernetes.default.svc.cluster.local >/dev/null 2>&1

if [ $? -ne 0 ]; then

    echo "Unable to connect to Kubernetes API server"

    exit

fi

# Verify network connectivity to other pods in the same namespace

echo "Testing network connectivity to other pods in the same namespace..."

POD_IP=$(kubectl --namespace=$K8S_NAMESPACE exec $K8S_POD_NAME -- ip route get 8.8.8.8 | awk '{print $7}')

for pod in $(kubectl --namespace=$K8S_NAMESPACE get pods -o jsonpath='{range .items[*]}{@.metadata.name}{"\n"}{end}' | grep -v $K8S_POD_NAME); do

    kubectl --namespace=$K8S_NAMESPACE exec $K8S_POD_NAME -- curl -s -I -m 5 http://$pod.$K8S_NAMESPACE.svc.cluster.local >/dev/null 2>&1

    if [ $? -eq 0 ]; then

        echo "Successfully connected to $pod"

    else

        echo "Unable to connect to $pod"

    fi

done

```


### Check if the routing tables are correctly configured to ensure that the nodes can communicate with each other.

```shell
bash
#!/bin/bash

# Set the namespace and pod name of the affected node

NAMESPACE=${NAMESPACE}

POD_NAME=${POD_NAME}

# Check the routing tables of the affected node

kubectl exec -n $NAMESPACE $POD_NAME -- sh -c "ip route show"

# If the routing tables are incorrect, update them to enable communication between nodes

kubectl exec -n $NAMESPACE $POD_NAME -- sh -c "ip route add ${DESTINATION_NETWORK} via ${GATEWAY_IP}"


```

### Check for any network security policies that may be blocking traffic between nodes and adjust them accordingly.

```shell
#!/bin/bash

# Set variables

NAMESPACE=${NAMESPACE}

# Check for network security policies

kubectl get networkpolicies -n $NAMESPACE

# Adjust network security policies

kubectl apply -f ${POLICY_FILE}.yaml -n $NAMESPACE


```


Kubernetes Nodes with Network Unavailable

Overview

Parameters

Debug

Check if Kubernetes nodes are available

Check the status of each Kubernetes node

Check the network configuration of each Kubernetes node

Check if there are any pods that are failing due to network issues

Check the status of the Kubernetes network components

Check if there are any network policies that could be blocking traffic

Check if there are any issues with the Kubernetes service

Check if there are any issues with the Kubernetes endpoint

Check if there are any issues with the Kubernetes ingress

Firewall or security group settings blocking network traffic on the affected nodes

Routing issues in the cluster

Repair

Check if the routing tables are correctly configured to ensure that the nodes can communicate with each other.

Check for any network security policies that may be blocking traffic between nodes and adjust them accordingly.

Learn more

Related Runbooks

Unauthorized Access to Kubernetes API Server Detected

Kubernetes Statefulset Replicas Monitoring Incident

Kubernetes Pods Pending

Kubernetes Nodes with Memorypressure incident

Support