---
id: 765ea4c1-f268-4c9e-b3a9-e5fa0e8847e8
---

# Cassandra connection timeouts incident
---

This incident refers to a situation where there are connection timeouts between nodes in a Cassandra cluster, resulting in a high level of urgency. The incident is triggered automatically by an alert system and assigned to an engineer for resolution. It requires immediate attention as it can potentially impact the availability and performance of the Cassandra service.

### Parameters
```shell
# Environment Variables

export NODE_IP_ADDRESS="PLACEHOLDER"

export TIMEOUT_VALUE_FOR_PING_COMMAND="PLACEHOLDER"

export PATH_TO_CASSANDRA_LOG_FILE="PLACEHOLDER"

export NODE_NAME="PLACEHOLDER"

export PORT_NUMBER="PLACEHOLDER"
```

## Debug

### Check Cassandra service status
```shell
systemctl status cassandra
```

### Check Cassandra cluster status
```shell
nodetool status
```

### Check if there are any network issues
```shell
ping ${NODE_IP_ADDRESS}
```

### Check if there are any Cassandra connection issues
```shell
cqlsh ${NODE_IP_ADDRESS}
```

### Check Cassandra logs for any errors or warnings related to timeouts
```shell
tail -f /var/log/cassandra/system.log | grep timeout
```

### Check for any Cassandra nodetool errors or warnings related to timeouts
```shell
nodetool tpstats | grep Timeouts
```

### Network connectivity issues between nodes in the Cassandra cluster.
```shell


#!/bin/bash



# Set variables

IP_ADDRESS=${NODE_IP_ADDRESS}

PING_TIMEOUT=${TIMEOUT_VALUE_FOR_PING_COMMAND}

CASSANDRA_LOG=${PATH_TO_CASSANDRA_LOG_FILE}



# Check network connectivity

ping -c 3 -W $PING_TIMEOUT $IP_ADDRESS > /dev/null 2>&1

if [ $? -eq 0 ]; then

    echo "Network connectivity between nodes is working fine."

else

    echo "Network connectivity issues detected between nodes."

    echo "Checking Cassandra log file for errors..."

    grep -i "ERROR" $CASSANDRA_LOG

fi


```

## Repair

### Identify the root cause of the connection timeouts
```shell
echo "Checking system logs for hardware or network failures..."

grep -i "error" /var/log/syslog
```

### Restart the Cassandra nodes that are causing the timeouts and monitor the system for any improvements.
```shell


#!/bin/bash



# Set the Cassandra node name and IP address

NODE_NAME=${NODE_NAME}

NODE_IP=${NODE_IP}



# Check if the node is running

if nodetool status | grep $NODE_NAME | grep -q "UN"; then

  echo "Node $NODE_NAME is running and healthy"

else

  echo "Node $NODE_NAME is not running. Starting the node..."

  sudo service cassandra start

fi



# Restart the node

echo "Restarting node $NODE_NAME ($NODE_IP)..."

sudo nodetool drain

sudo service cassandra stop

sudo service cassandra start



# Monitor the system for any improvements

echo "Checking the Cassandra log for any errors..."

tail -n 100 /var/log/cassandra/system.log


```


This incident refers to a situation where there are connection timeouts between nodes in a Cassandra cluster, resulting in a high level of urgency. The incident is triggered automatically by an alert system and assigned to an engineer for resolution. It requires immediate attention as it can potentially impact the availability and performance of the Cassandra service.


This incident type refers to a scenario where there is a high rate of statement timeouts in a Postgresql database instance. This can lead to degraded performance and potentially impact the availability of the database. It is important to quickly identify and address the underlying cause of the timeouts to ensure the stability of the system.


Postgresql high rate statement timeout incident.

The MongoDB cursors timeouts incident refers to an issue where too many cursors are timing out in a MongoDB instance. This can cause disruptions in the normal functioning of the database and lead to delays in processing queries. This incident type requires immediate attention from the software engineer to resolve the issue and prevent any further damage to the system.


MongoDB cursors timeouts incident

This incident type refers to a situation where there is a significant delay in the execution of queries on a Cassandra cluster. This delay can cause the system to become unresponsive and result in slower performance. It may be caused by a variety of factors such as an increase in traffic, inefficient queries, or hardware issues. The issue can impact the functionality of the system and requires immediate attention to prevent further disruption.


Slow Query Performance on Cassandra Cluster.

This incident type refers to a situation where a delay or slowness occurs in a system that uses Cassandra database due to the shared storage. Shared storage means multiple servers are accessing the same storage unit, and this can cause latency issues. This type of incident can lead to performance degradation, and it needs to be addressed promptly to ensure optimal system performance.


Latency Caused by Shared Storage in Cassandra

This incident type refers to a situation where a high number of mutations are being dropped on a Cassandra database. Mutations are changes made to the database, such as inserting new data or updating existing data. When mutations are dropped, it means that they were not successfully recorded in the database. This can be caused by a variety of factors, such as hardware or network issues, configuration problems, or bugs in the software. When this occurs, it can result in data inconsistencies or loss, and can impact the performance and reliability of the application that relies on the database.


High Number of Dropped Mutations on Cassandra Database

```shell
# Environment Variables

export NODE_IP_ADDRESS="PLACEHOLDER"

export TIMEOUT_VALUE_FOR_PING_COMMAND="PLACEHOLDER"

export PATH_TO_CASSANDRA_LOG_FILE="PLACEHOLDER"

export NODE_NAME="PLACEHOLDER"

export PORT_NUMBER="PLACEHOLDER"
```


### Check Cassandra service status

```shell
systemctl status cassandra
```

### Check Cassandra cluster status

```shell
nodetool status
```

### Check if there are any network issues

```shell
ping ${NODE_IP_ADDRESS}
```

### Check if there are any Cassandra connection issues

```shell
cqlsh ${NODE_IP_ADDRESS}
```

### Check Cassandra logs for any errors or warnings related to timeouts

```shell
tail -f /var/log/cassandra/system.log | grep timeout
```

### Check for any Cassandra nodetool errors or warnings related to timeouts

```shell
nodetool tpstats | grep Timeouts
```

### Network connectivity issues between nodes in the Cassandra cluster.

```shell


#!/bin/bash



# Set variables

IP_ADDRESS=${NODE_IP_ADDRESS}

PING_TIMEOUT=${TIMEOUT_VALUE_FOR_PING_COMMAND}

CASSANDRA_LOG=${PATH_TO_CASSANDRA_LOG_FILE}



# Check network connectivity

ping -c 3 -W $PING_TIMEOUT $IP_ADDRESS > /dev/null 2>&1

if [ $? -eq 0 ]; then

    echo "Network connectivity between nodes is working fine."

else

    echo "Network connectivity issues detected between nodes."

    echo "Checking Cassandra log file for errors..."

    grep -i "ERROR" $CASSANDRA_LOG

fi


```


### Identify the root cause of the connection timeouts

```shell
echo "Checking system logs for hardware or network failures..."

grep -i "error" /var/log/syslog
```

### Restart the Cassandra nodes that are causing the timeouts and monitor the system for any improvements.

```shell


#!/bin/bash



# Set the Cassandra node name and IP address

NODE_NAME=${NODE_NAME}

NODE_IP=${NODE_IP}



# Check if the node is running

if nodetool status | grep $NODE_NAME | grep -q "UN"; then

  echo "Node $NODE_NAME is running and healthy"

else

  echo "Node $NODE_NAME is not running. Starting the node..."

  sudo service cassandra start

fi



# Restart the node

echo "Restarting node $NODE_NAME ($NODE_IP)..."

sudo nodetool drain

sudo service cassandra stop

sudo service cassandra start



# Monitor the system for any improvements

echo "Checking the Cassandra log for any errors..."

tail -n 100 /var/log/cassandra/system.log


```


Cassandra connection timeouts incident

Overview

Parameters

Debug

Check Cassandra service status

Check Cassandra cluster status

Check if there are any network issues

Check if there are any Cassandra connection issues

Network connectivity issues between nodes in the Cassandra cluster.

Repair

Identify the root cause of the connection timeouts

Restart the Cassandra nodes that are causing the timeouts and monitor the system for any improvements.

Learn more

Related Runbooks

Postgresql high rate statement timeout incident.

MongoDB cursors timeouts incident

Slow Query Performance on Cassandra Cluster.

Latency Caused by Shared Storage in Cassandra

Support