---
id: e44df9ec-616d-11ee-8c99-0242ac120002
---

# Cassandra Disk Space Not Reclaimed After Adding New Nodes
---

This incident type occurs when disk space in a Cassandra cluster is not reclaimed after adding new nodes. This can lead to the cluster running out of disk space, causing performance issues or even complete failure. The root cause can vary, but it is often due to misconfigurations or software bugs. It is important to address this issue promptly to prevent data loss and ensure the stability of the Cassandra cluster.

### Parameters
```shell
export HOSTNAME="PLACEHOLDER"

export PATH_TO_CASSANDRA_CONFIG_FILE="PLACEHOLDER"

export NUMBER_OF_EXPECTED_NODES="PLACEHOLDER"

export NUMBER_OF_CURRENT_NODES="PLACEHOLDER"

export PATH_TO_CASSANDRA_HOME="PLACEHOLDER"
```

## Debug

### Check disk usage on all nodes
```shell
ssh ${HOSTNAME} 'df -h'
```

### Check available disk space
```shell
ssh ${HOSTNAME} 'du -sh /var/lib/cassandra/*'
```

### Check if compaction is stalled
```shell
nodetool compactionstats
```

### Check if there are any pending tasks
```shell
nodetool tpstats
```

### Check if there are any errors in the system log
```shell
tail -f /var/log/cassandra/system.log
```

### Check if there are any pending repairs
```shell
nodetool repair -pr
```

### Check if nodetool cleanup has been run
```shell
nodetool cleanup
```

### Check if nodetool scrub has been run
```shell
nodetool scrub
```

### Check if nodetool compact has been run
```shell
nodetool compact
```

## Repair

### Review the Cassandra cluster configuration to ensure that it is properly set up for the number of nodes currently in use and the expected growth of the cluster.
```shell


#!/bin/bash



# Define variables

CASSANDRA_CONFIG_FILE=${PATH_TO_CASSANDRA_CONFIG_FILE}

EXPECTED_NODES=${NUMBER_OF_EXPECTED_NODES}

CURRENT_NODES=${NUMBER_OF_CURRENT_NODES}



# Check if the configuration file exists

if [ ! -f "$CASSANDRA_CONFIG_FILE" ]; then

    echo "Error: Cassandra configuration file not found."

    exit 1

fi



# Check if the number of nodes is correct

if [ "$CURRENT_NODES" -ne "$EXPECTED_NODES" ]; then

    echo "Warning: Current number of nodes does not match expected number of nodes."

fi



# Check if the configuration file is properly set up

if grep -q "^num_tokens: $EXPECTED_NODES$" "$CASSANDRA_CONFIG_FILE"; then

    echo "Cassandra cluster configuration is properly set up."

else

    echo "Error: Cassandra cluster configuration is not properly set up."

    sed -i "s/^num_tokens:.*$/num_tokens: $EXPECTED_NODES/" "$CASSANDRA_CONFIG_FILE"

    echo "Fixed Cassandra cluster configuration."

fi


```

### Run a repair on the Cassandra cluster to clean up any old or unused data that may be taking up disk space.
```shell


#!/bin/bash



# Set the Cassandra home directory

CASSANDRA_HOME=${PATH_TO_CASSANDRA_HOME}



# Stop the Cassandra service

sudo service cassandra stop



# Run the repair command

$CASSANDRA_HOME/bin/nodetool repair -pr



# Start the Cassandra service

sudo service cassandra start


```


This incident type occurs when disk space in a Cassandra cluster is not reclaimed after adding new nodes. This can lead to the cluster running out of disk space, causing performance issues or even complete failure. The root cause can vary, but it is often due to misconfigurations or software bugs. It is important to address this issue promptly to prevent data loss and ensure the stability of the Cassandra cluster.


This incident type refers to the failure of a replica node in a PostgreSQL database system that is running on a Linux-based operating system. A replica node is a copy of the primary database node that is used to provide high availability and fault tolerance. When a replica node fails, it can result in data loss, decreased system performance, and potential downtime for users. This type of incident requires immediate attention from a software engineer to diagnose and resolve the issue as quickly as possible.


PostgreSQL Replica Node Failure on Linux.

The Kafka Data Loss Incident refers to an incident where there is a misconfiguration of the data retention policy in Kafka, which can result in data loss. Kafka is a distributed streaming platform that is commonly used in big data environments. When data retention policies are misconfigured, data that should be retained is deleted, or data that should be deleted is retained. This can lead to significant data loss and can have serious consequences for businesses that rely on the data stored in Kafka.


Kafka Data Loss Incident

Compaction is a process in Cassandra that merges multiple SSTables (Sorted String Tables) into a single SSTable, eliminating any redundant data and improving read performance. However, sometimes compaction can fail due to various reasons such as insufficient disk space or corrupted data, resulting in degraded performance or even complete failure of the database. Troubleshooting compaction merging SSTables involves identifying and resolving the root cause of such failures to ensure the smooth functioning of the Cassandra database.


Troubleshooting Compaction Merging SSTables in Cassandra

This incident type refers to a problem in a Cassandra cluster where the token range imbalances cause uneven distribution of data across the cluster. This can result in slower read and write performance that can impact the overall functionality of the system. Token range imbalances occur when the distribution of the tokens that define the ranges of data each node is responsible for is not evenly spread across the cluster. As a result, certain nodes may be responsible for a disproportionate amount of data, leading to performance issues and potential failure of the system.


Token Range Imbalances Causing Uneven Data Distribution and Performance Issues in Cassandra Cluster

This incident type refers to a situation where there is a significant delay in the execution of queries on a Cassandra cluster. This delay can cause the system to become unresponsive and result in slower performance. It may be caused by a variety of factors such as an increase in traffic, inefficient queries, or hardware issues. The issue can impact the functionality of the system and requires immediate attention to prevent further disruption.


Slow Query Performance on Cassandra Cluster.

```shell
export HOSTNAME="PLACEHOLDER"

export PATH_TO_CASSANDRA_CONFIG_FILE="PLACEHOLDER"

export NUMBER_OF_EXPECTED_NODES="PLACEHOLDER"

export NUMBER_OF_CURRENT_NODES="PLACEHOLDER"

export PATH_TO_CASSANDRA_HOME="PLACEHOLDER"
```


### Check disk usage on all nodes

```shell
ssh ${HOSTNAME} 'df -h'
```

### Check available disk space

```shell
ssh ${HOSTNAME} 'du -sh /var/lib/cassandra/*'
```

### Check if compaction is stalled

```shell
nodetool compactionstats
```

### Check if there are any pending tasks

```shell
nodetool tpstats
```

### Check if there are any errors in the system log

```shell
tail -f /var/log/cassandra/system.log
```

### Check if there are any pending repairs

```shell
nodetool repair -pr
```

### Check if nodetool cleanup has been run

```shell
nodetool cleanup
```

### Check if nodetool scrub has been run

```shell
nodetool scrub
```

### Check if nodetool compact has been run

```shell
nodetool compact
```


### Review the Cassandra cluster configuration to ensure that it is properly set up for the number of nodes currently in use and the expected growth of the cluster.

```shell


#!/bin/bash



# Define variables

CASSANDRA_CONFIG_FILE=${PATH_TO_CASSANDRA_CONFIG_FILE}

EXPECTED_NODES=${NUMBER_OF_EXPECTED_NODES}

CURRENT_NODES=${NUMBER_OF_CURRENT_NODES}



# Check if the configuration file exists

if [ ! -f "$CASSANDRA_CONFIG_FILE" ]; then

    echo "Error: Cassandra configuration file not found."

    exit 1

fi



# Check if the number of nodes is correct

if [ "$CURRENT_NODES" -ne "$EXPECTED_NODES" ]; then

    echo "Warning: Current number of nodes does not match expected number of nodes."

fi



# Check if the configuration file is properly set up

if grep -q "^num_tokens: $EXPECTED_NODES$" "$CASSANDRA_CONFIG_FILE"; then

    echo "Cassandra cluster configuration is properly set up."

else

    echo "Error: Cassandra cluster configuration is not properly set up."

    sed -i "s/^num_tokens:.*$/num_tokens: $EXPECTED_NODES/" "$CASSANDRA_CONFIG_FILE"

    echo "Fixed Cassandra cluster configuration."

fi


```

### Run a repair on the Cassandra cluster to clean up any old or unused data that may be taking up disk space.

```shell


#!/bin/bash



# Set the Cassandra home directory

CASSANDRA_HOME=${PATH_TO_CASSANDRA_HOME}



# Stop the Cassandra service

sudo service cassandra stop



# Run the repair command

$CASSANDRA_HOME/bin/nodetool repair -pr



# Start the Cassandra service

sudo service cassandra start


```


Cassandra Disk Space Not Reclaimed After Adding New Nodes

Overview

Parameters

Debug

Check disk usage on all nodes

Check available disk space

Check if compaction is stalled

Check if there are any pending tasks

Check if there are any errors in the system log

Check if there are any pending repairs

Check if nodetool cleanup has been run

Check if nodetool scrub has been run

Check if nodetool compact has been run

Repair

Review the Cassandra cluster configuration to ensure that it is properly set up for the number of nodes currently in use and the expected growth of the cluster.

Run a repair on the Cassandra cluster to clean up any old or unused data that may be taking up disk space.

Learn more

Related Runbooks

PostgreSQL Replica Node Failure on Linux.

Kafka Data Loss Incident

Troubleshooting Compaction Merging SSTables in Cassandra

Token Range Imbalances Causing Uneven Data Distribution and Performance Issues in Cassandra Cluster

Support