---
id: bc21bcd1-a028-4d47-9bd5-21079e4f48bc
---

# Token Range Imbalances Causing Uneven Data Distribution and Performance Issues in Cassandra Cluster
---

This incident type refers to a problem in a Cassandra cluster where the token range imbalances cause uneven distribution of data across the cluster. This can result in slower read and write performance that can impact the overall functionality of the system. Token range imbalances occur when the distribution of the tokens that define the ranges of data each node is responsible for is not evenly spread across the cluster. As a result, certain nodes may be responsible for a disproportionate amount of data, leading to performance issues and potential failure of the system.

### Parameters
```shell
export KEYSPACE="PLACEHOLDER"

export KEY="PLACEHOLDER"

export TABLE="PLACEHOLDER"

export NODE_ADDRESS="PLACEHOLDER"

export PORT_NUMBER="PLACEHOLDER"

export NAME_OF_CLUSTER="PLACEHOLDER"

export CLUSTER_NAME="PLACEHOLDER"

export KEYSPACE_NAME="PLACEHOLDER"
```

## Debug

### Check the status of the Cassandra cluster
```shell
nodetool status
```

### Check for any inconsistencies in the token ranges
```shell
nodetool describecluster
```

### Check for any imbalanced token ranges
```shell
nodetool ring
```

### Check for any under-replicated data
```shell
nodetool getendpoints ${KEYSPACE} ${TABLE} ${KEY}
```

### Check for any over-replicated data
```shell
nodetool cleanup ${KEYSPACE} ${TABLE}
```

### Check for any read performance issues
```shell
nodetool cfstats
```

### Check for any write performance issues
```shell
nodetool tpstats
```

## Repair

### Rebalancing the cluster: Token range imbalances can occur due to nodes being added or removed from the cluster. Rebalancing the cluster can help redistribute the data evenly across all nodes. This can be achieved by using nodetool or by performing an automatic rebalance.
```shell


#!/bin/bash



# Set the variables

${CLUSTER_NAME}="${NAME_OF_CLUSTER}"

${NODE_ADDRESS}="${NODE_ADDRESS}"



# Check if nodetool is installed

if ! command -v nodetool &> /dev/null

then

    echo "nodetool could not be found. Please install Cassandra before running this script."

    exit 1

fi



# Run nodetool to rebalance the cluster

nodetool -- ${NODE_ADDRESS} -p ${PORT_NUMBER} repair --full ${KEYSPACE_NAME}



# Check if the rebalance was successful

if [ $? -eq 0 ]

then

    echo "Cluster rebalanced successfully."

else

    echo "Cluster rebalance failed. Please check the logs for more information."

    exit 1

fi



exit 0


```

This incident type refers to a problem in a Cassandra cluster where the token range imbalances cause uneven distribution of data across the cluster. This can result in slower read and write performance that can impact the overall functionality of the system. Token range imbalances occur when the distribution of the tokens that define the ranges of data each node is responsible for is not evenly spread across the cluster. As a result, certain nodes may be responsible for a disproportionate amount of data, leading to performance issues and potential failure of the system.


Data skew slowing down a Spark job is a common incident that occurs when there is an uneven distribution of data across partitions. This can lead to some partitions handling a much larger amount of data than others, causing delays and slowing down the entire job. The result is a significant decrease in performance and a longer processing time for the Spark job.


Data Skew Slowing Down Spark Job.

The Kafka Consumer Group Lag incident refers to a situation where the lag time for a Kafka consumer group exceeds the expected threshold. This delay can result in delayed or lost data processing, leading to service degradation or failure.


Kafka Consumer Group Lag Incident

This incident type refers to a situation where the number of partitions for a Kafka topic is insufficient to handle the incoming data stream. This can lead to low throughput and/or increased write latency. In order to resolve this issue, it may be necessary to increase the number of Kafka topic partitions. Failure to address this issue can lead to degraded performance and potential data loss.


Inadequate Kafka Topic Partitions Incident

This incident type refers to a situation where there is a significant delay in the execution of queries on a Cassandra cluster. This delay can cause the system to become unresponsive and result in slower performance. It may be caused by a variety of factors such as an increase in traffic, inefficient queries, or hardware issues. The issue can impact the functionality of the system and requires immediate attention to prevent further disruption.


Slow Query Performance on Cassandra Cluster.

In this incident type, there is an issue with a Cassandra cluster where one or more disks are running slow. This can cause performance issues and potentially lead to data loss or downtime. The goal is to identify and address the specific disk(s) causing the problem in order to restore normal cluster operations.


Slow Disk in Cassandra Cluster

```shell
export KEYSPACE="PLACEHOLDER"

export KEY="PLACEHOLDER"

export TABLE="PLACEHOLDER"

export NODE_ADDRESS="PLACEHOLDER"

export PORT_NUMBER="PLACEHOLDER"

export NAME_OF_CLUSTER="PLACEHOLDER"

export CLUSTER_NAME="PLACEHOLDER"

export KEYSPACE_NAME="PLACEHOLDER"
```


### Check the status of the Cassandra cluster

```shell
nodetool status
```

### Check for any inconsistencies in the token ranges

```shell
nodetool describecluster
```

### Check for any imbalanced token ranges

```shell
nodetool ring
```

### Check for any under-replicated data

```shell
nodetool getendpoints ${KEYSPACE} ${TABLE} ${KEY}
```

### Check for any over-replicated data

```shell
nodetool cleanup ${KEYSPACE} ${TABLE}
```

### Check for any read performance issues

```shell
nodetool cfstats
```

### Check for any write performance issues

```shell
nodetool tpstats
```


### Rebalancing the cluster: Token range imbalances can occur due to nodes being added or removed from the cluster. Rebalancing the cluster can help redistribute the data evenly across all nodes. This can be achieved by using nodetool or by performing an automatic rebalance.

```shell


#!/bin/bash



# Set the variables

${CLUSTER_NAME}="${NAME_OF_CLUSTER}"

${NODE_ADDRESS}="${NODE_ADDRESS}"



# Check if nodetool is installed

if ! command -v nodetool &> /dev/null

then

    echo "nodetool could not be found. Please install Cassandra before running this script."

    exit 1

fi



# Run nodetool to rebalance the cluster

nodetool -- ${NODE_ADDRESS} -p ${PORT_NUMBER} repair --full ${KEYSPACE_NAME}



# Check if the rebalance was successful

if [ $? -eq 0 ]

then

    echo "Cluster rebalanced successfully."

else

    echo "Cluster rebalance failed. Please check the logs for more information."

    exit 1

fi



exit 0


```


Token Range Imbalances Causing Uneven Data Distribution and Performance Issues in Cassandra Cluster

Overview

Parameters

Debug

Check the status of the Cassandra cluster

Check for any inconsistencies in the token ranges

Check for any imbalanced token ranges

Check for any under-replicated data

Check for any over-replicated data

Check for any read performance issues

Check for any write performance issues

Repair

Rebalancing the cluster: Token range imbalances can occur due to nodes being added or removed from the cluster. Rebalancing the cluster can help redistribute the data evenly across all nodes. This can be achieved by using nodetool or by performing an automatic rebalance.

Learn more

Related Runbooks

Data Skew Slowing Down Spark Job.

Kafka Consumer Group Lag Incident

Inadequate Kafka Topic Partitions Incident

Slow Query Performance on Cassandra Cluster.

Support