---
id: 6674db66-8c04-4f25-89b2-0327ba614932
---

# Inadequate Replication on Cassandra Cluster
---

Inadequate replication refers to a situation where there is an insufficient replication factor or replication strategy implemented on a distributed system like a Cassandra cluster. This can result in data loss if one or more nodes fail. Without adequate replication, the data stored on the failed nodes cannot be retrieved, leading to a substantial data loss. This incident requires immediate attention to ensure that the replication factor and strategy are optimized to prevent data loss in the future.

### Parameters
```shell
export NODE_IP="PLACEHOLDER"

export DATA_CENTER="PLACEHOLDER"

export CASSANDRA_PASSWORD="PLACEHOLDER"

export KEYSPACE_NAME="PLACEHOLDER"

export TABLE_NAME="PLACEHOLDER"

export DESIRED_REPLICATION_FACTOR="PLACEHOLDER"

export CASSANDRA_USERNAME="PLACEHOLDER"

export CASSANDRA_HOST="PLACEHOLDER"
```

## Debug

### Check the number of nodes in the Cassandra cluster
```shell
nodetool status
```

### Check the replication factor of the affected keyspace
```shell
nodetool getendpoints ${KEYSPACE_NAME} ${KEY}
```

### Check the replication strategy of the affected keyspace
```shell
nodetool describecluster | grep ${KEYSPACE_NAME}
```

### Check the logs for any warnings or errors related to inadequate replication
```shell
grep -i "inadequate replication" /var/log/cassandra/system.log
```

### Check the status of the affected node(s)
```shell
nodetool status ${NODE_IP}
```

### Check the status of the affected data center(s)
```shell
nodetool status --dc ${DATA_CENTER}
```

### Check the health of the Cassandra cluster
```shell
nodetool health
```

### Check the consistency level of the affected queries
```shell
grep -r "consistency" /etc/cassandra/
```

### Check the cluster and node configuration files for any replication-related issues
```shell
grep -r "replication" /etc/cassandra/
```

## Repair

### Increase the replication factor: Increase the number of replicas for each data center in the Cassandra cluster. This ensures that multiple copies of the data are stored across different nodes, which reduces the risk of data loss if a node fails.
```shell


#!/bin/bash



# Set the replication factor

REPLICATION_FACTOR=${DESIRED_REPLICATION_FACTOR}



# Set the keyspace and table names

KEYSPACE=${KEYSPACE_NAME}

TABLE=${TABLE_NAME}



# Connect to the Cassandra cluster

cqlsh ${CASSANDRA_HOST} -u ${CASSANDRA_USERNAME} -p ${CASSANDRA_PASSWORD}



# Alter the keyspace with the new replication factor

ALTER KEYSPACE $KEYSPACE_NAME WITH replication = {'class': 'SimpleStrategy', 'replication_factor': $REPLICATION_FACTOR};



# Alter the table with the new replication factor

ALTER TABLE $KEYSPACE_NAME.$TABLE WITH replication = {'class': 'SimpleStrategy', 'replication_factor': $REPLICATION_FACTOR};



# Exit cqlsh

exit


```

Inadequate replication refers to a situation where there is an insufficient replication factor or replication strategy implemented on a distributed system like a Cassandra cluster. This can result in data loss if one or more nodes fail. Without adequate replication, the data stored on the failed nodes cannot be retrieved, leading to a substantial data loss. This incident requires immediate attention to ensure that the replication factor and strategy are optimized to prevent data loss in the future.


This incident type refers to when a Kafka node runs out of storage space to store data. This can cause issues with data processing and storage, potentially leading to data loss and system downtime.


Kafka node insufficient storage check

In a Kafka cluster, replication factor is the number of copies of a topic's partitions across different brokers. The replication factor is set at the time of topic creation and cannot be changed once the topic is created. In case the replication factor is less than the minimum required number, there is a risk of data loss if one or more brokers go down. This incident type involves checking the replication factor of Kafka clusters to ensure that the number of replicas is sufficient, and that data is safe in case of a broker failure.


Kafka Cluster Replication Factor Check

This incident type refers to an alert triggered by the Kafka system when one or more topics have under-replicated data due to an insufficient number of in-sync replicas. In-sync replicas are replicas that are fully caught up with the leader replica and are considered reliable for serving data to consumers. When the number of in-sync replicas drops below a certain threshold, the Kafka system raises an alert to notify administrators to investigate and take corrective action to ensure data consistency and availability.


Kafka In Sync Replica Alert for Under-Replicated Topics

This incident type refers to situations where data inconsistencies arise due to issues with Kafka broker replication. Kafka brokers are responsible for replicating data across multiple nodes, ensuring that data is available even if one node fails. If there are issues with this replication process, it can result in data inconsistencies, where different nodes have different versions of the same data. This can cause a range of issues, including errors, data corruption, and other problems that can impact the functionality and reliability of the software system.


Kafka Data Inconsistencies from Kafka Broker Replication Issues.

This incident type refers to a situation where the number of partitions for a Kafka topic is insufficient to handle the incoming data stream. This can lead to low throughput and/or increased write latency. In order to resolve this issue, it may be necessary to increase the number of Kafka topic partitions. Failure to address this issue can lead to degraded performance and potential data loss.


Inadequate Kafka Topic Partitions Incident

```shell
export NODE_IP="PLACEHOLDER"

export DATA_CENTER="PLACEHOLDER"

export CASSANDRA_PASSWORD="PLACEHOLDER"

export KEYSPACE_NAME="PLACEHOLDER"

export TABLE_NAME="PLACEHOLDER"

export DESIRED_REPLICATION_FACTOR="PLACEHOLDER"

export CASSANDRA_USERNAME="PLACEHOLDER"

export CASSANDRA_HOST="PLACEHOLDER"
```


### Check the number of nodes in the Cassandra cluster

```shell
nodetool status
```

### Check the replication factor of the affected keyspace

```shell
nodetool getendpoints ${KEYSPACE_NAME} ${KEY}
```

### Check the replication strategy of the affected keyspace

```shell
nodetool describecluster | grep ${KEYSPACE_NAME}
```

### Check the logs for any warnings or errors related to inadequate replication

```shell
grep -i "inadequate replication" /var/log/cassandra/system.log
```

### Check the status of the affected node(s)

```shell
nodetool status ${NODE_IP}
```

### Check the status of the affected data center(s)

```shell
nodetool status --dc ${DATA_CENTER}
```

### Check the health of the Cassandra cluster

```shell
nodetool health
```

### Check the consistency level of the affected queries

```shell
grep -r "consistency" /etc/cassandra/
```

### Check the cluster and node configuration files for any replication-related issues

```shell
grep -r "replication" /etc/cassandra/
```


### Increase the replication factor: Increase the number of replicas for each data center in the Cassandra cluster. This ensures that multiple copies of the data are stored across different nodes, which reduces the risk of data loss if a node fails.

```shell


#!/bin/bash



# Set the replication factor

REPLICATION_FACTOR=${DESIRED_REPLICATION_FACTOR}



# Set the keyspace and table names

KEYSPACE=${KEYSPACE_NAME}

TABLE=${TABLE_NAME}



# Connect to the Cassandra cluster

cqlsh ${CASSANDRA_HOST} -u ${CASSANDRA_USERNAME} -p ${CASSANDRA_PASSWORD}



# Alter the keyspace with the new replication factor

ALTER KEYSPACE $KEYSPACE_NAME WITH replication = {'class': 'SimpleStrategy', 'replication_factor': $REPLICATION_FACTOR};



# Alter the table with the new replication factor

ALTER TABLE $KEYSPACE_NAME.$TABLE WITH replication = {'class': 'SimpleStrategy', 'replication_factor': $REPLICATION_FACTOR};



# Exit cqlsh

exit


```


Inadequate Replication on Cassandra Cluster

Overview

Parameters

Debug

Check the number of nodes in the Cassandra cluster

Check the replication factor of the affected keyspace

Check the replication strategy of the affected keyspace

Check the status of the affected node(s)

Check the status of the affected data center(s)

Check the health of the Cassandra cluster

Check the consistency level of the affected queries

Repair

Increase the replication factor: Increase the number of replicas for each data center in the Cassandra cluster. This ensures that multiple copies of the data are stored across different nodes, which reduces the risk of data loss if a node fails.

Learn more

Related Runbooks

Kafka node insufficient storage check

Kafka Cluster Replication Factor Check

Kafka In Sync Replica Alert for Under-Replicated Topics

Kafka Data Inconsistencies from Kafka Broker Replication Issues.

Support