---
id: 656e98e4-7324-410e-ac72-8e27a8a8a995
---

# Cassandra Commitlog Pending Tasks Incident
---

The Cassandra Commitlog Pending Tasks incident type refers to an issue with pending tasks in the commitlog for a Cassandra database instance. This can potentially cause data loss and other performance issues if not resolved promptly. The incident is typically triggered by monitoring tools that detect high levels of pending tasks in the commitlog. It requires immediate attention and action from the responsible team to investigate and resolve the underlying issue.

### Parameters
```shell
# Environment Variables

export KEYSPACE="PLACEHOLDER"

export TABLE="PLACEHOLDER"

export COMMITLOG_DIRECTORY="PLACEHOLDER"

export NEW_COMMITLOG_SIZE="PLACEHOLDER"

export INSTANCE_NAME="PLACEHOLDER"

export CASSANDRA_HOST="PLACEHOLDER"

export CASSANDRA_PORT="PLACEHOLDER"
```

## Debug

### 1. Check the status of the Cassandra service
```shell
systemctl status cassandra
```

### 2. Check the Cassandra logs for any errors or warnings
```shell
tail -n 100 /var/log/cassandra/system.log
```

### 3. Check the commitlog directory for any pending tasks
```shell
ls -l /var/lib/cassandra/commitlog | grep -i "pending_tasks"
```

### 4. Check the Cassandra metrics for commitlog-related values
```shell
nodetool tablestats ${KEYSPACE}.${TABLE} | grep -i "commitlog"

nodetool cfstats | grep -i "commitlog"
```

### 5. Check the disk usage and availability on the Cassandra nodes
```shell
df -h
```

### Misconfiguration of the Cassandra database settings or cluster topology, causing issues with commitlog processing and pending tasks accumulation.
```shell
bash

#!/bin/bash



# Set the Cassandra instance host and port

HOST=${CASSANDRA_HOST}

PORT=${CASSANDRA_PORT}



# Check if Cassandra is running

if ! nc -z $HOST $PORT; then

  echo "Cassandra is not running"

  exit 1

fi



# Check the Cassandra cluster topology

if ! nodetool status | grep -q "UN"; then

  echo "Cassandra cluster is not healthy"

  exit 1

fi



# Check the Cassandra configuration settings

if ! grep -q "commitlog_total_space_in_mb" /etc/cassandra/cassandra.yaml; then

  echo "commitlog_total_space_in_mb setting is missing"

  exit 1

fi



# Check the current commitlog status

if ! nodetool cfstats | grep -q "Pending Tasks"; then

  echo "No pending tasks in the commitlog"

  exit 0

else

  echo "Pending tasks in the commitlog"

  exit 1

fi


```

## Repair

### Set variables
```shell
INSTANCE=${INSTANCE_NAME}

COMMITLOG_DIR=${COMMITLOG_DIRECTORY}

COMMITLOG_SIZE=${NEW_COMMITLOG_SIZE}
```

### Stop Cassandra service
```shell
sudo service cassandra stop
```

### Increase commitlog disk space
```shell
sudo truncate -s ${COMMITLOG_SIZE} ${COMMITLOG_DIR}/CommitLog-*
```

### Start Cassandra service
```shell
sudo service cassandra start
```

### Next Step
```shell
echo "Commitlog disk space increased to ${COMMITLOG_SIZE} for instance ${INSTANCE}."
```

### Restart the Cassandra instance to clear out the pending tasks and allow for a fresh start. This can be a quick fix but should not be relied upon as a long-term solution.
```shell


#!/bin/bash



# Stop the Cassandra service

sudo systemctl stop cassandra



# Wait for the service to stop

while sudo systemctl is-active cassandra; do

    sleep 1

done



# Clear out the commitlog directory

sudo rm -rf ${COMMITLOG_DIRECTORY}/*



# Start the Cassandra service

sudo systemctl start cassandra


```

The Cassandra Commitlog Pending Tasks incident type refers to an issue with pending tasks in the commitlog for a Cassandra database instance. This can potentially cause data loss and other performance issues if not resolved promptly. The incident is typically triggered by monitoring tools that detect high levels of pending tasks in the commitlog. It requires immediate attention and action from the responsible team to investigate and resolve the underlying issue.


The Vault too many pending tokens incident refers to an issue where the Vault server has too many pending tokens. This can happen when the number of tokens generated exceeds the limit of what the server can handle, causing a backlog of requests. As a result, users may experience difficulty accessing certain resources or functions that require token authentication. This incident type typically requires the attention of a system administrator or engineer to investigate and resolve the underlying cause, such as increasing the token limit or optimizing the token generation process.


Vault too many pending tokens incident on kubernetes

This incident type refers to a situation where the number of partitions for a Kafka topic is insufficient to handle the incoming data stream. This can lead to low throughput and/or increased write latency. In order to resolve this issue, it may be necessary to increase the number of Kafka topic partitions. Failure to address this issue can lead to degraded performance and potential data loss.


Inadequate Kafka Topic Partitions Incident

This incident type refers to an alert triggered by a monitoring system indicating that the number of pending tasks in ElasticSearch is high. This can be an issue because it may indicate that the system is overloaded and unable to process all the incoming tasks, which can result in performance degradation or even downtime. The incident needs to be investigated and resolved as soon as possible to ensure the system is functioning properly.


High number of pending tasks in ElasticSearch.

This incident type refers to a situation where there is a significant delay in the execution of queries on a Cassandra cluster. This delay can cause the system to become unresponsive and result in slower performance. It may be caused by a variety of factors such as an increase in traffic, inefficient queries, or hardware issues. The issue can impact the functionality of the system and requires immediate attention to prevent further disruption.


Slow Query Performance on Cassandra Cluster.

This incident type refers to a situation where a delay or slowness occurs in a system that uses Cassandra database due to the shared storage. Shared storage means multiple servers are accessing the same storage unit, and this can cause latency issues. This type of incident can lead to performance degradation, and it needs to be addressed promptly to ensure optimal system performance.


Latency Caused by Shared Storage in Cassandra

```shell
# Environment Variables

export KEYSPACE="PLACEHOLDER"

export TABLE="PLACEHOLDER"

export COMMITLOG_DIRECTORY="PLACEHOLDER"

export NEW_COMMITLOG_SIZE="PLACEHOLDER"

export INSTANCE_NAME="PLACEHOLDER"

export CASSANDRA_HOST="PLACEHOLDER"

export CASSANDRA_PORT="PLACEHOLDER"
```


### 1. Check the status of the Cassandra service

```shell
systemctl status cassandra
```

### 2. Check the Cassandra logs for any errors or warnings

```shell
tail -n 100 /var/log/cassandra/system.log
```

### 3. Check the commitlog directory for any pending tasks

```shell
ls -l /var/lib/cassandra/commitlog | grep -i "pending_tasks"
```

### 4. Check the Cassandra metrics for commitlog-related values

```shell
nodetool tablestats ${KEYSPACE}.${TABLE} | grep -i "commitlog"

nodetool cfstats | grep -i "commitlog"
```

### 5. Check the disk usage and availability on the Cassandra nodes

```shell
df -h
```

### Misconfiguration of the Cassandra database settings or cluster topology, causing issues with commitlog processing and pending tasks accumulation.

```shell
bash

#!/bin/bash



# Set the Cassandra instance host and port

HOST=${CASSANDRA_HOST}

PORT=${CASSANDRA_PORT}



# Check if Cassandra is running

if ! nc -z $HOST $PORT; then

  echo "Cassandra is not running"

  exit 1

fi



# Check the Cassandra cluster topology

if ! nodetool status | grep -q "UN"; then

  echo "Cassandra cluster is not healthy"

  exit 1

fi



# Check the Cassandra configuration settings

if ! grep -q "commitlog_total_space_in_mb" /etc/cassandra/cassandra.yaml; then

  echo "commitlog_total_space_in_mb setting is missing"

  exit 1

fi



# Check the current commitlog status

if ! nodetool cfstats | grep -q "Pending Tasks"; then

  echo "No pending tasks in the commitlog"

  exit 0

else

  echo "Pending tasks in the commitlog"

  exit 1

fi


```


### Set variables

```shell
INSTANCE=${INSTANCE_NAME}

COMMITLOG_DIR=${COMMITLOG_DIRECTORY}

COMMITLOG_SIZE=${NEW_COMMITLOG_SIZE}
```

### Stop Cassandra service

```shell
sudo service cassandra stop
```

### Increase commitlog disk space

```shell
sudo truncate -s ${COMMITLOG_SIZE} ${COMMITLOG_DIR}/CommitLog-*
```

### Start Cassandra service

```shell
sudo service cassandra start
```

### Next Step

```shell
echo "Commitlog disk space increased to ${COMMITLOG_SIZE} for instance ${INSTANCE}."
```

### Restart the Cassandra instance to clear out the pending tasks and allow for a fresh start. This can be a quick fix but should not be relied upon as a long-term solution.

```shell


#!/bin/bash



# Stop the Cassandra service

sudo systemctl stop cassandra



# Wait for the service to stop

while sudo systemctl is-active cassandra; do

    sleep 1

done



# Clear out the commitlog directory

sudo rm -rf ${COMMITLOG_DIRECTORY}/*



# Start the Cassandra service

sudo systemctl start cassandra


```


Cassandra Commitlog Pending Tasks Incident

Overview

Parameters

Debug

1. Check the status of the Cassandra service

2. Check the Cassandra logs for any errors or warnings

3. Check the commitlog directory for any pending tasks

5. Check the disk usage and availability on the Cassandra nodes

Misconfiguration of the Cassandra database settings or cluster topology, causing issues with commitlog processing and pending tasks accumulation.

Repair

Set variables

Stop Cassandra service

Increase commitlog disk space

Start Cassandra service

Next Step

Restart the Cassandra instance to clear out the pending tasks and allow for a fresh start. This can be a quick fix but should not be relied upon as a long-term solution.

Learn more

Related Runbooks

Vault too many pending tokens incident on kubernetes

Inadequate Kafka Topic Partitions Incident

High number of pending tasks in ElasticSearch.

Slow Query Performance on Cassandra Cluster.

Support