---
id: c90d6eaa-616e-11ee-8c99-0242ac120002
---

# Performance Degradation Due to New Generation Garbage Collection on Cassandra
---

This incident type refers to a performance degradation issue in a Cassandra database caused by the new generation garbage collection (GC) process. Garbage collection is a process by which the unused memory occupied by objects in a program is cleared. In some cases, the new generation GC process may take longer than usual, causing performance issues in the Cassandra database. This could lead to slowness in queries or even crashing of the database.

### Parameters
```shell
export PATH_TO_CASSANDRA_HOME="PLACEHOLDER"

export NEW_HEAP_SIZE="PLACEHOLDER"

export HEAP_SIZE="PLACEHOLDER"

export CASSANDRA_HOME="PLACEHOLDER"

export PATH_TO_CASSANDRA_INSTALLATION_DIRECTORY="PLACEHOLDER"
```

## Debug

### Check Cassandra process status
```shell
systemctl status cassandra
```

### Check Cassandra logs for any garbage collection errors
```shell
journalctl -u cassandra -n 100 | grep -i gc
```

### Check the current garbage collection settings for Cassandra
```shell
nodetool getlogginglevels | grep -i gc
```

### Check the current heap usage of Cassandra
```shell
nodetool info | grep -i heap
```

### Check the garbage collection logs for detailed information
```shell
cat /var/log/cassandra/gc.log
```

### Check the JVM arguments being used by Cassandra
```shell
ps aux | grep -i cassandra | grep -v grep | awk '{print $11}' | xargs -I{} sh -c '{} -version'
```

### Check the configuration file for Cassandra
```shell
cat /etc/cassandra/cassandra.yaml | grep -i gc
```

### Check the current Java version being used by Cassandra
```shell
java -version
```

## Repair

### Increase the heap size: One possible solution is to increase the heap size of the Cassandra database. This provides more memory space for the garbage collector to work with, which can reduce the time it takes to perform garbage collection.
```shell
bash

#!/bin/bash

# Set variables

CASSANDRA_HOME=${PATH_TO_CASSANDRA_HOME}

HEAP_SIZE=${NEW_HEAP_SIZE}



# Update Cassandra configuration

sed -i "s|JVM_OPTS=\"\$JVM_OPTS -Xmx[0-9]*M\"|JVM_OPTS=\"\$JVM_OPTS -Xmx${HEAP_SIZE}M\"|" ${CASSANDRA_HOME}/conf/cassandra-env.sh



# Restart Cassandra

sudo systemctl restart cassandra


```

### Adjust the garbage collection settings: The garbage collection settings can be adjusted to optimize performance. This can include adjusting the frequency of garbage collection or changing the garbage collector algorithm.
```shell
bash

#!/bin/bash



# Set the Cassandra installation directory

CASSANDRA_HOME=${PATH_TO_CASSANDRA_INSTALLATION_DIRECTORY}



# Set the garbage collection settings

GC_OPTS="-XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:InitiatingHeapOccupancyPercent=70"



# Update the Cassandra configuration file

sed -i "s/#JVM_OPTS=\"\$JVM_OPTS -XX:+UseG1GC\"/JVM_OPTS=\"\$JVM_OPTS $GC_OPTS\"/g" $CASSANDRA_HOME/conf/cassandra-env.sh



# Restart Cassandra service

systemctl restart cassandra


```


This incident type refers to a performance degradation issue in a Cassandra database caused by the new generation garbage collection (GC) process. Garbage collection is a process by which the unused memory occupied by objects in a program is cleared. In some cases, the new generation GC process may take longer than usual, causing performance issues in the Cassandra database. This could lead to slowness in queries or even crashing of the database.


Tomcat JVM OutOfMemory incident occurs when the Java Virtual Machine (JVM) running the Apache Tomcat server runs out of memory, which causes the server to crash or become unresponsive. This type of incident can happen due to a variety of reasons such as incorrect configuration settings, memory leaks, or insufficient memory allocation. When this incident occurs, it can impact the availability and performance of the web application running on the Tomcat server.


Tomcat JVM OutOfMemory Incident

Tomcat is a widely used open source web server and servlet container software. Full GC (Garbage Collection) is a process that runs in Java Virtual Machine (JVM) to reclaim memory occupied by objects that are no longer in use. When Full GC events happen too frequently, it can cause performance issues and lead to application downtime. This incident type refers to the occurrence of frequent Full GC events in Tomcat.


Tomcat Frequent Full GC Events

This incident type describes an issue where the efficiency of queries in MongoDB is degrading. The incident may be triggered by an alert monitoring system that detects an anomaly in the performance of queries executed in the database. The incident may result in a decrease in system performance and may require the attention of a software engineer to troubleshoot and resolve the issue.


MongoDB Efficiency of queries is degrading

This incident type refers to a problem in a Cassandra cluster where the token range imbalances cause uneven distribution of data across the cluster. This can result in slower read and write performance that can impact the overall functionality of the system. Token range imbalances occur when the distribution of the tokens that define the ranges of data each node is responsible for is not evenly spread across the cluster. As a result, certain nodes may be responsible for a disproportionate amount of data, leading to performance issues and potential failure of the system.


Token Range Imbalances Causing Uneven Data Distribution and Performance Issues in Cassandra Cluster

This incident type refers to a situation where there is a significant delay in the execution of queries on a Cassandra cluster. This delay can cause the system to become unresponsive and result in slower performance. It may be caused by a variety of factors such as an increase in traffic, inefficient queries, or hardware issues. The issue can impact the functionality of the system and requires immediate attention to prevent further disruption.


Slow Query Performance on Cassandra Cluster.

```shell
export PATH_TO_CASSANDRA_HOME="PLACEHOLDER"

export NEW_HEAP_SIZE="PLACEHOLDER"

export HEAP_SIZE="PLACEHOLDER"

export CASSANDRA_HOME="PLACEHOLDER"

export PATH_TO_CASSANDRA_INSTALLATION_DIRECTORY="PLACEHOLDER"
```


### Check Cassandra process status

```shell
systemctl status cassandra
```

### Check Cassandra logs for any garbage collection errors

```shell
journalctl -u cassandra -n 100 | grep -i gc
```

### Check the current garbage collection settings for Cassandra

```shell
nodetool getlogginglevels | grep -i gc
```

### Check the current heap usage of Cassandra

```shell
nodetool info | grep -i heap
```

### Check the garbage collection logs for detailed information

```shell
cat /var/log/cassandra/gc.log
```

### Check the JVM arguments being used by Cassandra

```shell
ps aux | grep -i cassandra | grep -v grep | awk '{print $11}' | xargs -I{} sh -c '{} -version'
```

### Check the configuration file for Cassandra

```shell
cat /etc/cassandra/cassandra.yaml | grep -i gc
```

### Check the current Java version being used by Cassandra

```shell
java -version
```


### Increase the heap size: One possible solution is to increase the heap size of the Cassandra database. This provides more memory space for the garbage collector to work with, which can reduce the time it takes to perform garbage collection.

```shell
bash

#!/bin/bash

# Set variables

CASSANDRA_HOME=${PATH_TO_CASSANDRA_HOME}

HEAP_SIZE=${NEW_HEAP_SIZE}



# Update Cassandra configuration

sed -i "s|JVM_OPTS=\"\$JVM_OPTS -Xmx[0-9]*M\"|JVM_OPTS=\"\$JVM_OPTS -Xmx${HEAP_SIZE}M\"|" ${CASSANDRA_HOME}/conf/cassandra-env.sh



# Restart Cassandra

sudo systemctl restart cassandra


```

### Adjust the garbage collection settings: The garbage collection settings can be adjusted to optimize performance. This can include adjusting the frequency of garbage collection or changing the garbage collector algorithm.

```shell
bash

#!/bin/bash



# Set the Cassandra installation directory

CASSANDRA_HOME=${PATH_TO_CASSANDRA_INSTALLATION_DIRECTORY}



# Set the garbage collection settings

GC_OPTS="-XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:InitiatingHeapOccupancyPercent=70"



# Update the Cassandra configuration file

sed -i "s/#JVM_OPTS=\"\$JVM_OPTS -XX:+UseG1GC\"/JVM_OPTS=\"\$JVM_OPTS $GC_OPTS\"/g" $CASSANDRA_HOME/conf/cassandra-env.sh



# Restart Cassandra service

systemctl restart cassandra


```


Performance Degradation Due to New Generation Garbage Collection on Cassandra

Overview

Parameters

Debug

Check Cassandra process status

Check Cassandra logs for any garbage collection errors

Check the current garbage collection settings for Cassandra

Check the current heap usage of Cassandra

Check the garbage collection logs for detailed information

Check the JVM arguments being used by Cassandra

Check the configuration file for Cassandra

Check the current Java version being used by Cassandra

Repair

Increase the heap size: One possible solution is to increase the heap size of the Cassandra database. This provides more memory space for the garbage collector to work with, which can reduce the time it takes to perform garbage collection.

Adjust the garbage collection settings: The garbage collection settings can be adjusted to optimize performance. This can include adjusting the frequency of garbage collection or changing the garbage collector algorithm.

Learn more

Related Runbooks

Tomcat JVM OutOfMemory Incident

Tomcat Frequent Full GC Events

MongoDB Efficiency of queries is degrading

Token Range Imbalances Causing Uneven Data Distribution and Performance Issues in Cassandra Cluster

Support