---
id: 32982e12-db13-419e-9480-ca191bd40b5c
---

# Misconfigured Compaction Strategy.
---

Misconfigured compaction strategy is an incident type that occurs when the way data is compacted within a database cluster is not properly configured. This can lead to excessive compaction activity that can negatively impact the cluster's performance and stability. The incident can cause slow query response times, increased disk usage, and high CPU utilization, among other issues. It is crucial to identify and fix the misconfiguration promptly to ensure the cluster's smooth operation.

### Parameters
```shell
export CASSANDRA_HOST="PLACEHOLDER"

export KEYSPACE="PLACEHOLDER"

export TABLE="PLACEHOLDER"

export DEVICE="PLACEHOLDER"

export PID="PLACEHOLDER"

```

## Debug

### 1. Check compaction status
```shell
nodetool compactionstats
```

### 2. Check compaction settings
```shell
nodetool describeconf compaction
```

### 8. Check if there are any Java exceptions related to compaction
```shell
grep -i "compaction" /var/log/cassandra/system.log | grep -i "exception"
```

### 4. Check if there are any compaction errors in the system log
```shell
grep "Compaction" /var/log/system.log
```

### 5. Check if there are any disk space issues
```shell
df -h
```

### 6. Check if there are any network issues
```shell
ping ${HOSTNAME}

traceroute ${HOSTNAME}
```

### 7. Check if there are any hardware issues
```shell
dmesg | grep -i error
```

### 3. Check if compaction is causing high CPU usage
```shell
top -bn1 | grep "Cpu(s)" | sed "s/.*, *$[0-9.]*$%* id.*/\1/" | awk '{print 100 - $1"% CPU usage"}'
```

### 9. Check if there are any other processes competing for resources
```shell
ps -eo pid,ppid,cmd,%mem,%cpu --sort=-%mem | head
```

### 10. Check if there are any compaction-related metrics exceeding the threshold
```shell
nodetool tablehistograms ${KEYSPACE}.${TABLE}
```

### 11. Check if there are any pending compactions
```shell
nodetool compactionstats -H | grep pending
```

### 12. Check if there are any tombstones causing excessive compaction
```shell
nodetool cfstats ${KEYSPACE}.${TABLE} | grep "Number of Tombstones"
```

### 13. Check if there are any disk I/O issues
```shell
iostat -xm 5 3 | grep ${DEVICE}
```

### 14. Check if there are any compaction-related threads stuck
```shell
jstack ${PID} | grep "CompactionExecutor" -A 20
```

### 15. Check if there are any file descriptor limits reached
```shell
ulimit -n

cat /proc/${PID}/limits | grep "Max open files"
```

### 16. Check if there are any memory usage issues
```shell
free -m

cat /proc/meminfo | grep MemAvailable
```

## Repair

### Identify and analyze the configuration settings of the compaction strategy and rectify the misconfiguration.
```shell


#!/bin/bash



# Set the variables

COMPACT_STRATEGY=${COMPACTION_STRATEGY}   # Replace with the name of the compaction strategy

CONFIG_FILE=${CONFIG_FILE_PATH}          # Replace with the path of the configuration file



# Identify and analyze the configuration settings

if grep -q "$COMPACT_STRATEGY" "$CONFIG_FILE"

then

  # Rectify the misconfiguration

  sed -i "s/$COMPACT_STRATEGY/${CORRECTED_VALUE}/" "$CONFIG_FILE"

  echo "Misconfiguration in $COMPACT_STRATEGY has been rectified."

else

  echo "$COMPACT_STRATEGY is already configured correctly."

fi



exit 0


```

Misconfigured compaction strategy is an incident type that occurs when the way data is compacted within a database cluster is not properly configured. This can lead to excessive compaction activity that can negatively impact the cluster's performance and stability. The incident can cause slow query response times, increased disk usage, and high CPU utilization, among other issues. It is crucial to identify and fix the misconfiguration promptly to ensure the cluster's smooth operation.


This incident type refers to a problem with a PostgreSQL database where it is reporting disk full errors. This means that the database has reached its full capacity and cannot store any more data. This can cause issues with the performance of the database and may result in data loss if not addressed promptly.


PostgreSQL database reporting disk full errors.

This incident type refers to cases where the MySQL service, which is responsible for managing and storing data in a database, runs out of available space. This can happen due to a variety of reasons such as an unexpected increase in data volume or insufficient allocation of storage resources. When the MySQL service runs out of space, it can cause data loss, application errors, and even system crashes if not addressed promptly.


MySQL Service Out of Space.

This incident type describes an issue where the efficiency of queries in MongoDB is degrading. The incident may be triggered by an alert monitoring system that detects an anomaly in the performance of queries executed in the database. The incident may result in a decrease in system performance and may require the attention of a software engineer to troubleshoot and resolve the issue.


MongoDB Efficiency of queries is degrading

The Kafka Data Loss Incident refers to an incident where there is a misconfiguration of the data retention policy in Kafka, which can result in data loss. Kafka is a distributed streaming platform that is commonly used in big data environments. When data retention policies are misconfigured, data that should be retained is deleted, or data that should be deleted is retained. This can lead to significant data loss and can have serious consequences for businesses that rely on the data stored in Kafka.


Kafka Data Loss Incident

Compaction is a process in Cassandra that merges multiple SSTables (Sorted String Tables) into a single SSTable, eliminating any redundant data and improving read performance. However, sometimes compaction can fail due to various reasons such as insufficient disk space or corrupted data, resulting in degraded performance or even complete failure of the database. Troubleshooting compaction merging SSTables involves identifying and resolving the root cause of such failures to ensure the smooth functioning of the Cassandra database.


Troubleshooting Compaction Merging SSTables in Cassandra

```shell
export CASSANDRA_HOST="PLACEHOLDER"

export KEYSPACE="PLACEHOLDER"

export TABLE="PLACEHOLDER"

export DEVICE="PLACEHOLDER"

export PID="PLACEHOLDER"

```


### 1. Check compaction status

```shell
nodetool compactionstats
```

### 2. Check compaction settings

```shell
nodetool describeconf compaction
```

### 8. Check if there are any Java exceptions related to compaction

```shell
grep -i "compaction" /var/log/cassandra/system.log | grep -i "exception"
```

### 4. Check if there are any compaction errors in the system log

```shell
grep "Compaction" /var/log/system.log
```

### 5. Check if there are any disk space issues

```shell
df -h
```

### 6. Check if there are any network issues

```shell
ping ${HOSTNAME}

traceroute ${HOSTNAME}
```

### 7. Check if there are any hardware issues

```shell
dmesg | grep -i error
```

### 3. Check if compaction is causing high CPU usage

```shell
top -bn1 | grep "Cpu(s)" | sed "s/.*, *$[0-9.]*$%* id.*/\1/" | awk '{print 100 - $1"% CPU usage"}'
```

### 9. Check if there are any other processes competing for resources

```shell
ps -eo pid,ppid,cmd,%mem,%cpu --sort=-%mem | head
```

### 10. Check if there are any compaction-related metrics exceeding the threshold

```shell
nodetool tablehistograms ${KEYSPACE}.${TABLE}
```

### 11. Check if there are any pending compactions

```shell
nodetool compactionstats -H | grep pending
```

### 12. Check if there are any tombstones causing excessive compaction

```shell
nodetool cfstats ${KEYSPACE}.${TABLE} | grep "Number of Tombstones"
```

### 13. Check if there are any disk I/O issues

```shell
iostat -xm 5 3 | grep ${DEVICE}
```

### 14. Check if there are any compaction-related threads stuck

```shell
jstack ${PID} | grep "CompactionExecutor" -A 20
```

### 15. Check if there are any file descriptor limits reached

```shell
ulimit -n

cat /proc/${PID}/limits | grep "Max open files"
```

### 16. Check if there are any memory usage issues

```shell
free -m

cat /proc/meminfo | grep MemAvailable
```


### Identify and analyze the configuration settings of the compaction strategy and rectify the misconfiguration.

```shell


#!/bin/bash



# Set the variables

COMPACT_STRATEGY=${COMPACTION_STRATEGY}   # Replace with the name of the compaction strategy

CONFIG_FILE=${CONFIG_FILE_PATH}          # Replace with the path of the configuration file



# Identify and analyze the configuration settings

if grep -q "$COMPACT_STRATEGY" "$CONFIG_FILE"

then

  # Rectify the misconfiguration

  sed -i "s/$COMPACT_STRATEGY/${CORRECTED_VALUE}/" "$CONFIG_FILE"

  echo "Misconfiguration in $COMPACT_STRATEGY has been rectified."

else

  echo "$COMPACT_STRATEGY is already configured correctly."

fi



exit 0


```


Misconfigured Compaction Strategy.

Overview

Parameters

Debug

1. Check compaction status

2. Check compaction settings

4. Check if there are any compaction errors in the system log

5. Check if there are any disk space issues

6. Check if there are any network issues

7. Check if there are any hardware issues

3. Check if compaction is causing high CPU usage

9. Check if there are any other processes competing for resources

11. Check if there are any pending compactions

12. Check if there are any tombstones causing excessive compaction

13. Check if there are any disk I/O issues

15. Check if there are any file descriptor limits reached

16. Check if there are any memory usage issues

Repair

Identify and analyze the configuration settings of the compaction strategy and rectify the misconfiguration.

Learn more

Related Runbooks

PostgreSQL database reporting disk full errors.

MySQL Service Out of Space.

MongoDB Efficiency of queries is degrading

Kafka Data Loss Incident

Support