---
id: c13a38d8-cf7c-437d-a991-7d84ab649cc8
---

# RabbitMQ Unacknowledged Messages Alert
---

This incident type occurs when the ratio of unacknowledged to acknowledged messages in a RabbitMQ queue is higher than usual. This may indicate a problem in the message pipeline and requires investigation to ensure that messages are being processed properly. It is often accompanied by a query alert monitor triggered by a specific metric threshold.

### Parameters
```shell
# Environment Variables

export RABBITMQ_HOSTNAME="PLACEHOLDER"

export NEW_MEMORY_IN_KB="PLACEHOLDER"

export TOTAL_MEMORY_IN_KB="PLACEHOLDER"

export RABBITMQ_MEMORY_IN_KB="PLACEHOLDER"

export RABBITMQ_PASSWORD="PLACEHOLDER"

export QUEUE_NAME="PLACEHOLDER"

export RABBITMQ_USERNAME="PLACEHOLDER"
```

## Debug

### 1. Check RabbitMQ status
```shell
systemctl status rabbitmq-server
```

### 2. Check RabbitMQ logs for errors
```shell
journalctl -u rabbitmq-server -f
```

### 3. Check RabbitMQ queue metrics
```shell
rabbitmqctl list_queues name messages_ready messages_unacknowledged
```

### 4. Check for network issues or latency
```shell
ping ${RABBITMQ_HOSTNAME}
```

### 5. Check for disk space issues on RabbitMQ server
```shell
df -h
```

### 6. Check for CPU and memory usage on RabbitMQ server
```shell
top
```

### 7. Check for any process that may be blocking RabbitMQ
```shell
sudo netstat -ltnp | grep 5672
```

### 8. Check for any RabbitMQ plugins that may be causing issues
```shell
rabbitmq-plugins list
```

### 9. Check RabbitMQ configuration files for errors
```shell
sudo rabbitmqctl eval 'application:ensure_all_started(rabbit).' && sudo rabbitmqctl eval 'rabbit_config_checks:run().'
```

### 10. Check for any network configurations that may be causing issues
```shell
sudo rabbitmqctl report | grep listeners
```

## Repair

### Increase the resources allocated to RabbitMQ, such as increasing server memory or CPU, to improve its processing capacity.
```shell


#!/bin/bash



# Set variables

MEM_TOTAL=${TOTAL_MEMORY_IN_KB} # total memory of the server in KB

MEM_RABBITMQ=${RABBITMQ_MEMORY_IN_KB} # memory allocated to RabbitMQ in KB

MEM_NEW=$((MEM_TOTAL - MEM_RABBITMQ + ${NEW_MEMORY_IN_KB})) # new memory allocation in KB



# Update memory allocation

echo "Updating memory allocation for RabbitMQ..."

sed -i "s/{mem_limit, .*}/{mem_limit, $MEM_NEW}/" /etc/rabbitmq/rabbitmq.conf



# Restart RabbitMQ service

echo "Restarting RabbitMQ service..."

systemctl restart rabbitmq-server



echo "Memory allocation updated successfully."


```

### Check the RabbitMQ configuration settings to ensure that the message queue is properly configured and optimized for the volume of messages being processed.
```shell


#!/bin/bash



# Set the RabbitMQ connection parameters

RABBITMQ_HOST=${RABBITMQ_HOST}

RABBITMQ_PORT=5672

RABBITMQ_USERNAME=${RABBITMQ_USERNAME}

RABBITMQ_PASSWORD=${RABBITMQ_PASSWORD}



# Check the RabbitMQ configuration settings

rabbitmqctl status

rabbitmqctl list_queues

rabbitmqctl list_exchanges

rabbitmqctl list_bindings



# Check the RabbitMQ configuration parameters for the message queue

rabbitmqctl list_queue ${QUEUE_NAME}

rabbitmqctl list_queue_bindings ${QUEUE_NAME}



# Check the RabbitMQ configuration parameters for all queues

rabbitmqctl list_queues name messages_ready messages_unacknowledged


```


This incident type occurs when the ratio of unacknowledged to acknowledged messages in a RabbitMQ queue is higher than usual. This may indicate a problem in the message pipeline and requires investigation to ensure that messages are being processed properly. It is often accompanied by a query alert monitor triggered by a specific metric threshold.


This incident type refers to a situation where the Tomcat server's JDBC connection pool usage surpasses its configured limits. This can lead to a shortage of available connections for incoming requests, resulting in slow response times or even server crashes. It is important to monitor and manage the JDBC connection pool usage to prevent such incidents from occurring.


Tomcat High JDBC Connection Pool Usage Incident.

This incident type refers to a situation where Redis, a popular in-memory data structure store, has rejected connections. This could happen due to various reasons such as server overload, network issues, or misconfiguration. When this occurs, it can cause disruptions in the performance of applications that rely on Redis for data storage or caching. Thus, it requires immediate attention and resolution to minimize the impact on the applications and users.


Redis Rejected Connections Incident

A Rabbitmq memory high incident occurs when a node uses more than 90% of the allocated RAM. This can cause performance issues and impact the overall functionality of the system. It is important to promptly address this issue to prevent any further damage.


Rabbitmq memory high incident.

This incident type refers to an alert triggered when the available disk space on RabbitMQ is low. This can be caused by an overload of messages or persistent messages that are being mismanaged. It is important to resolve this issue promptly to ensure the proper functioning of RabbitMQ and avoid any potential downtime.


Low Disk Space on RabbitMQ

```shell
export KAFKA_BROKER_IP="PLACEHOLDER"

export PORT="PLACEHOLDER"

export TOPIC_NAME="PLACEHOLDER"

export ZOOKEEPER_IP="PLACEHOLDER"

export NEW_RETENTION_TIME="PLACEHOLDER"

export GROUP_NAME="PLACEHOLDER"

export PATH_TO_KAFKA_CONFIG_FILE="PLACEHOLDER"
```


Parameters

```shell
# Environment Variables

export RABBITMQ_HOSTNAME="PLACEHOLDER"

export NEW_MEMORY_IN_KB="PLACEHOLDER"

export TOTAL_MEMORY_IN_KB="PLACEHOLDER"

export RABBITMQ_MEMORY_IN_KB="PLACEHOLDER"

export RABBITMQ_PASSWORD="PLACEHOLDER"

export QUEUE_NAME="PLACEHOLDER"

export RABBITMQ_USERNAME="PLACEHOLDER"
```


### 1. Check RabbitMQ status

```shell
systemctl status rabbitmq-server
```

### 2. Check RabbitMQ logs for errors

```shell
journalctl -u rabbitmq-server -f
```

### 3. Check RabbitMQ queue metrics

```shell
rabbitmqctl list_queues name messages_ready messages_unacknowledged
```

### 4. Check for network issues or latency

```shell
ping ${RABBITMQ_HOSTNAME}
```

### 5. Check for disk space issues on RabbitMQ server

```shell
df -h
```

### 6. Check for CPU and memory usage on RabbitMQ server

```shell
top
```

### 7. Check for any process that may be blocking RabbitMQ

```shell
sudo netstat -ltnp | grep 5672
```

### 8. Check for any RabbitMQ plugins that may be causing issues

```shell
rabbitmq-plugins list
```

### 9. Check RabbitMQ configuration files for errors

```shell
sudo rabbitmqctl eval 'application:ensure_all_started(rabbit).' && sudo rabbitmqctl eval 'rabbit_config_checks:run().'
```

### 10. Check for any network configurations that may be causing issues

```shell
sudo rabbitmqctl report | grep listeners
```


### Increase the resources allocated to RabbitMQ, such as increasing server memory or CPU, to improve its processing capacity.

```shell


#!/bin/bash



# Set variables

MEM_TOTAL=${TOTAL_MEMORY_IN_KB} # total memory of the server in KB

MEM_RABBITMQ=${RABBITMQ_MEMORY_IN_KB} # memory allocated to RabbitMQ in KB

MEM_NEW=$((MEM_TOTAL - MEM_RABBITMQ + ${NEW_MEMORY_IN_KB})) # new memory allocation in KB



# Update memory allocation

echo "Updating memory allocation for RabbitMQ..."

sed -i "s/{mem_limit, .*}/{mem_limit, $MEM_NEW}/" /etc/rabbitmq/rabbitmq.conf



# Restart RabbitMQ service

echo "Restarting RabbitMQ service..."

systemctl restart rabbitmq-server



echo "Memory allocation updated successfully."


```

### Check the RabbitMQ configuration settings to ensure that the message queue is properly configured and optimized for the volume of messages being processed.

```shell


#!/bin/bash



# Set the RabbitMQ connection parameters

RABBITMQ_HOST=${RABBITMQ_HOST}

RABBITMQ_PORT=5672

RABBITMQ_USERNAME=${RABBITMQ_USERNAME}

RABBITMQ_PASSWORD=${RABBITMQ_PASSWORD}



# Check the RabbitMQ configuration settings

rabbitmqctl status

rabbitmqctl list_queues

rabbitmqctl list_exchanges

rabbitmqctl list_bindings



# Check the RabbitMQ configuration parameters for the message queue

rabbitmqctl list_queue ${QUEUE_NAME}

rabbitmqctl list_queue_bindings ${QUEUE_NAME}



# Check the RabbitMQ configuration parameters for all queues

rabbitmqctl list_queues name messages_ready messages_unacknowledged


```


RabbitMQ Unacknowledged Messages Alert

Overview

Parameters

Debug

1. Check RabbitMQ status

2. Check RabbitMQ logs for errors

3. Check RabbitMQ queue metrics

4. Check for network issues or latency

5. Check for disk space issues on RabbitMQ server

6. Check for CPU and memory usage on RabbitMQ server

7. Check for any process that may be blocking RabbitMQ

8. Check for any RabbitMQ plugins that may be causing issues

9. Check RabbitMQ configuration files for errors

10. Check for any network configurations that may be causing issues

Repair

Increase the resources allocated to RabbitMQ, such as increasing server memory or CPU, to improve its processing capacity.

Check the RabbitMQ configuration settings to ensure that the message queue is properly configured and optimized for the volume of messages being processed.

Learn more

Related Runbooks

Tomcat High JDBC Connection Pool Usage Incident.

Redis Rejected Connections Incident

Rabbitmq memory high incident.

Low Disk Space on RabbitMQ

Support