---
id: 6e06b6bd-4679-4a62-a1b0-5bee0654648f
---
# Rabbitmq memory high incident.
---

A Rabbitmq memory high incident occurs when a node uses more than 90% of the allocated RAM. This can cause performance issues and impact the overall functionality of the system. It is important to promptly address this issue to prevent any further damage.

### Parameters
```shell
export RABBITMQ_PID="PLACEHOLDER"

export RABBITMQ_PID_FILE_PATH="PLACEHOLDER"

export RABBITMQ_LOG_FILE_PATH="PLACEHOLDER"

export RABBITMQ_MAX_CONNECTIONS="PLACEHOLDER"

export NEW_MEMORY_LIMIT="PLACEHOLDER"

export OLD_MEMORY_LIMIT="PLACEHOLDER"
```

## Debug

### Check the status of the Rabbitmq service
```shell
sudo systemctl status rabbitmq-server
```

### Check the memory usage of the Rabbitmq process
```shell
sudo ps aux | grep rabbitmq
```

### Check the memory usage of the Rabbitmq process using pidstat
```shell
sudo pidstat -r -p ${RABBITMQ_PID}
```

### Check the Rabbitmq log for any errors related to memory usage
```shell
sudo tail -f /var/log/rabbitmq/rabbit*.log | grep "memory"
```

### Check the system memory usage to see if there is any other process consuming too much memory
```shell
sudo free -h
```

### Check the Rabbitmq queue size and message count
```shell
sudo rabbitmqctl list_queues
```

### Restart the Rabbitmq service to clear any memory leaks
```shell
sudo systemctl restart rabbitmq-server
```

### An increase in the number of connections to the Rabbitmq server, causing it to consume more memory resources.
```shell
bash

#!/bin/bash



# Define variables

RABBITMQ_PID_FILE=${RABBITMQ_PID_FILE_PATH}

RABBITMQ_LOG_FILE=${RABBITMQ_LOG_FILE_PATH}

RABBITMQ_MAX_CONNECTIONS=${RABBITMQ_MAX_CONNECTIONS}



# Get the current number of open connections to Rabbitmq

CURRENT_CONNECTIONS=$(rabbitmqctl list_connections | grep -v "total" | wc -l)



# Check if the current number of connections is greater than the maximum allowed connections

if [[ "$CURRENT_CONNECTIONS" -gt "$RABBITMQ_MAX_CONNECTIONS" ]]; then

    echo "Number of open connections ($CURRENT_CONNECTIONS) is greater than the maximum allowed connections ($RABBITMQ_MAX_CONNECTIONS)." >> "$RABBITMQ_LOG_FILE"

    

    # Get the process ID of the Rabbitmq server

    RABBITMQ_PID=$(cat "$RABBITMQ_PID_FILE")

    

    # Get the current memory usage of the Rabbitmq server

    MEMORY_USAGE=$(ps -p $RABBITMQ_PID -o rss | grep -v RSS)

    

    echo "Current memory usage of Rabbitmq server: $MEMORY_USAGE" >> "$RABBITMQ_LOG_FILE"

    

    # Restart the Rabbitmq server to free up memory resources

    systemctl restart rabbitmq-server

    

    echo "Rabbitmq server restarted." >> "$RABBITMQ_LOG_FILE"

fi


```

## Repair

### Increase the memory allocation for Rabbitmq by adding more RAM or increasing the memory limit.
```shell
bash

#!/bin/bash



# Set the new memory limit value

NEW_MEMORY_LIMIT=${NEW_MEMORY_LIMIT}



# Stop Rabbitmq

sudo service rabbitmq stop



# Modify the memory limit in the configuration file

sudo sed -i "s/MEMORY_LIMIT=${OLD_MEMORY_LIMIT}/MEMORY_LIMIT=$NEW_MEMORY_LIMIT/g" /etc/rabbitmq/rabbitmq-env.conf



# Start Rabbitmq

sudo service rabbitmq start


```



A Rabbitmq memory high incident occurs when a node uses more than 90% of the allocated RAM. This can cause performance issues and impact the overall functionality of the system. It is important to promptly address this issue to prevent any further damage.


This incident type occurs when the PermGen or Metaspace usage of the Tomcat server exceeds 90%. PermGen and Metaspace are memory pools used for storing class definitions and metadata of the Java virtual machine (JVM). When these memory pools reach their maximum capacity, the server may experience performance issues or even crash. This incident requires immediate attention from the software engineer to investigate the root cause and take appropriate actions to resolve the issue.


Tomcat PermGen or Metaspace Usage exceeds 90%

Tomcat JVM OutOfMemory incident occurs when the Java Virtual Machine (JVM) running the Apache Tomcat server runs out of memory, which causes the server to crash or become unresponsive. This type of incident can happen due to a variety of reasons such as incorrect configuration settings, memory leaks, or insufficient memory allocation. When this incident occurs, it can impact the availability and performance of the web application running on the Tomcat server.


Tomcat JVM OutOfMemory Incident

This incident type refers to a situation where the Tomcat server's JDBC connection pool usage surpasses its configured limits. This can lead to a shortage of available connections for incoming requests, resulting in slow response times or even server crashes. It is important to monitor and manage the JDBC connection pool usage to prevent such incidents from occurring.


Tomcat High JDBC Connection Pool Usage Incident.

This incident type refers to an increase in the number of errors per second on a Tomcat server, which could indicate an issue with the server itself, the host, a deployed application, or an application servlet. This could include errors generated when the Tomcat server runs out of memory, can't find a requested file or servlet, or is unable to serve a JSP due to syntax errors in the servlet codebase. This incident type requires immediate attention to diagnose and address the underlying issue.


Increase of the errors/second rate for Tomcat server

This incident type refers to the situation where the Tomcat server is utilizing more memory than it should, causing system instability and performance issues. This can happen due to a variety of factors such as memory leaks, improper configuration, or application code issues. It is important to address this incident promptly to prevent system failures and ensure optimal application performance.


Excessive memory usage by Tomcat.

```shell
export RABBITMQ_PID="PLACEHOLDER"

export RABBITMQ_PID_FILE_PATH="PLACEHOLDER"

export RABBITMQ_LOG_FILE_PATH="PLACEHOLDER"

export RABBITMQ_MAX_CONNECTIONS="PLACEHOLDER"

export NEW_MEMORY_LIMIT="PLACEHOLDER"

export OLD_MEMORY_LIMIT="PLACEHOLDER"
```


### Check the status of the Rabbitmq service

```shell
sudo systemctl status rabbitmq-server
```

### Check the memory usage of the Rabbitmq process

```shell
sudo ps aux | grep rabbitmq
```

### Check the memory usage of the Rabbitmq process using pidstat

```shell
sudo pidstat -r -p ${RABBITMQ_PID}
```

### Check the Rabbitmq log for any errors related to memory usage

```shell
sudo tail -f /var/log/rabbitmq/rabbit*.log | grep "memory"
```

### Check the system memory usage to see if there is any other process consuming too much memory

```shell
sudo free -h
```

### Check the Rabbitmq queue size and message count

```shell
sudo rabbitmqctl list_queues
```

### Restart the Rabbitmq service to clear any memory leaks

```shell
sudo systemctl restart rabbitmq-server
```

### An increase in the number of connections to the Rabbitmq server, causing it to consume more memory resources.

```shell
bash

#!/bin/bash



# Define variables

RABBITMQ_PID_FILE=${RABBITMQ_PID_FILE_PATH}

RABBITMQ_LOG_FILE=${RABBITMQ_LOG_FILE_PATH}

RABBITMQ_MAX_CONNECTIONS=${RABBITMQ_MAX_CONNECTIONS}



# Get the current number of open connections to Rabbitmq

CURRENT_CONNECTIONS=$(rabbitmqctl list_connections | grep -v "total" | wc -l)



# Check if the current number of connections is greater than the maximum allowed connections

if [[ "$CURRENT_CONNECTIONS" -gt "$RABBITMQ_MAX_CONNECTIONS" ]]; then

    echo "Number of open connections ($CURRENT_CONNECTIONS) is greater than the maximum allowed connections ($RABBITMQ_MAX_CONNECTIONS)." >> "$RABBITMQ_LOG_FILE"

    

    # Get the process ID of the Rabbitmq server

    RABBITMQ_PID=$(cat "$RABBITMQ_PID_FILE")

    

    # Get the current memory usage of the Rabbitmq server

    MEMORY_USAGE=$(ps -p $RABBITMQ_PID -o rss | grep -v RSS)

    

    echo "Current memory usage of Rabbitmq server: $MEMORY_USAGE" >> "$RABBITMQ_LOG_FILE"

    

    # Restart the Rabbitmq server to free up memory resources

    systemctl restart rabbitmq-server

    

    echo "Rabbitmq server restarted." >> "$RABBITMQ_LOG_FILE"

fi


```


### Increase the memory allocation for Rabbitmq by adding more RAM or increasing the memory limit.

```shell
bash

#!/bin/bash



# Set the new memory limit value

NEW_MEMORY_LIMIT=${NEW_MEMORY_LIMIT}



# Stop Rabbitmq

sudo service rabbitmq stop



# Modify the memory limit in the configuration file

sudo sed -i "s/MEMORY_LIMIT=${OLD_MEMORY_LIMIT}/MEMORY_LIMIT=$NEW_MEMORY_LIMIT/g" /etc/rabbitmq/rabbitmq-env.conf



# Start Rabbitmq

sudo service rabbitmq start


```


Rabbitmq memory high incident.

Overview

Parameters

Debug

Check the status of the Rabbitmq service

Check the memory usage of the Rabbitmq process

Check the memory usage of the Rabbitmq process using pidstat

Check the system memory usage to see if there is any other process consuming too much memory

Check the Rabbitmq queue size and message count

Restart the Rabbitmq service to clear any memory leaks

An increase in the number of connections to the Rabbitmq server, causing it to consume more memory resources.

Repair

Increase the memory allocation for Rabbitmq by adding more RAM or increasing the memory limit.

Learn more

Related Runbooks

Tomcat PermGen or Metaspace Usage exceeds 90%

Tomcat JVM OutOfMemory Incident

Tomcat High JDBC Connection Pool Usage Incident.

Increase of the errors/second rate for Tomcat server

Support