---
id: dab19ac0-6501-454d-9f32-0f8efe34eb7e
---

# "Host Memory is underutilized incident"
---

This incident type refers to a situation where the host memory of a system is not being fully utilized and may be causing performance issues. It may trigger when the memory load is below the expected level for a certain period of time. The incident may require investigation to determine the root cause and may involve reducing the amount of memory or other memory management strategies to optimize system performance.

### Parameters
```shell
export CONFIGURED_AMOUNT_OF_MEMORY="PLACEHOLDER"

export HOST_VM_NAME="PLACEHOLDER"

export DECREASED_MEMORY_SIZE="PLACEHOLDER"
```

## Debug

### 1. Check the memory usage of the affected host
```shell
free -m
```

### 2. Check the current CPU load
```shell
top
```

### 6. Check the system load average
```shell
uptime
```

### 3. List all running processes and their memory usage
```shell
ps -aux --sort=-%mem | head
```

### 4. Check the amount of available disk space
```shell
df -h
```

### 5. Check the system logs for any relevant error messages
```shell
dmesg | tail
```

### 7. List all running Docker containers and their memory usage
```shell
docker stats $(docker ps --format={{.Names}})
```

### 8. Check the system swap usage
```shell
swapon -s
```

### 9. Check the system kernel version
```shell
uname -r
```

### 10. List all installed packages and their versions
```shell
dpkg -l
```

### The server may be overprovisioned with more memory than required for the workload it is running.
```shell


#!/bin/bash



# Get the total amount of memory installed on the system

total_memory=$(free -m | awk '/^Mem:/{print $2}')



# Get the amount of memory currently in use

used_memory=$(free -m | awk '/^Mem:/{print $3}')



# Get the percentage of memory that is currently in use

memory_percent=$(echo "scale=2; $used_memory / $total_memory * 100" | bc)



# Get the amount of memory that is free and available for use

available_memory=$(free -m | awk '/^Mem:/{print $4}')



# Compare the amount of available memory to the configured amount of memory

configured_memory=${CONFIGURED_AMOUNT_OF_MEMORY}

if [ $available_memory -gt $configured_memory ]; then

    echo "The server may be overprovisioned with more memory than required for the workload it is running."

else

    echo "The available memory is within the configured amount, so this is likely not the issue."

fi



# Output the current memory usage information for further analysis

echo "Total memory: $total_memory MB"

echo "Used memory: $used_memory MB ($memory_percent%)"

echo "Available memory: $available_memory MB"


```

## Repair

### Decrease memory allocation: Allocate less memory to the host vm to ensure that it does not have too much unused memory.
```shell
bash

#!/bin/bash



# Define variables

HOST_VM_NAME=${HOST_VM_NAME}

DECREASED_MEMORY_SIZE=${DECREASED_MEMORY_SIZE}



# Check if the host vm exists

if ! virsh list --name --all | grep -q "\${HOST_VM_NAME_}"; then

  echo "Error: Host VM $HOST_VM_NAME does not exist"

  exit 1

fi



# Check if the specified memory size is valid

if ! [[ $DECREASED_MEMORY_SIZE =~ ^[0-9]+$ ]]; then

  echo "Error: Invalid memory size $DECREASED_MEMORY_SIZE"

  exit 1

fi



# Stop the host vm

virsh shutdown $HOST_VM_NAME



# Decrease memory allocation for host vm

virsh setmem $HOST_VM_NAME $DECREASED_MEMORY_SIZE --config



# Start the host vm

virsh start $HOST_VM_NAME



# Verify that the memory allocation has been decreased

virsh dommemstat $HOST_VM_NAME | grep actual


```

This incident type refers to a situation where the host memory of a system is not being fully utilized and may be causing performance issues. It may trigger when the memory load is below the expected level for a certain period of time. The incident may require investigation to determine the root cause and may involve reducing the amount of memory or other memory management strategies to optimize system performance.


The Host Out of Memory (OOM) Incident occurs when a server or system runs out of memory, causing it to crash or become unresponsive. This can be caused by various factors, such as an unexpected surge in traffic or insufficient resources allocated to the system. Resolving this type of incident requires identifying the root cause of the memory issue and taking appropriate measures such as optimizing system resources or increasing memory capacity.


Host Out of Memory (OOM) Incident

This incident type refers to an increase in the number of errors per second on a Tomcat server, which could indicate an issue with the server itself, the host, a deployed application, or an application servlet. This could include errors generated when the Tomcat server runs out of memory, can't find a requested file or servlet, or is unable to serve a JSP due to syntax errors in the servlet codebase. This incident type requires immediate attention to diagnose and address the underlying issue.


Increase of the errors/second rate for Tomcat server

This incident type refers to the situation where the Tomcat server is utilizing more memory than it should, causing system instability and performance issues. This can happen due to a variety of factors such as memory leaks, improper configuration, or application code issues. It is important to address this incident promptly to prevent system failures and ensure optimal application performance.


Excessive memory usage by Tomcat.

This incident type refers to a situation where Spark tasks are failing due to out of memory errors. Spark is a distributed computing system used for big data processing. When the data volume exceeds the allocated memory, the Spark tasks fail, and the system generates an out of memory error. This type of incident can cause data processing delays or even system downtime, which can impact the overall performance of the application.


Spark tasks failing due to out of memory errors.

This incident type indicates that there is a high latency issue in the execution of a Spark job. Spark is a distributed computing framework that is used for processing large datasets. High latency in this context means that the time taken to execute the Spark job is significantly longer than expected or normal. This can result in delays in processing data and can impact the performance of the application or system that is utilizing Spark.


High Latency Incident for Spark Job Execution.

```shell
export CONFIGURED_AMOUNT_OF_MEMORY="PLACEHOLDER"

export HOST_VM_NAME="PLACEHOLDER"

export DECREASED_MEMORY_SIZE="PLACEHOLDER"
```


### 1. Check the memory usage of the affected host

```shell
free -m
```

### 2. Check the current CPU load

```shell
top
```

### 6. Check the system load average

```shell
uptime
```

### 3. List all running processes and their memory usage

```shell
ps -aux --sort=-%mem | head
```

### 4. Check the amount of available disk space

```shell
df -h
```

### 5. Check the system logs for any relevant error messages

```shell
dmesg | tail
```

### 7. List all running Docker containers and their memory usage

```shell
docker stats $(docker ps --format={{.Names}})
```

### 8. Check the system swap usage

```shell
swapon -s
```

### 9. Check the system kernel version

```shell
uname -r
```

### 10. List all installed packages and their versions

```shell
dpkg -l
```

### The server may be overprovisioned with more memory than required for the workload it is running.

```shell


#!/bin/bash



# Get the total amount of memory installed on the system

total_memory=$(free -m | awk '/^Mem:/{print $2}')



# Get the amount of memory currently in use

used_memory=$(free -m | awk '/^Mem:/{print $3}')



# Get the percentage of memory that is currently in use

memory_percent=$(echo "scale=2; $used_memory / $total_memory * 100" | bc)



# Get the amount of memory that is free and available for use

available_memory=$(free -m | awk '/^Mem:/{print $4}')



# Compare the amount of available memory to the configured amount of memory

configured_memory=${CONFIGURED_AMOUNT_OF_MEMORY}

if [ $available_memory -gt $configured_memory ]; then

    echo "The server may be overprovisioned with more memory than required for the workload it is running."

else

    echo "The available memory is within the configured amount, so this is likely not the issue."

fi



# Output the current memory usage information for further analysis

echo "Total memory: $total_memory MB"

echo "Used memory: $used_memory MB ($memory_percent%)"

echo "Available memory: $available_memory MB"


```


### Decrease memory allocation: Allocate less memory to the host vm to ensure that it does not have too much unused memory.

```shell
bash

#!/bin/bash



# Define variables

HOST_VM_NAME=${HOST_VM_NAME}

DECREASED_MEMORY_SIZE=${DECREASED_MEMORY_SIZE}



# Check if the host vm exists

if ! virsh list --name --all | grep -q "\${HOST_VM_NAME_}"; then

  echo "Error: Host VM $HOST_VM_NAME does not exist"

  exit 1

fi



# Check if the specified memory size is valid

if ! [[ $DECREASED_MEMORY_SIZE =~ ^[0-9]+$ ]]; then

  echo "Error: Invalid memory size $DECREASED_MEMORY_SIZE"

  exit 1

fi



# Stop the host vm

virsh shutdown $HOST_VM_NAME



# Decrease memory allocation for host vm

virsh setmem $HOST_VM_NAME $DECREASED_MEMORY_SIZE --config



# Start the host vm

virsh start $HOST_VM_NAME



# Verify that the memory allocation has been decreased

virsh dommemstat $HOST_VM_NAME | grep actual


```


"Host Memory is underutilized incident"

Overview

Parameters

Debug

1. Check the memory usage of the affected host

2. Check the current CPU load

6. Check the system load average

3. List all running processes and their memory usage

4. Check the amount of available disk space

5. Check the system logs for any relevant error messages

7. List all running Docker containers and their memory usage

8. Check the system swap usage

9. Check the system kernel version

10. List all installed packages and their versions

The server may be overprovisioned with more memory than required for the workload it is running.

Repair

Decrease memory allocation: Allocate less memory to the host vm to ensure that it does not have too much unused memory.

Learn more

Related Runbooks

Host Out of Memory (OOM) Incident

Increase of the errors/second rate for Tomcat server

Excessive memory usage by Tomcat.

Spark tasks failing due to out of memory errors.

Support