---
id: 66ee9ee5-8aac-4453-b6c0-bda26532ea25
---

# Kubernetes - Persistent Volumes Running Out of Storage Space.
---

This incident type refers to a situation where the persistent volumes used in a software application are running out of storage space. Persistent volumes are used to store data in a containerized environment, and if the storage space is limited, it can lead to errors and service disruptions. This issue can occur due to various reasons such as a sudden increase in data volume, incorrect configuration, or inefficient utilization of storage resources. It is crucial to monitor and manage the available storage space to ensure the smooth functioning of the software application.

### Parameters
```shell
export PV_NAME="PLACEHOLDER"

export NAMESPACE="PLACEHOLDER"

export PVC_NAME="PLACEHOLDER"

export POD_NAME="PLACEHOLDER"

export NEW_SIZE="PLACEHOLDER"
```

## Debug

### List all persistent volumes in the cluster
```shell
kubectl get pv
```

### Check the usage and capacity of each persistent volume
```shell
kubectl describe pv ${PV_NAME}
```

### Check the usage and capacity of the persistent volume claim(s) that use the persistent volume
```shell
kubectl describe pvc -n ${NAMESPACE} ${PVC_NAME}
```

### Check the usage and capacity of the pod(s) that use the persistent volume claim(s)
```shell
kubectl describe pod -n ${NAMESPACE} ${POD_NAME}
```

### Check the logs of the pod(s) to see if there are any errors related to storage
```shell
kubectl logs -n ${NAMESPACE} ${POD_NAME}
```

### Check if there are any pending events related to storage in the cluster
```shell
kubectl get events
```

### Check if there are any nodes with low disk space available
```shell
kubectl describe node
```

### There might be a sudden increase in data volume that the persistent volume cannot handle.
```shell


#!/bin/bash



# Get the name of the persistent volume that is running out of space

PV_NAME=${PV_NAME}



# Check the available space in the persistent volume

PV_CAPACITY=$(kubectl get pv $PV_NAME -o jsonpath='{.spec.capacity.storage}')



# Check the current usage of the persistent volume

PV_USAGE=$(kubectl get pv $PV_NAME -o jsonpath='{.status.capacity.storage}')



# Calculate the percentage of space used

PERCENTAGE_USED=$(echo "scale=2; ($PV_USAGE/$PV_CAPACITY)*100" | bc)



# Check if the percentage of space used has exceeded a threshold

THRESHOLD=80

if (( $(echo "$PERCENTAGE_USED > $THRESHOLD" | bc -l) )); then

  echo "The persistent volume $PV_NAME is running out of storage space."

  echo "Current capacity: $PV_CAPACITY"

  echo "Current usage: $PV_USAGE"

  echo "Percentage used: $PERCENTAGE_USED%"

else

  echo "The persistent volume $PV_NAME has sufficient storage space."

  echo "Current capacity: $PV_CAPACITY"

  echo "Current usage: $PV_USAGE"

  echo "Percentage used: $PERCENTAGE_USED%"

fi


```

## Repair

### Increase the storage capacity of the persistent volumes.
```shell


#!/bin/bash



# Set variables

PVC_NAME=${PVC_NAME} # Name of the persistent volume claim to modify

NEW_SIZE=${NEW_SIZE} # New size to set for the persistent volume claim



# Update the persistent volume claim with the new size

kubectl patch pvc $PVC_NAME -p '{"spec":{"resources":{"requests":{"storage":"'$NEW_SIZE'"}}}}'





./increase-pvc-size.sh --pvc-name my-pvc --new-size 10Gi


```

This incident type refers to a situation where the persistent volumes used in a software application are running out of storage space. Persistent volumes are used to store data in a containerized environment, and if the storage space is limited, it can lead to errors and service disruptions. This issue can occur due to various reasons such as a sudden increase in data volume, incorrect configuration, or inefficient utilization of storage resources. It is crucial to monitor and manage the available storage space to ensure the smooth functioning of the software application.


The Vault cluster health incident is related to the health of a Vault cluster instance. This incident type is triggered when the cluster instance is not healthy and requires attention to ensure it is functioning properly. The incident typically involves evaluating the current state of the cluster instance, diagnosing the issue, and taking corrective action to restore the health of the instance.


Vault cluster health incident on kubernetes

This incident type refers to cases where the MySQL service, which is responsible for managing and storing data in a database, runs out of available space. This can happen due to a variety of reasons such as an unexpected increase in data volume or insufficient allocation of storage resources. When the MySQL service runs out of space, it can cause data loss, application errors, and even system crashes if not addressed promptly.


MySQL Service Out of Space.

A Host Out of Memory(OOM) Incident occurs when a server or system runs out of memory, causing it to crash or become unresponsive. This can be caused by a variety of factors, such as an unexpected surge in traffic or insufficient resources allocated to the system. Resolving this type of incident requires identifying the root cause of the memory issue and taking appropriate measures such as optimizing system resources or increasing memory capacity.


Host Out of Memory (OOM) Incident

This incident type involves monitoring the replicas of a Kubernetes Statefulset, which is a type of workload in Kubernetes used for stateful applications. The incident is triggered when more than one replica's pods are down, creating an unsafe situation for manual operations. This incident is critical and requires immediate attention to resolve the issue and ensure the smooth functioning of the stateful applications.


Kubernetes Statefulset Replicas Monitoring Incident

The Kubernetes Nodes with Memorypressure incident type occurs when a Kubernetes cluster node is running low on memory, which can be caused by a memory leak in an application. This incident type requires immediate attention to prevent any downtime and ensure the proper functioning of the Kubernetes cluster. Typically, this incident type is monitored by DevOps teams using various monitoring tools, including PagerDuty, to identify and address memory pressure issues quickly.


Kubernetes Nodes with Memorypressure incident

```shell
export PV_NAME="PLACEHOLDER"

export NAMESPACE="PLACEHOLDER"

export PVC_NAME="PLACEHOLDER"

export POD_NAME="PLACEHOLDER"

export NEW_SIZE="PLACEHOLDER"
```


### List all persistent volumes in the cluster

```shell
kubectl get pv
```

### Check the usage and capacity of each persistent volume

```shell
kubectl describe pv ${PV_NAME}
```

### Check the usage and capacity of the persistent volume claim(s) that use the persistent volume

```shell
kubectl describe pvc -n ${NAMESPACE} ${PVC_NAME}
```

### Check the usage and capacity of the pod(s) that use the persistent volume claim(s)

```shell
kubectl describe pod -n ${NAMESPACE} ${POD_NAME}
```

### Check the logs of the pod(s) to see if there are any errors related to storage

```shell
kubectl logs -n ${NAMESPACE} ${POD_NAME}
```

### Check if there are any pending events related to storage in the cluster

```shell
kubectl get events
```

### Check if there are any nodes with low disk space available

```shell
kubectl describe node
```

### There might be a sudden increase in data volume that the persistent volume cannot handle.

```shell


#!/bin/bash



# Get the name of the persistent volume that is running out of space

PV_NAME=${PV_NAME}



# Check the available space in the persistent volume

PV_CAPACITY=$(kubectl get pv $PV_NAME -o jsonpath='{.spec.capacity.storage}')



# Check the current usage of the persistent volume

PV_USAGE=$(kubectl get pv $PV_NAME -o jsonpath='{.status.capacity.storage}')



# Calculate the percentage of space used

PERCENTAGE_USED=$(echo "scale=2; ($PV_USAGE/$PV_CAPACITY)*100" | bc)



# Check if the percentage of space used has exceeded a threshold

THRESHOLD=80

if (( $(echo "$PERCENTAGE_USED > $THRESHOLD" | bc -l) )); then

  echo "The persistent volume $PV_NAME is running out of storage space."

  echo "Current capacity: $PV_CAPACITY"

  echo "Current usage: $PV_USAGE"

  echo "Percentage used: $PERCENTAGE_USED%"

else

  echo "The persistent volume $PV_NAME has sufficient storage space."

  echo "Current capacity: $PV_CAPACITY"

  echo "Current usage: $PV_USAGE"

  echo "Percentage used: $PERCENTAGE_USED%"

fi


```


### Increase the storage capacity of the persistent volumes.

```shell


#!/bin/bash



# Set variables

PVC_NAME=${PVC_NAME} # Name of the persistent volume claim to modify

NEW_SIZE=${NEW_SIZE} # New size to set for the persistent volume claim



# Update the persistent volume claim with the new size

kubectl patch pvc $PVC_NAME -p '{"spec":{"resources":{"requests":{"storage":"'$NEW_SIZE'"}}}}'





./increase-pvc-size.sh --pvc-name my-pvc --new-size 10Gi


```


Kubernetes - Persistent Volumes Running Out of Storage Space.

Overview

Parameters

Debug

List all persistent volumes in the cluster

Check the usage and capacity of each persistent volume

Check the usage and capacity of the persistent volume claim(s) that use the persistent volume

Check the usage and capacity of the pod(s) that use the persistent volume claim(s)

Check if there are any nodes with low disk space available

There might be a sudden increase in data volume that the persistent volume cannot handle.

Repair

Increase the storage capacity of the persistent volumes.

Learn more

Related Runbooks

Vault cluster health incident on kubernetes

MySQL Service Out of Space.

Host Out of Memory (OOM) Incident

Kubernetes Statefulset Replicas Monitoring Incident

Support