---
id: 957aaf1f-2bad-4eab-bea3-e97d3f5148c3
---
# AWS EBS Burst Credits Exhausted
---
​
AWS EBS (Elastic Block Store) Burst Credits Exhausted is an incident that occurs when an Amazon EBS volume has exhausted its burst credit balance. This happens when the volume has been used continuously at its maximum IOPS (input/output operations per second) rate for an extended period of time, causing the volume to consume its burst credits faster than it can earn them. As a result, the volume's performance is significantly degraded until the burst credits are replenished.
​
### Parameters
```shell
export VOLUME_ID="PLACEHOLDER"
​
export METRIC_ID="PLACEHOLDER"
​
export REGION="PLACEHOLDER"
​
export NEW_SIZE="PLACEHOLDER"
​
export INSTANCE_ID="PLACEHOLDER"
​
export EBS_VOLUME_ID="PLACEHOLDER"
​
export AWS_REGION="PLACEHOLDER"
```
​
## Debug
​
### List all EBS volumes in the current region
```shell
aws ec2 describe-volumes
```
​
### Get details about a specific EBS volume using its <volume-id>
```shell
aws ec2 describe-volumes --volume-ids ${VOLUME_ID}
```
​
### Check the burst credit balance for an EBS volume
```shell
aws cloudwatch get-metric-data --metric-data-queries '[{"Id":"${METRIC_ID}","MetricStat":{"Metric":{"Namespace":"AWS/EBS","MetricName":"BurstBalance","Dimensions":[{"Name":"VolumeId","Value":"${VOLUME_ID}"}]},"Period":300,"Stat":"Average"}}]' --start-time $(date -u +%Y-%m-%dT%TZ --date '-10 minutes') --end-time $(date -u +%Y-%m-%dT%TZ)
```
​
### Check the IOPS usage for an EBS volume
```shell
aws cloudwatch get-metric-data --metric-data-queries '[{"Id":"${METRIC_ID}","MetricStat":{"Metric":{"Namespace":"AWS/EBS","MetricName":"VolumeReadOps","Dimensions":[{"Name":"VolumeId","Value":"${VOLUME_ID}"}]},"Period":300,"Stat":"Sum"}},{"Id":"${METRIC_ID}","MetricStat":{"Metric":{"Namespace":"AWS/EBS","MetricName":"VolumeWriteOps","Dimensions":[{"Name":"VolumeId","Value":"${VOLUME_ID}"}]},"Period":300,"Stat":"Sum"}}]' --start-time $(date -u +%Y-%m-%dT%TZ --date '-10 minutes') --end-time $(date -u +%Y-%m-%dT%TZ)
```
​
### Check the burst credit balance usage for an EC2 instance
```shell
aws cloudwatch get-metric-data --metric-data-queries '[{"Id":"${METRIC_ID}","MetricStat":{"Metric":{"Namespace":"AWS/EBS","MetricName":"BurstBalance","Dimensions":[{"Name":"InstanceId","Value":"${INSTANCE_ID}"}]},"Period":300,"Stat":"Average"}}]' --start-time $(date -u +%Y-%m-%dT%TZ --date '-10 minutes') --end-time $(date -u +%Y-%m-%dT%TZ)
```
​
### The application or system that is using the EBS volume is experiencing a sudden spike in traffic or usage, causing it to perform more I/O operations than usual.
```shell
​
​
#!/bin/bash
​
​
​
# Set AWS region and EBS volume ID
​
REGION=${AWS_REGION}
​
VOL_ID=${EBS_VOLUME_ID}
​
​
​
# Get the current burst credit balance and maximum burst credit balance for the EBS volume
​
CURRENT_CREDIT=$(aws ec2 describe-volumes --region $REGION --volume-ids $VOL_ID --query 'Volumes[0].BurstBalance')
​
MAX_CREDIT=$(aws ec2 describe-volumes --region $REGION --volume-ids $VOL_ID --query 'Volumes[0].VolumeType' | grep -q "gp2" && echo "6144" || echo "0")
​
​
​
# Calculate the percentage of burst credits remaining
​
CREDIT_PERCENT=$(echo "scale=2; $CURRENT_CREDIT/$MAX_CREDIT*100" | bc)
​
​
​
# Check if the burst credit balance is below a certain threshold (e.g. 20%)
​
if [ $(echo "$CREDIT_PERCENT < 20" | bc) -eq 1 ]; then
​
    # Send a notification via email, Slack, or other means to alert the team
​
    echo "EBS volume $VOL_ID has only $CREDIT_PERCENT% burst credits remaining. Please investigate the cause of the sudden spike in I/O operations."
​
else
​
    # The burst credit balance is normal, so exit without errors
​
    exit 0
​
fi
​
​
```
​
## Repair
​
### Increase the volume size or switch to a provisioned IOPS volume to prevent burst credit exhaustion.
```shell
​
​
#!/bin/bash
​
​
​
# Define variables
​
REGION="${REGION}"
​
VOLUME_ID="${VOLUME_ID}"
​
NEW_SIZE=${NEW_SIZE}
​
​
​
# Stop the instance using the volume
​
INSTANCE_ID=$(aws ec2 describe-volumes --region $REGION --volume-ids $VOLUME_ID --query "Volumes[0].Attachments[0].InstanceId" --output text)
​
aws ec2 stop-instances --region $REGION --instance-ids $INSTANCE_ID
​
aws ec2 wait instance-stopped --region $REGION --instance-ids $INSTANCE_ID
​
​
​
# Modify the volume size
​
aws ec2 modify-volume --region $REGION --volume-id $VOLUME_ID --size $NEW_SIZE
​
​
​
# Start the instance
​
aws ec2 start-instances --region $REGION --instance-ids $INSTANCE_ID
​
aws ec2 wait instance-running --region $REGION --instance-ids $INSTANCE_ID
​
​
```


​
AWS EBS (Elastic Block Store) Burst Credits Exhausted is an incident that occurs when an Amazon EBS volume has exhausted its burst credit balance. This happens when the volume has been used continuously at its maximum IOPS (input/output operations per second) rate for an extended period of time, causing the volume to consume its burst credits faster than it can earn them. As a result, the volume's performance is significantly degraded until the burst credits are replenished.
​


Tomcat Connection Pool Exhaustion Incident occurs when the maximum number of connections that a Tomcat server can handle has been reached. This can happen due to an increase in traffic or a sudden surge in user activity. When the connection pool is exhausted, new requests cannot be handled, and the application may become unavailable. This type of incident requires immediate attention to ensure that the application is restored to its normal functioning state.


Tomcat Connection Pool Exhaustion Incident

Apache Spark driver failure refers to an incident where the driver program in an Apache Spark cluster fails to execute or crashes during runtime. This can happen due to a variety of reasons such as hardware failure, software bugs, resource constraints, or programming errors. As the driver program is responsible for coordinating the execution of tasks across the cluster, any failure in the driver can result in the entire Spark job failing. This can lead to data loss, processing delays, and impact the overall performance of the Spark cluster.


Spark driver failure incident.

This incident type refers to a situation where a Spark cluster experiences performance bottlenecks when it is subjected to peak loads. In other words, the Spark cluster struggles to handle the high volume of requests it receives during times of heavy traffic or increased demand. This can lead to slower processing times, delays, or even system crashes. Identifying and resolving the root cause of the bottlenecks is crucial to ensure the smooth functioning of the Spark cluster during peak loads.


Spark cluster bottlenecks during peak loads.

This incident type is related to an issue with a Redis instance being down. Redis is an in-memory data structure store used as a database, cache, and message broker. This incident could potentially cause disruption to applications and services that rely on Redis for data storage or caching.


Redis Instance Down

The "High error rate on NGINX" incident type refers to a situation where the error rate on the NGINX server is above 1% for the last 5 minutes. This can result in degraded performance or downtime of the affected service, impacting user experience and potentially leading to lost revenue. The incident requires immediate attention and resolution to minimize the impact on users and prevent further damage.


High error rate on NGINX incident

```shell
export VOLUME_ID="PLACEHOLDER"
​
export METRIC_ID="PLACEHOLDER"
​
export REGION="PLACEHOLDER"
​
export NEW_SIZE="PLACEHOLDER"
​
export INSTANCE_ID="PLACEHOLDER"
​
export EBS_VOLUME_ID="PLACEHOLDER"
​
export AWS_REGION="PLACEHOLDER"
```

​


​

### List all EBS volumes in the current region

```shell
aws ec2 describe-volumes
```

​

### Get details about a specific EBS volume using its <volume-id>

```shell
aws ec2 describe-volumes --volume-ids ${VOLUME_ID}
```

​

### Check the burst credit balance for an EBS volume

```shell
aws cloudwatch get-metric-data --metric-data-queries '[{"Id":"${METRIC_ID}","MetricStat":{"Metric":{"Namespace":"AWS/EBS","MetricName":"BurstBalance","Dimensions":[{"Name":"VolumeId","Value":"${VOLUME_ID}"}]},"Period":300,"Stat":"Average"}}]' --start-time $(date -u +%Y-%m-%dT%TZ --date '-10 minutes') --end-time $(date -u +%Y-%m-%dT%TZ)
```

​

### Check the IOPS usage for an EBS volume

```shell
aws cloudwatch get-metric-data --metric-data-queries '[{"Id":"${METRIC_ID}","MetricStat":{"Metric":{"Namespace":"AWS/EBS","MetricName":"VolumeReadOps","Dimensions":[{"Name":"VolumeId","Value":"${VOLUME_ID}"}]},"Period":300,"Stat":"Sum"}},{"Id":"${METRIC_ID}","MetricStat":{"Metric":{"Namespace":"AWS/EBS","MetricName":"VolumeWriteOps","Dimensions":[{"Name":"VolumeId","Value":"${VOLUME_ID}"}]},"Period":300,"Stat":"Sum"}}]' --start-time $(date -u +%Y-%m-%dT%TZ --date '-10 minutes') --end-time $(date -u +%Y-%m-%dT%TZ)
```

​

### Check the burst credit balance usage for an EC2 instance

```shell
aws cloudwatch get-metric-data --metric-data-queries '[{"Id":"${METRIC_ID}","MetricStat":{"Metric":{"Namespace":"AWS/EBS","MetricName":"BurstBalance","Dimensions":[{"Name":"InstanceId","Value":"${INSTANCE_ID}"}]},"Period":300,"Stat":"Average"}}]' --start-time $(date -u +%Y-%m-%dT%TZ --date '-10 minutes') --end-time $(date -u +%Y-%m-%dT%TZ)
```

​

### The application or system that is using the EBS volume is experiencing a sudden spike in traffic or usage, causing it to perform more I/O operations than usual.

```shell
​
​
#!/bin/bash
​
​
​
# Set AWS region and EBS volume ID
​
REGION=${AWS_REGION}
​
VOL_ID=${EBS_VOLUME_ID}
​
​
​
# Get the current burst credit balance and maximum burst credit balance for the EBS volume
​
CURRENT_CREDIT=$(aws ec2 describe-volumes --region $REGION --volume-ids $VOL_ID --query 'Volumes[0].BurstBalance')
​
MAX_CREDIT=$(aws ec2 describe-volumes --region $REGION --volume-ids $VOL_ID --query 'Volumes[0].VolumeType' | grep -q "gp2" && echo "6144" || echo "0")
​
​
​
# Calculate the percentage of burst credits remaining
​
CREDIT_PERCENT=$(echo "scale=2; $CURRENT_CREDIT/$MAX_CREDIT*100" | bc)
​
​
​
# Check if the burst credit balance is below a certain threshold (e.g. 20%)
​
if [ $(echo "$CREDIT_PERCENT < 20" | bc) -eq 1 ]; then
​
    # Send a notification via email, Slack, or other means to alert the team
​
    echo "EBS volume $VOL_ID has only $CREDIT_PERCENT% burst credits remaining. Please investigate the cause of the sudden spike in I/O operations."
​
else
​
    # The burst credit balance is normal, so exit without errors
​
    exit 0
​
fi
​
​
```

​


​

### Increase the volume size or switch to a provisioned IOPS volume to prevent burst credit exhaustion.

```shell
​
​
#!/bin/bash
​
​
​
# Define variables
​
REGION="${REGION}"
​
VOLUME_ID="${VOLUME_ID}"
​
NEW_SIZE=${NEW_SIZE}
​
​
​
# Stop the instance using the volume
​
INSTANCE_ID=$(aws ec2 describe-volumes --region $REGION --volume-ids $VOLUME_ID --query "Volumes[0].Attachments[0].InstanceId" --output text)
​
aws ec2 stop-instances --region $REGION --instance-ids $INSTANCE_ID
​
aws ec2 wait instance-stopped --region $REGION --instance-ids $INSTANCE_ID
​
​
​
# Modify the volume size
​
aws ec2 modify-volume --region $REGION --volume-id $VOLUME_ID --size $NEW_SIZE
​
​
​
# Start the instance
​
aws ec2 start-instances --region $REGION --instance-ids $INSTANCE_ID
​
aws ec2 wait instance-running --region $REGION --instance-ids $INSTANCE_ID
​
​
```


AWS EBS Burst Credits Exhausted

Overview

Parameters

Debug

List all EBS volumes in the current region

Get details about a specific EBS volume using its <volume-id>

Check the burst credit balance for an EBS volume

Check the IOPS usage for an EBS volume

Check the burst credit balance usage for an EC2 instance

The application or system that is using the EBS volume is experiencing a sudden spike in traffic or usage, causing it to perform more I/O operations than usual.

Repair

Increase the volume size or switch to a provisioned IOPS volume to prevent burst credit exhaustion.

Learn more

Related Runbooks

Tomcat Connection Pool Exhaustion Incident

Spark driver failure incident.

Spark cluster bottlenecks during peak loads.

Redis Instance Down

Support