---
id: d40b07b5-35df-4190-be5f-60bc63df38c1
---

# Redis instance down incident
---

A Redis instance down incident refers to a situation where the Redis server, which is a popular open-source in-memory data structure store, is not functioning or accessible. This can cause interruption in the service and impact the performance of applications that rely on Redis for caching or data storage. Typically, this type of incident is considered high urgency and requires immediate attention from the responsible team to investigate, diagnose, and resolve the issue.

### Parameters
```shell
export PATH_TO_REDIS_LOG_FILE="PLACEHOLDER"
```

## Debug

### Check if Redis service is running
```shell
systemctl status redis
```

### Check Redis logs for any errors
```shell
journalctl -u redis | tail
```

### Check Redis configuration file for any issues
```shell
redis-cli config get *
```

### Check Redis server connectivity
```shell
redis-cli ping
```

### Check Redis memory usage
```shell
redis-cli info memory
```

### Check Redis CPU usage
```shell
redis-cli info cpu
```

### Check Redis replication status
```shell
redis-cli info replication
```

### Check Redis keyspace status
```shell
redis-cli info keyspace
```

### Redis server might have been facing heavy load or high traffic, causing it to crash or become unresponsive.
```shell


if systemctl is-active --quiet redis.service; then

  echo "Redis server is running."

else

  echo "Redis server is not running."

fi





if grep -q "ERROR" /var/log/redis/redis.log; then

  echo "Redis server has errors in logs."

fi



if grep -q "CRASH" /var/log/redis/redis.log; then

  echo "Redis server has crashed."

fi





if grep -q "used_memory_peak" /var/log/redis/redis.log; then

  echo "Redis server has high memory usage."

fi



if grep -q "total_commands_processed" /var/log/redis/redis.log; then

  echo "Redis server has high command processing rate."

fi





if grep -q "cpu" /proc/stat; then

  echo "System CPU usage is high."

fi



if free | awk 'NR==2{printf "%.2f\n", $3/$2*100}' | grep -q "^[7-9][0-9]\|[1-9][0-9][0-9]"; then

  echo "System memory usage is high."

fi



if df | awk '$NF=="/"{printf "%.2f\n", $5}' | grep -q "^[7-9][0-9]\|[1-9][0-9][0-9]"; then

  echo "System disk usage is high."

fi





if netstat -s | grep -q "segments retransmitted"; then

  echo "Network traffic is high and causing retransmissions."

fi


```

## Repair

### Check the Redis server logs to understand the root cause of the issue.
```shell


#!/bin/bash



# Set the log file path

LOG_FILE=${PATH_TO_REDIS_LOG_FILE}



# Check if the log file exists

if [ -f "$LOG_FILE" ]; then

    # Print the last 10 lines of the log file

    tail -n 10 "$LOG_FILE"

else

    echo "Error: Redis log file not found!"

fi


```


A Redis instance down incident refers to a situation where the Redis server, which is a popular open-source in-memory data structure store, is not functioning or accessible. This can cause interruption in the service and impact the performance of applications that rely on Redis for caching or data storage. Typically, this type of incident is considered high urgency and requires immediate attention from the responsible team to investigate, diagnose, and resolve the issue.


This incident type refers to a situation where a SQLServer database goes offline, which can potentially cause disruptions in software or applications that rely on it. The incident is triggered by a monitoring system that detects the database's state is not available. A prompt response is necessary to resolve the issue and prevent any adverse impact on the system.


Database Offline (SQLServer)

The Redis too many masters incident occurs when there are too many master nodes in a Redis cluster, leading to connection issues and potential data loss. This can happen due to misconfiguration, network issues, or other factors, and requires immediate attention to prevent further damage.


Redis too many masters incident

This incident type refers to an issue with Redis replication, which means that there is a problem with the synchronization of data between Redis instances. This issue could impact the availability and performance of the system and may require immediate attention to restore the replication and ensure data consistency. The incident could be caused by various factors, such as network problems, hardware failures, or configuration issues. The incident must be investigated and resolved as soon as possible to avoid any data loss or downtime.


Redis replication broken incident.

This incident type refers to a situation where Redis, a popular in-memory data structure store, has rejected connections. This could happen due to various reasons such as server overload, network issues, or misconfiguration. When this occurs, it can cause disruptions in the performance of applications that rely on Redis for data storage or caching. Thus, it requires immediate attention and resolution to minimize the impact on the applications and users.


Redis Rejected Connections Incident

The Redis missing master incident occurs when the Redis cluster has no node marked as master. This can cause problems with the Redis service and may result in service disruptions or failures. It is important to address this issue promptly to ensure the smooth operation of the Redis service.


Redis missing master incident

```shell
export PATH_TO_REDIS_LOG_FILE="PLACEHOLDER"
```


### Check if Redis service is running

```shell
systemctl status redis
```

### Check Redis logs for any errors

```shell
journalctl -u redis | tail
```

### Check Redis configuration file for any issues

```shell
redis-cli config get *
```

### Check Redis server connectivity

```shell
redis-cli ping
```

### Check Redis memory usage

```shell
redis-cli info memory
```

### Check Redis CPU usage

```shell
redis-cli info cpu
```

### Check Redis replication status

```shell
redis-cli info replication
```

### Check Redis keyspace status

```shell
redis-cli info keyspace
```

### Redis server might have been facing heavy load or high traffic, causing it to crash or become unresponsive.

```shell


if systemctl is-active --quiet redis.service; then

  echo "Redis server is running."

else

  echo "Redis server is not running."

fi





if grep -q "ERROR" /var/log/redis/redis.log; then

  echo "Redis server has errors in logs."

fi



if grep -q "CRASH" /var/log/redis/redis.log; then

  echo "Redis server has crashed."

fi





if grep -q "used_memory_peak" /var/log/redis/redis.log; then

  echo "Redis server has high memory usage."

fi



if grep -q "total_commands_processed" /var/log/redis/redis.log; then

  echo "Redis server has high command processing rate."

fi





if grep -q "cpu" /proc/stat; then

  echo "System CPU usage is high."

fi



if free | awk 'NR==2{printf "%.2f\n", $3/$2*100}' | grep -q "^[7-9][0-9]\|[1-9][0-9][0-9]"; then

  echo "System memory usage is high."

fi



if df | awk '$NF=="/"{printf "%.2f\n", $5}' | grep -q "^[7-9][0-9]\|[1-9][0-9][0-9]"; then

  echo "System disk usage is high."

fi





if netstat -s | grep -q "segments retransmitted"; then

  echo "Network traffic is high and causing retransmissions."

fi


```


### Check the Redis server logs to understand the root cause of the issue.

```shell


#!/bin/bash



# Set the log file path

LOG_FILE=${PATH_TO_REDIS_LOG_FILE}



# Check if the log file exists

if [ -f "$LOG_FILE" ]; then

    # Print the last 10 lines of the log file

    tail -n 10 "$LOG_FILE"

else

    echo "Error: Redis log file not found!"

fi


```


Redis instance down incident

Overview

Parameters

Debug

Check if Redis service is running

Check Redis logs for any errors

Check Redis configuration file for any issues

Check Redis server connectivity

Check Redis memory usage

Check Redis CPU usage

Check Redis replication status

Check Redis keyspace status

Redis server might have been facing heavy load or high traffic, causing it to crash or become unresponsive.

Repair

Check the Redis server logs to understand the root cause of the issue.

Learn more

Related Runbooks

Database Offline (SQLServer)

Redis too many masters incident

Redis replication broken incident.

Redis Rejected Connections Incident

Support