---
id: fe7f3084-bea5-4ab4-a00e-ea25cfe03587
---

# Redis too many masters incident
---

The Redis too many masters incident occurs when there are too many master nodes in a Redis cluster, leading to connection issues and potential data loss. This can happen due to misconfiguration, network issues, or other factors, and requires immediate attention to prevent further damage.

### Parameters
```shell
# Environment Variables

export REDIS_CLUSTER_HOST="PLACEHOLDER"

export REDIS_CLUSTER_PORT="PLACEHOLDER"

export REDIS_PORT="PLACEHOLDER"

export REDIS_CLUSTER_NAME="PLACEHOLDER"

export MAX_NUMBER_OF_MASTERS="PLACEHOLDER"

export REDIS_CLUSTER_IP="PLACEHOLDER"
```

## Debug

### Check if Redis is running on the affected instance
```shell
systemctl status redis-server.service
```

### Check the Redis configuration file
```shell
cat /etc/redis/redis.conf
```

### Check the number of master nodes in the Redis cluster
```shell
redis-cli -h ${REDIS_CLUSTER_HOST} -p ${REDIS_CLUSTER_PORT} cluster nodes | grep "master" | wc -l
```

### Check the status of the Redis cluster
```shell
redis-cli -h ${REDIS_CLUSTER_HOST} -p ${REDIS_CLUSTER_PORT} cluster info
```

### Check the number of connections to the Redis cluster
```shell
redis-cli -h ${REDIS_CLUSTER_HOST} -p ${REDIS_CLUSTER_PORT} info clients | grep "connected_clients"
```

### Check the Redis logs for any errors or warnings
```shell
tail -f /var/log/redis/redis-server.log
```

### Restart the Redis service
```shell
systemctl restart redis-server.service
```

### Network connectivity issues between Redis nodes, causing nodes to become isolated and designate themselves as master nodes.
```shell
bash

#!/bin/bash



# Define variables

REDIS_CLUSTER=${REDIS_CLUSTER_IP}

REDIS_PORT=${REDIS_PORT}



# Check network connectivity between Redis nodes

for i in {1..3}; do

  redis-cli -h $REDIS_CLUSTER -p $REDIS_PORT ping

  if [ "$?" -ne "0" ]; then

    echo "Error: Redis node $REDIS_CLUSTER is not responding"

    exit 1

  fi

done



# Check Redis cluster status

redis-cli -h $REDIS_CLUSTER -p $REDIS_PORT cluster info

if [ "$?" -ne "0" ]; then

  echo "Error: Redis cluster is not responding"

  exit 1

fi



# Check Redis cluster nodes

redis-cli -h $REDIS_CLUSTER -p $REDIS_PORT cluster nodes

if [ "$?" -ne "0" ]; then

  echo "Error: Redis cluster nodes are not responding"

  exit 1

fi



# Check Redis cluster configuration

redis-cli -h $REDIS_CLUSTER -p $REDIS_PORT config get cluster-enabled

if [ "$?" -ne "0" ]; then

  echo "Error: Redis cluster configuration is invalid"

  exit 1

fi



# Check Redis cluster node roles

redis-cli -h $REDIS_CLUSTER -p $REDIS_PORT cluster nodes | awk '{print $3}' | sort | uniq -c

if [ "$?" -ne "0" ]; then

  echo "Error: Redis cluster node roles are not consistent"

  exit 1

fi



echo "Success: Redis cluster is healthy"

exit 0


```

## Repair

### Reduce the number of master nodes in the Redis cluster to the recommended maximum, which varies based on the size and complexity of the system.
```shell


#!/bin/bash



# Set variables

REDIS_CLUSTER=${REDIS_CLUSTER_NAME}

MAX_MASTERS=${MAX_NUMBER_OF_MASTERS}



# Get current number of master nodes

current_masters=$(redis-cli -c -h $REDIS_CLUSTER cluster info | grep "masters" | awk -F: '{print $2}')



# Check if current number of masters is greater than the maximum recommended

if [ $current_masters -gt $MAX_MASTERS ]; then

    # Get a list of all master nodes

    master_nodes=$(redis-cli -c -h $REDIS_CLUSTER cluster nodes | grep "master" | awk '{print $2}')



    # Select the first $MAX_MASTERS nodes as new master nodes

    new_masters=$(echo "$master_nodes" | head -n $MAX_MASTERS)



    # Set new master nodes for the Redis cluster

    for node in $new_masters; do

        redis-cli -c -h $REDIS_CLUSTER cluster replicate $node

    done



    # Failover to the new master nodes

    redis-cli -c -h $REDIS_CLUSTER cluster failover force



    echo "Successfully reduced the number of master nodes to $MAX_MASTERS."

else

    echo "The current number of master nodes is already within the recommended limit."

fi




```

The Redis too many masters incident occurs when there are too many master nodes in a Redis cluster, leading to connection issues and potential data loss. This can happen due to misconfiguration, network issues, or other factors, and requires immediate attention to prevent further damage.


This incident type refers to an issue with Redis replication, which means that there is a problem with the synchronization of data between Redis instances. This issue could impact the availability and performance of the system and may require immediate attention to restore the replication and ensure data consistency. The incident could be caused by various factors, such as network problems, hardware failures, or configuration issues. The incident must be investigated and resolved as soon as possible to avoid any data loss or downtime.


Redis replication broken incident.

This incident type refers to a situation where Redis, a popular in-memory data structure store, has rejected connections. This could happen due to various reasons such as server overload, network issues, or misconfiguration. When this occurs, it can cause disruptions in the performance of applications that rely on Redis for data storage or caching. Thus, it requires immediate attention and resolution to minimize the impact on the applications and users.


Redis Rejected Connections Incident

The Redis missing master incident occurs when the Redis cluster has no node marked as master. This can cause problems with the Redis service and may result in service disruptions or failures. It is important to address this issue promptly to ensure the smooth operation of the Redis service.


Redis missing master incident

The Redis missing backup incident refers to a situation where Redis, a popular in-memory data structure store, has not been backed up for a certain period of time. This could lead to data loss or corruption in the event of a system failure or other issues. The incident typically requires immediate attention to ensure that the data is properly backed up and that any potential issues are resolved quickly to avoid further complications.


Redis missing backup incident.

A Redis instance down incident refers to a situation where the Redis server, which is a popular open-source in-memory data structure store, is not functioning or accessible. This can cause interruption in the service and impact the performance of applications that rely on Redis for caching or data storage. Typically, this type of incident is considered high urgency and requires immediate attention from the responsible team to investigate, diagnose, and resolve the issue.


Redis instance down incident

```shell
# Environment Variables

export REDIS_CLUSTER_HOST="PLACEHOLDER"

export REDIS_CLUSTER_PORT="PLACEHOLDER"

export REDIS_PORT="PLACEHOLDER"

export REDIS_CLUSTER_NAME="PLACEHOLDER"

export MAX_NUMBER_OF_MASTERS="PLACEHOLDER"

export REDIS_CLUSTER_IP="PLACEHOLDER"
```


### Check if Redis is running on the affected instance

```shell
systemctl status redis-server.service
```

### Check the Redis configuration file

```shell
cat /etc/redis/redis.conf
```

### Check the number of master nodes in the Redis cluster

```shell
redis-cli -h ${REDIS_CLUSTER_HOST} -p ${REDIS_CLUSTER_PORT} cluster nodes | grep "master" | wc -l
```

### Check the status of the Redis cluster

```shell
redis-cli -h ${REDIS_CLUSTER_HOST} -p ${REDIS_CLUSTER_PORT} cluster info
```

### Check the number of connections to the Redis cluster

```shell
redis-cli -h ${REDIS_CLUSTER_HOST} -p ${REDIS_CLUSTER_PORT} info clients | grep "connected_clients"
```

### Check the Redis logs for any errors or warnings

```shell
tail -f /var/log/redis/redis-server.log
```

### Restart the Redis service

```shell
systemctl restart redis-server.service
```

### Network connectivity issues between Redis nodes, causing nodes to become isolated and designate themselves as master nodes.

```shell
bash

#!/bin/bash



# Define variables

REDIS_CLUSTER=${REDIS_CLUSTER_IP}

REDIS_PORT=${REDIS_PORT}



# Check network connectivity between Redis nodes

for i in {1..3}; do

  redis-cli -h $REDIS_CLUSTER -p $REDIS_PORT ping

  if [ "$?" -ne "0" ]; then

    echo "Error: Redis node $REDIS_CLUSTER is not responding"

    exit 1

  fi

done



# Check Redis cluster status

redis-cli -h $REDIS_CLUSTER -p $REDIS_PORT cluster info

if [ "$?" -ne "0" ]; then

  echo "Error: Redis cluster is not responding"

  exit 1

fi



# Check Redis cluster nodes

redis-cli -h $REDIS_CLUSTER -p $REDIS_PORT cluster nodes

if [ "$?" -ne "0" ]; then

  echo "Error: Redis cluster nodes are not responding"

  exit 1

fi



# Check Redis cluster configuration

redis-cli -h $REDIS_CLUSTER -p $REDIS_PORT config get cluster-enabled

if [ "$?" -ne "0" ]; then

  echo "Error: Redis cluster configuration is invalid"

  exit 1

fi



# Check Redis cluster node roles

redis-cli -h $REDIS_CLUSTER -p $REDIS_PORT cluster nodes | awk '{print $3}' | sort | uniq -c

if [ "$?" -ne "0" ]; then

  echo "Error: Redis cluster node roles are not consistent"

  exit 1

fi



echo "Success: Redis cluster is healthy"

exit 0


```


### Reduce the number of master nodes in the Redis cluster to the recommended maximum, which varies based on the size and complexity of the system.

```shell


#!/bin/bash



# Set variables

REDIS_CLUSTER=${REDIS_CLUSTER_NAME}

MAX_MASTERS=${MAX_NUMBER_OF_MASTERS}



# Get current number of master nodes

current_masters=$(redis-cli -c -h $REDIS_CLUSTER cluster info | grep "masters" | awk -F: '{print $2}')



# Check if current number of masters is greater than the maximum recommended

if [ $current_masters -gt $MAX_MASTERS ]; then

    # Get a list of all master nodes

    master_nodes=$(redis-cli -c -h $REDIS_CLUSTER cluster nodes | grep "master" | awk '{print $2}')



    # Select the first $MAX_MASTERS nodes as new master nodes

    new_masters=$(echo "$master_nodes" | head -n $MAX_MASTERS)



    # Set new master nodes for the Redis cluster

    for node in $new_masters; do

        redis-cli -c -h $REDIS_CLUSTER cluster replicate $node

    done



    # Failover to the new master nodes

    redis-cli -c -h $REDIS_CLUSTER cluster failover force



    echo "Successfully reduced the number of master nodes to $MAX_MASTERS."

else

    echo "The current number of master nodes is already within the recommended limit."

fi




```


Redis too many masters incident

Overview

Parameters

Debug

Check if Redis is running on the affected instance

Check the Redis configuration file

Check the number of master nodes in the Redis cluster

Check the status of the Redis cluster

Check the number of connections to the Redis cluster

Check the Redis logs for any errors or warnings

Restart the Redis service

Network connectivity issues between Redis nodes, causing nodes to become isolated and designate themselves as master nodes.

Repair

Reduce the number of master nodes in the Redis cluster to the recommended maximum, which varies based on the size and complexity of the system.

Learn more

Related Runbooks

Redis replication broken incident.

Redis Rejected Connections Incident

Redis missing master incident

Redis missing backup incident.

Support