---
id: fd804b28-d4e3-49ac-a0db-6ef74e6ef3fb
---

# Redis replication broken incident.
---

This incident type refers to an issue with Redis replication, which means that there is a problem with the synchronization of data between Redis instances. This issue could impact the availability and performance of the system and may require immediate attention to restore the replication and ensure data consistency. The incident could be caused by various factors, such as network problems, hardware failures, or configuration issues. The incident must be investigated and resolved as soon as possible to avoid any data loss or downtime.

### Parameters
```shell
# Environment Variables

export SLAVE_HOSTNAME="PLACEHOLDER"

export MASTER_HOSTNAME="PLACEHOLDER"

export REDIS_PORT="PLACEHOLDER"

export MASTER_IP="PLACEHOLDER"

export REDIS_CONFIG_DIR="PLACEHOLDER"

export REDIS_VERSION="PLACEHOLDER"


```

## Debug

### Check if Redis is running
```shell
systemctl status redis
```

### Check Redis logs for errors
```shell
tail -n 100 /var/log/redis/redis-server.log | grep -i error
```

### Verify Redis replication status
```shell
redis-cli INFO replication
```

### Check Redis slave status
```shell
redis-cli -h ${SLAVE_HOSTNAME} -p ${REDIS_PORT} INFO replication
```

### Check Redis configuration file
```shell
cat /etc/redis/redis.conf
```

### Verify Redis master configuration
```shell
redis-cli -h ${MASTER_HOSTNAME} -p ${REDIS_PORT} INFO server
```

### Verify Redis slave configuration
```shell
redis-cli -h ${SLAVE_HOSTNAME} -p ${REDIS_PORT} INFO server
```

### Check network connectivity between master and slave
```shell
ping ${MASTER_HOSTNAME}

ping ${SLAVE_HOSTNAME}
```

### Check network ports for Redis instance
```shell
netstat -anp | grep ${REDIS_PORT}
```

### Check Redis replication delay
```shell
redis-cli -h ${SLAVE_HOSTNAME} -p ${REDIS_PORT} INFO replication | grep -i delay
```

### Network connectivity issues between Redis instances, causing replication failures.
```shell


#!/bin/bash



# Check network connectivity between Redis instances

redis_ip1=${REDIS_INSTANCE1_IP}

redis_ip2=${REDIS_INSTANCE2_IP}

ping -c 5 $redis_ip1

ping -c 5 $redis_ip2



# Check Redis replication status

redis-cli -h $redis_ip1 info replication

redis-cli -h $redis_ip2 info replication


```

## Repair

### Check the Redis replication configuration to ensure it's correctly set up and that there are no misconfigurations that could cause replication failures.
```shell


#!/bin/bash



# Set variables

REDIS_CONF_DIR=${REDIS_CONFIG_DIR}



# Check Redis replication configuration

if grep -q "^slaveof" "${REDIS_CONF_DIR}/redis.conf"; then

    echo "Redis replication is enabled. Configuration is correct."

else

    echo "Redis replication is not enabled. Updating configuration."

    echo "slaveof ${MASTER_IP} ${REDIS_PORT}" >> "${REDIS_CONF_DIR}/redis.conf"

    echo "Redis replication configuration updated."

fi


```

### Verify the Redis version and ensure it's up-to-date with the latest patches and updates.
```shell
bash

#!/bin/bash



# set the Redis version to check against

REDIS_VERSION="${REDIS_VERSION}"



# get the current Redis version

CURRENT_VERSION=$(redis-server --version | awk '{print $3}')



if [ "$CURRENT_VERSION" = "$REDIS_VERSION" ]; then

  echo "Redis is already up-to-date with version $REDIS_VERSION."

else

  # update Redis to the latest version

  sudo apt-get update

  sudo apt-get install redis-server -y

  echo "Redis has been updated to the latest version."

fi


```

This incident type refers to an issue with Redis replication, which means that there is a problem with the synchronization of data between Redis instances. This issue could impact the availability and performance of the system and may require immediate attention to restore the replication and ensure data consistency. The incident could be caused by various factors, such as network problems, hardware failures, or configuration issues. The incident must be investigated and resolved as soon as possible to avoid any data loss or downtime.


This incident type refers to a situation where a SQL Server database is not marked for replication sync, which means it may not be synced with its backup. This can result in data loss or inconsistencies between the primary database and its backup. It is important to address this issue promptly to ensure data integrity and prevent potential downtime or data loss.


SQL Server database not marked for replication sync.

The Redis too many masters incident occurs when there are too many master nodes in a Redis cluster, leading to connection issues and potential data loss. This can happen due to misconfiguration, network issues, or other factors, and requires immediate attention to prevent further damage.


Redis too many masters incident

This incident type refers to a situation where Redis, a popular in-memory data structure store, has rejected connections. This could happen due to various reasons such as server overload, network issues, or misconfiguration. When this occurs, it can cause disruptions in the performance of applications that rely on Redis for data storage or caching. Thus, it requires immediate attention and resolution to minimize the impact on the applications and users.


Redis Rejected Connections Incident

The Redis missing master incident occurs when the Redis cluster has no node marked as master. This can cause problems with the Redis service and may result in service disruptions or failures. It is important to address this issue promptly to ensure the smooth operation of the Redis service.


Redis missing master incident

The Redis missing backup incident refers to a situation where Redis, a popular in-memory data structure store, has not been backed up for a certain period of time. This could lead to data loss or corruption in the event of a system failure or other issues. The incident typically requires immediate attention to ensure that the data is properly backed up and that any potential issues are resolved quickly to avoid further complications.


Redis missing backup incident.

```shell
# Environment Variables

export SLAVE_HOSTNAME="PLACEHOLDER"

export MASTER_HOSTNAME="PLACEHOLDER"

export REDIS_PORT="PLACEHOLDER"

export MASTER_IP="PLACEHOLDER"

export REDIS_CONFIG_DIR="PLACEHOLDER"

export REDIS_VERSION="PLACEHOLDER"


```


### Check if Redis is running

```shell
systemctl status redis
```

### Check Redis logs for errors

```shell
tail -n 100 /var/log/redis/redis-server.log | grep -i error
```

### Verify Redis replication status

```shell
redis-cli INFO replication
```

### Check Redis slave status

```shell
redis-cli -h ${SLAVE_HOSTNAME} -p ${REDIS_PORT} INFO replication
```

### Check Redis configuration file

```shell
cat /etc/redis/redis.conf
```

### Verify Redis master configuration

```shell
redis-cli -h ${MASTER_HOSTNAME} -p ${REDIS_PORT} INFO server
```

### Verify Redis slave configuration

```shell
redis-cli -h ${SLAVE_HOSTNAME} -p ${REDIS_PORT} INFO server
```

### Check network connectivity between master and slave

```shell
ping ${MASTER_HOSTNAME}

ping ${SLAVE_HOSTNAME}
```

### Check network ports for Redis instance

```shell
netstat -anp | grep ${REDIS_PORT}
```

### Check Redis replication delay

```shell
redis-cli -h ${SLAVE_HOSTNAME} -p ${REDIS_PORT} INFO replication | grep -i delay
```

### Network connectivity issues between Redis instances, causing replication failures.

```shell


#!/bin/bash



# Check network connectivity between Redis instances

redis_ip1=${REDIS_INSTANCE1_IP}

redis_ip2=${REDIS_INSTANCE2_IP}

ping -c 5 $redis_ip1

ping -c 5 $redis_ip2



# Check Redis replication status

redis-cli -h $redis_ip1 info replication

redis-cli -h $redis_ip2 info replication


```


### Check the Redis replication configuration to ensure it's correctly set up and that there are no misconfigurations that could cause replication failures.

```shell


#!/bin/bash



# Set variables

REDIS_CONF_DIR=${REDIS_CONFIG_DIR}



# Check Redis replication configuration

if grep -q "^slaveof" "${REDIS_CONF_DIR}/redis.conf"; then

    echo "Redis replication is enabled. Configuration is correct."

else

    echo "Redis replication is not enabled. Updating configuration."

    echo "slaveof ${MASTER_IP} ${REDIS_PORT}" >> "${REDIS_CONF_DIR}/redis.conf"

    echo "Redis replication configuration updated."

fi


```

### Verify the Redis version and ensure it's up-to-date with the latest patches and updates.

```shell
bash

#!/bin/bash



# set the Redis version to check against

REDIS_VERSION="${REDIS_VERSION}"



# get the current Redis version

CURRENT_VERSION=$(redis-server --version | awk '{print $3}')



if [ "$CURRENT_VERSION" = "$REDIS_VERSION" ]; then

  echo "Redis is already up-to-date with version $REDIS_VERSION."

else

  # update Redis to the latest version

  sudo apt-get update

  sudo apt-get install redis-server -y

  echo "Redis has been updated to the latest version."

fi


```


Redis replication broken incident.

Overview

Parameters

Debug

Check if Redis is running

Check Redis logs for errors

Verify Redis replication status

Check Redis slave status

Check Redis configuration file

Verify Redis master configuration

Verify Redis slave configuration

Check network connectivity between master and slave

Check network ports for Redis instance

Check Redis replication delay

Network connectivity issues between Redis instances, causing replication failures.

Repair

Check the Redis replication configuration to ensure it's correctly set up and that there are no misconfigurations that could cause replication failures.

Verify the Redis version and ensure it's up-to-date with the latest patches and updates.

Learn more

Related Runbooks

SQL Server database not marked for replication sync.

Redis too many masters incident

Redis Rejected Connections Incident

Redis missing master incident

Support