---
id: 74d9877e-7871-4bfb-a1f5-f80cb66271b2
---

# PostgreSQL Replica Node Failure on Linux.
---

This incident type refers to the failure of a replica node in a PostgreSQL database system that is running on a Linux-based operating system. A replica node is a copy of the primary database node that is used to provide high availability and fault tolerance. When a replica node fails, it can result in data loss, decreased system performance, and potential downtime for users. This type of incident requires immediate attention from a software engineer to diagnose and resolve the issue as quickly as possible.

### Parameters
```shell
export PATH_TO_PSQL_BINARY="PLACEHOLDER"

export REPLICA_NODE_IP_ADDRESS="PLACEHOLDER"

export PRIMARY_NODE_IP_ADDRESS="PLACEHOLDER"

export PATH_TO_POSTGRESQL_LOG_FILE="PLACEHOLDER"
```

## Debug

### Check if the replica node is up and running
```shell
systemctl status postgresql
```

### Check the logs for any errors or warnings related to the replica node
```shell
journalctl -u postgresql
```

### Check the replication status of the replica node
```shell
${PATH_TO_PSQL_BINARY} -c "SELECT * FROM pg_stat_replication;"
```

### Check the disk space and usage on the replica node
```shell
df -h
```

### Check if the replica node is in sync with the primary node
```shell
${PATH_TO_PSQL_BINARY} -c "SELECT pg_last_xlog_receive_location();"

${PATH_TO_PSQL_BINARY} -c "SELECT pg_last_xlog_replay_location();"
```

## Repair

### Check the network connectivity between the primary and replica nodes. Ensure that there are no network issues such as high packet loss or latency.
```shell
bash

#!/bin/bash



# Get the IP addresses of the primary and replica nodes

PRIMARY_NODE_IP=${PRIMARY_NODE_IP_ADDRESS}

REPLICA_NODE_IP=${REPLICA_NODE_IP_ADDRESS}



# Ping the primary node to check network connectivity

echo "Pinging primary node..."

ping -c 3 $PRIMARY_NODE_IP



# Ping the replica node to check network connectivity

echo "Pinging replica node..."

ping -c 3 $REPLICA_NODE_IP



# Check for any packet loss or latency issues

echo "Checking for packet loss or latency issues..."

ping -c 10 $PRIMARY_NODE_IP | grep "packet loss\|rtt"

ping -c 10 $REPLICA_NODE_IP | grep "packet loss\|rtt"



echo "Network connectivity check complete."


```

### Restart the replica node and monitor the logs for any error messages.
```shell


#!/bin/bash



# Stop PostgreSQL service on replica node

sudo systemctl stop postgresql.service



# Start PostgreSQL service on replica node

sudo systemctl start postgresql.service



# Check logs for any error messages

sudo tail -f ${PATH_TO_POSTGRESQL_LOG_FILE}


```

This incident type refers to the failure of a replica node in a PostgreSQL database system that is running on a Linux-based operating system. A replica node is a copy of the primary database node that is used to provide high availability and fault tolerance. When a replica node fails, it can result in data loss, decreased system performance, and potential downtime for users. This type of incident requires immediate attention from a software engineer to diagnose and resolve the issue as quickly as possible.


This incident type refers to an issue with Redis replication, which means that there is a problem with the synchronization of data between Redis instances. This issue could impact the availability and performance of the system and may require immediate attention to restore the replication and ensure data consistency. The incident could be caused by various factors, such as network problems, hardware failures, or configuration issues. The incident must be investigated and resolved as soon as possible to avoid any data loss or downtime.


Redis replication broken incident.

This incident type occurs when there are too many locks acquired on a Postgresql database instance. This can cause issues with database performance and functionality. It may be necessary to adjust the max\_locks\_per\_transaction setting in Postgresql to prevent this issue from occurring.


Postgresql too many locks acquired

This incident type refers to a situation where the Postgresql database has a high rollback rate, which means that a high percentage of transactions are being aborted compared to the committed ones. This can cause issues with data consistency and performance, and may require investigation and resolution by the responsible team. The incident details may include information about the affected database instance, the service or system impacted, the urgency level, and any related alerts or escalations.


Postgresql high rollback rate incident

This incident type refers to a scenario where there is a high rate of statement timeouts in a Postgresql database instance. This can lead to degraded performance and potentially impact the availability of the database. It is important to quickly identify and address the underlying cause of the timeouts to ensure the stability of the system.


Postgresql high rate statement timeout incident.

This incident type occurs when there is a failure in the authentication process for the Postgres user's password. This can happen due to a variety of reasons such as incorrect login credentials, expired passwords, or database configuration issues. It is important to address this type of incident promptly to ensure that the database remains secure and accessible only to authorized users.


Password Authentication Failure for Postgres User

```shell
export PATH_TO_PSQL_BINARY="PLACEHOLDER"

export REPLICA_NODE_IP_ADDRESS="PLACEHOLDER"

export PRIMARY_NODE_IP_ADDRESS="PLACEHOLDER"

export PATH_TO_POSTGRESQL_LOG_FILE="PLACEHOLDER"
```


### Check if the replica node is up and running

```shell
systemctl status postgresql
```

### Check the logs for any errors or warnings related to the replica node

```shell
journalctl -u postgresql
```

### Check the replication status of the replica node

```shell
${PATH_TO_PSQL_BINARY} -c "SELECT * FROM pg_stat_replication;"
```

### Check the disk space and usage on the replica node

```shell
df -h
```

### Check if the replica node is in sync with the primary node

```shell
${PATH_TO_PSQL_BINARY} -c "SELECT pg_last_xlog_receive_location();"

${PATH_TO_PSQL_BINARY} -c "SELECT pg_last_xlog_replay_location();"
```


### Check the network connectivity between the primary and replica nodes. Ensure that there are no network issues such as high packet loss or latency.

```shell
bash

#!/bin/bash



# Get the IP addresses of the primary and replica nodes

PRIMARY_NODE_IP=${PRIMARY_NODE_IP_ADDRESS}

REPLICA_NODE_IP=${REPLICA_NODE_IP_ADDRESS}



# Ping the primary node to check network connectivity

echo "Pinging primary node..."

ping -c 3 $PRIMARY_NODE_IP



# Ping the replica node to check network connectivity

echo "Pinging replica node..."

ping -c 3 $REPLICA_NODE_IP



# Check for any packet loss or latency issues

echo "Checking for packet loss or latency issues..."

ping -c 10 $PRIMARY_NODE_IP | grep "packet loss\|rtt"

ping -c 10 $REPLICA_NODE_IP | grep "packet loss\|rtt"



echo "Network connectivity check complete."


```

### Restart the replica node and monitor the logs for any error messages.

```shell


#!/bin/bash



# Stop PostgreSQL service on replica node

sudo systemctl stop postgresql.service



# Start PostgreSQL service on replica node

sudo systemctl start postgresql.service



# Check logs for any error messages

sudo tail -f ${PATH_TO_POSTGRESQL_LOG_FILE}


```


PostgreSQL Replica Node Failure on Linux.

Overview

Parameters

Debug

Check if the replica node is up and running

Check the replication status of the replica node

Check the disk space and usage on the replica node

Check if the replica node is in sync with the primary node

Repair

Check the network connectivity between the primary and replica nodes. Ensure that there are no network issues such as high packet loss or latency.

Restart the replica node and monitor the logs for any error messages.

Learn more

Related Runbooks

Redis replication broken incident.

Postgresql too many locks acquired

Postgresql high rollback rate incident

Postgresql high rate statement timeout incident.

Support