---
id: 9c9482f1-b5ad-4e97-a434-bd8144028708
---

# Primary PostgreSQL Node Failure on Linux
---

This incident type refers to a failure in the primary PostgreSQL node on a Linux system. PostgreSQL is a popular open source relational database management system used by many organizations. The primary node is responsible for handling the majority of the traffic and data replication in a PostgreSQL cluster. When the primary node fails, it can result in data loss, slow performance, and potential downtime for users. This type of incident requires immediate attention and resolution to minimize the impact on users and ensure data integrity.

### Parameters
```shell
export VERSION="PLACEHOLDER"

export PRIMARY_NODE="PLACEHOLDER"

export STANDBY_NODE="PLACEHOLDER"
```

## Debug

### Check PostgreSQL service status
```shell
systemctl status postgresql
```

### Check PostgreSQL logs for any errors
```shell
tail -f /var/log/postgresql/postgresql-${VERSION}-main.log
```

### Check PostgreSQL configuration file for any misconfigurations
```shell
cat /etc/postgresql/${VERSION}/main/postgresql.conf
```

### Check disk space and resource utilization
```shell
df -h

top
```

## Repair

### Restart the failed PostgreSQL node service to bring it back online.
```shell


#!/bin/bash



# Stop PostgreSQL service

sudo systemctl stop postgresql.service



# Start PostgreSQL service

sudo systemctl start postgresql.service



# Check if PostgreSQL service is running

if sudo systemctl is-active --quiet postgresql.service; then

  echo "PostgreSQL service successfully restarted."

else

  echo "Failed to restart PostgreSQL service."

fi


```

### If restarting the service doesn't work, try to failover to a standby node in the PostgreSQL cluster.
```shell
bash

#!/bin/bash



# Set variables

PRIMARY_NODE=${PRIMARY_NODE}

STANDBY_NODE=${STANDBY_NODE}



# Restart the PostgreSQL service on the primary node

sudo systemctl restart postgresql



# Check if the service is running

if ! systemctl is-active --quiet postgresql; then

    # If the service is not running, failover to the standby node

    sudo -u postgres pg_ctl promote -D $STANDBY_NODE

fi


```

This incident type refers to a failure in the primary PostgreSQL node on a Linux system. PostgreSQL is a popular open source relational database management system used by many organizations. The primary node is responsible for handling the majority of the traffic and data replication in a PostgreSQL cluster. When the primary node fails, it can result in data loss, slow performance, and potential downtime for users. This type of incident requires immediate attention and resolution to minimize the impact on users and ensure data integrity.


This incident type occurs when there are too many locks acquired on a Postgresql database instance. This can cause issues with database performance and functionality. It may be necessary to adjust the max\_locks\_per\_transaction setting in Postgresql to prevent this issue from occurring.


Postgresql too many locks acquired

This incident type refers to a situation where the Postgresql database has a high rollback rate, which means that a high percentage of transactions are being aborted compared to the committed ones. This can cause issues with data consistency and performance, and may require investigation and resolution by the responsible team. The incident details may include information about the affected database instance, the service or system impacted, the urgency level, and any related alerts or escalations.


Postgresql high rollback rate incident

This incident type refers to a scenario where there is a high rate of statement timeouts in a Postgresql database instance. This can lead to degraded performance and potentially impact the availability of the database. It is important to quickly identify and address the underlying cause of the timeouts to ensure the stability of the system.


Postgresql high rate statement timeout incident.

This incident type occurs when there is a failure in the authentication process for the Postgres user's password. This can happen due to a variety of reasons such as incorrect login credentials, expired passwords, or database configuration issues. It is important to address this type of incident promptly to ensure that the database remains secure and accessible only to authorized users.


Password Authentication Failure for Postgres User

This incident type occurs when the maximum number of connections allowed to a Postgres database has been reached. This prevents new connections from being established and can result in errors or downtime. It is important to monitor connection usage and adjust the maximum connection limit as needed to prevent this issue from occurring.


Postgres Connection Limit Reached

```shell
export VERSION="PLACEHOLDER"

export PRIMARY_NODE="PLACEHOLDER"

export STANDBY_NODE="PLACEHOLDER"
```


### Check PostgreSQL service status

```shell
systemctl status postgresql
```

### Check PostgreSQL logs for any errors

```shell
tail -f /var/log/postgresql/postgresql-${VERSION}-main.log
```

### Check PostgreSQL configuration file for any misconfigurations

```shell
cat /etc/postgresql/${VERSION}/main/postgresql.conf
```

### Check disk space and resource utilization

```shell
df -h

top
```


### Restart the failed PostgreSQL node service to bring it back online.

```shell


#!/bin/bash



# Stop PostgreSQL service

sudo systemctl stop postgresql.service



# Start PostgreSQL service

sudo systemctl start postgresql.service



# Check if PostgreSQL service is running

if sudo systemctl is-active --quiet postgresql.service; then

  echo "PostgreSQL service successfully restarted."

else

  echo "Failed to restart PostgreSQL service."

fi


```

### If restarting the service doesn't work, try to failover to a standby node in the PostgreSQL cluster.

```shell
bash

#!/bin/bash



# Set variables

PRIMARY_NODE=${PRIMARY_NODE}

STANDBY_NODE=${STANDBY_NODE}



# Restart the PostgreSQL service on the primary node

sudo systemctl restart postgresql



# Check if the service is running

if ! systemctl is-active --quiet postgresql; then

    # If the service is not running, failover to the standby node

    sudo -u postgres pg_ctl promote -D $STANDBY_NODE

fi


```


Primary PostgreSQL Node Failure on Linux

Overview

Parameters

Debug

Check PostgreSQL service status

Check PostgreSQL logs for any errors

Check PostgreSQL configuration file for any misconfigurations

Check disk space and resource utilization

Repair

Restart the failed PostgreSQL node service to bring it back online.

If restarting the service doesn't work, try to failover to a standby node in the PostgreSQL cluster.

Learn more

Related Runbooks

Postgresql too many locks acquired

Postgresql high rollback rate incident

Postgresql high rate statement timeout incident.

Password Authentication Failure for Postgres User

Support