---
id: 9593df8f-9ff3-4710-a059-5ed975582471
---

# HAProxy number of backend connection failures for host is above normal.
---

This incident type occurs when the number of failed backend connections for a particular host in HAProxy is above normal. This could indicate issues with the backend server or other errors, such as a backend without an active frontend. Correlating this metric with other metrics like `haproxy.backend.errors.resp_rate` and response codes from both frontend and backend servers can give a better understanding of the causes of the increase in backend connection errors.

### Parameters
```shell
# Environment Variables

export CONFIG_FILE="PLACEHOLDER"

export HAPROXY_LOG_FILE="PLACEHOLDER"

export BACKEND_SERVER_PORT="PLACEHOLDER"

export BACKEND_SERVER_IP="PLACEHOLDER"

export BACKEND_SERVER_NAME="PLACEHOLDER"

export FRONTEND_SERVER_NAME="PLACEHOLDER"

export BACKEND_SERVER_URL="PLACEHOLDER"

export BACKEND_SERVICE_NAME="PLACEHOLDER"
```

## Debug

### Check HAProxy's overall status
```shell
systemctl status haproxy
```

### Check the haproxy configuration file
```shell
haproxy -c -f ${CONFIG_FILE}
```

### Check haproxy's log files
```shell
tail -f ${HAPROXY_LOG_FILE}
```

### Check the number of backend connections
```shell
echo "show stat" | socat stdio /var/run/haproxy.sock | awk -F',' '{print $1,$2,$5,$18}' | column -t
```

### Check the backend servers' status
```shell
echo "show backend" | socat stdio /var/run/haproxy.sock
```

### Check for general backend errors
```shell
grep -i "backend errors" ${HAPROXY_LOG_FILE}
```

### Check the frontend and backend servers' response codes
```shell
grep -i "HTTP/1.1" ${HAPROXY_LOG_FILE} | awk '{print $9}'
```

### The backend server is experiencing some issues or downtime, causing connection failures.
```shell


#!/bin/bash



# Define variables

BACKEND_SERVER=${BACKEND_SERVER_IP}

BACKEND_PORT=${BACKEND_SERVER_PORT}




# Check if backend server is reachable

if ping -c 1 -W 30 $BACKEND_SERVER &>/dev/null; then

  echo "Backend server is reachable."

else

  echo "Backend server is unreachable. Check network connectivity."

  exit 1

fi



# Check if backend server is up and listening on specified port

if nc -z -w $PING_TIMEOUT $BACKEND_SERVER $BACKEND_PORT; then

  echo "Backend server is up and listening on port $BACKEND_PORT."

else

  echo "Backend server is down or not listening on port $BACKEND_PORT."

  exit 1

fi



# Check if backend server is experiencing any issues or downtime

if curl -sSf $BACKEND_SERVER:$BACKEND_PORT &>/dev/null; then

  echo "Backend server is responding to HTTP requests."

else

  echo "Backend server is not responding to HTTP requests. Check server logs for errors."

fi


```

## Repair

### Verify that the backend server is running and responding correctly to requests.
```shell


#!/bin/bash


# Check if backend server is running

if systemctl status ${BACKEND_SERVICE_NAME} | grep -q "Active: active (running)"

then

    echo "Backend server is running."

else

    # Restart the backend server

    echo "Backend server is not running. Restarting..."

    systemctl restart ${BACKEND_SERVICE_NAME}

    echo "Backend server has been restarted."

fi



# Check if backend server is responding correctly

if curl -IsS ${BACKEND_SERVER_URL} | head -1 | grep "200 OK"

then

    echo "Backend server is responding correctly."

else

    # Restart the backend server

    echo "Backend server is not responding correctly. Restarting..."

    systemctl restart ${BACKEND_SERVICE_NAME}

    echo "Backend server has been restarted."

fi


```


This incident type occurs when the number of failed backend connections for a particular host in HAProxy is above normal. This could indicate issues with the backend server or other errors, such as a backend without an active frontend. Correlating this metric with other metrics like `haproxy.backend.errors.resp_rate` and response codes from both frontend and backend servers can give a better understanding of the causes of the increase in backend connection errors.


This incident type occurs when the number of 5xx errors on Traffic Server is higher than usual. This can indicate issues with server performance or connectivity problems. It requires investigation to identify the root cause and resolve the issue.


High 5xx Errors on Traffic Server

This incident type refers to a situation where the Tomcat server's JDBC connection pool usage surpasses its configured limits. This can lead to a shortage of available connections for incoming requests, resulting in slow response times or even server crashes. It is important to monitor and manage the JDBC connection pool usage to prevent such incidents from occurring.


Tomcat High JDBC Connection Pool Usage Incident.

Tomcat Connection Pool Exhaustion Incident occurs when the maximum number of connections that a Tomcat server can handle has been reached. This can happen due to an increase in traffic or a sudden surge in user activity. When the connection pool is exhausted, new requests cannot be handled, and the application may become unavailable. This type of incident requires immediate attention to ensure that the application is restored to its normal functioning state.


Tomcat Connection Pool Exhaustion Incident

This incident type involves an issue with NGINX upstream peers failing, which has triggered an alert. The details of the alert include information about the percentage of failures and how they compare to predicted values over a certain time period. The incident may be resolved or ongoing, and it may have impacted a specific service or team.


Nginx Upstream peers fails

This incident type refers to a situation where the number of 5xx errors on NGINX upstreams is higher than usual. It is an indication that there could be an issue with the NGINX server or the upstream application server. This type of incident requires immediate attention as it can impact the availability and performance of the application.


Higher than usual 5xx Errors for NGINX

```shell
# Environment Variables

export CONFIG_FILE="PLACEHOLDER"

export HAPROXY_LOG_FILE="PLACEHOLDER"

export BACKEND_SERVER_PORT="PLACEHOLDER"

export BACKEND_SERVER_IP="PLACEHOLDER"

export BACKEND_SERVER_NAME="PLACEHOLDER"

export FRONTEND_SERVER_NAME="PLACEHOLDER"

export BACKEND_SERVER_URL="PLACEHOLDER"

export BACKEND_SERVICE_NAME="PLACEHOLDER"
```


### Check HAProxy's overall status

```shell
systemctl status haproxy
```

### Check the haproxy configuration file

```shell
haproxy -c -f ${CONFIG_FILE}
```

### Check haproxy's log files

```shell
tail -f ${HAPROXY_LOG_FILE}
```

### Check the number of backend connections

```shell
echo "show stat" | socat stdio /var/run/haproxy.sock | awk -F',' '{print $1,$2,$5,$18}' | column -t
```

### Check the backend servers' status

```shell
echo "show backend" | socat stdio /var/run/haproxy.sock
```

### Check for general backend errors

```shell
grep -i "backend errors" ${HAPROXY_LOG_FILE}
```

### Check the frontend and backend servers' response codes

```shell
grep -i "HTTP/1.1" ${HAPROXY_LOG_FILE} | awk '{print $9}'
```

### The backend server is experiencing some issues or downtime, causing connection failures.

```shell


#!/bin/bash



# Define variables

BACKEND_SERVER=${BACKEND_SERVER_IP}

BACKEND_PORT=${BACKEND_SERVER_PORT}




# Check if backend server is reachable

if ping -c 1 -W 30 $BACKEND_SERVER &>/dev/null; then

  echo "Backend server is reachable."

else

  echo "Backend server is unreachable. Check network connectivity."

  exit 1

fi



# Check if backend server is up and listening on specified port

if nc -z -w $PING_TIMEOUT $BACKEND_SERVER $BACKEND_PORT; then

  echo "Backend server is up and listening on port $BACKEND_PORT."

else

  echo "Backend server is down or not listening on port $BACKEND_PORT."

  exit 1

fi



# Check if backend server is experiencing any issues or downtime

if curl -sSf $BACKEND_SERVER:$BACKEND_PORT &>/dev/null; then

  echo "Backend server is responding to HTTP requests."

else

  echo "Backend server is not responding to HTTP requests. Check server logs for errors."

fi


```


### Verify that the backend server is running and responding correctly to requests.

```shell


#!/bin/bash


# Check if backend server is running

if systemctl status ${BACKEND_SERVICE_NAME} | grep -q "Active: active (running)"

then

    echo "Backend server is running."

else

    # Restart the backend server

    echo "Backend server is not running. Restarting..."

    systemctl restart ${BACKEND_SERVICE_NAME}

    echo "Backend server has been restarted."

fi



# Check if backend server is responding correctly

if curl -IsS ${BACKEND_SERVER_URL} | head -1 | grep "200 OK"

then

    echo "Backend server is responding correctly."

else

    # Restart the backend server

    echo "Backend server is not responding correctly. Restarting..."

    systemctl restart ${BACKEND_SERVICE_NAME}

    echo "Backend server has been restarted."

fi


```


HAProxy number of backend connection failures for host is above normal.

Overview

Parameters

Debug

Check HAProxy's overall status

Check the haproxy configuration file

Check haproxy's log files

Check the number of backend connections

Check the backend servers' status

Check for general backend errors

Check the frontend and backend servers' response codes

The backend server is experiencing some issues or downtime, causing connection failures.

Repair

Verify that the backend server is running and responding correctly to requests.

Learn more

Related Runbooks

High 5xx Errors on Traffic Server

Tomcat High JDBC Connection Pool Usage Incident.

Tomcat Connection Pool Exhaustion Incident

Nginx Upstream peers fails

Support