---
id: 89603dbb-606e-4478-86c2-6f599853c0bb
---

# High Nginx Latency
---

High Nginx latency typically refers to a situation where the response time of Nginx servers is significantly slower than expected. This may occur due to a variety of reasons, such as increased traffic or a misconfiguration in the server settings. High Nginx latency can cause website downtime, slow page load times, and poor user experience. It is important to address this incident type promptly to prevent any negative impact on users or business operations.

### Parameters
```shell
# Environment Variables

export SERVER_IP="PLACEHOLDER"

export PORT="PLACEHOLDER"

export NGINX_CONFIG_FILE="PLACEHOLDER"

export PATH_TO_NGINX_CONFIG_FILE="PLACEHOLDER"

export UPSTREAM_SERVER_URL="PLACEHOLDER"

export UPSTREAM_CONFIG_KEY="PLACEHOLDER"
```

## Debug

### Step 1: Check CPU usage
```shell
top
```

### Step 2: Check memory usage
```shell
free -m
```

### Step 3: Check disk space usage
```shell
df -h
```

### Step 4: Check network traffic
```shell
iftop
```

### Step 5: Check Nginx status
```shell
systemctl status nginx
```

### Step 6: Check Nginx error logs
```shell
tail -f /var/log/nginx/error.log
```

### Step 7: Check Nginx access logs
```shell
tail -f /var/log/nginx/access.log
```

### Step 8: Check Nginx configuration file for errors
```shell
nginx -t
```

### Step 9: Check server load
```shell
uptime
```

### Step 10: Check for any processes consuming high CPU usage
```shell
ps aux --sort=-%cpu | head
```

### Step 11: Check for any network connectivity issues
```shell
ping ${SERVER_IP}
```

### Step 12: Check for port connectivity
```shell
telnet ${SERVER_IP} ${PORT}
```

### Step 13: Check for the response time of the server
```shell
curl -o /dev/null -s -w "Server response time: %{time_total}\n" http://${SERVER_IP}
```

### Step 14: Check for the response time of Nginx
```shell
curl -o /dev/null -s -w "Nginx response time: %{time_total}\n" http://${SERVER_IP}:${PORT}
```

## Repair

### Check if Nginx configuration file exists
```shell
if [ ! -f ${NGINX_CONFIG_FILE} ]; then

  echo "Nginx configuration file does not exist."

  exit 1

fi
```

### Test Nginx configuration file
```shell
nginx -t -c ${NGINX_CONFIG_FILE}
```

### Check if Nginx configuration file test is successful
```shell
if [ $? -eq 0 ]; then

  echo "Nginx configuration file test is successful."

else

  echo "Nginx configuration file test failed."

  exit 1

fi
```

### Check the Nginx configuration file and ensure that all the directives are properly configured.
```shell


#!/bin/bash



# Set the path to the Nginx configuration file

nginx_config_file=${PATH_TO_NGINX_CONFIG_FILE}



# Check if the Nginx configuration file exists

if [ ! -f "$nginx_config_file" ]; then

  echo "Error: Nginx configuration file not found at $nginx_config_file"

  exit 1

fi



# Check the configuration file for any syntax errors

nginx -t -c "$nginx_config_file"

if [ $? -ne 0 ]; then

  echo "Error: Nginx configuration file contains syntax errors"

  exit 1

fi



# Restart Nginx to apply any changes to the configuration file

systemctl restart nginx



echo "Nginx configuration file has been checked and Nginx has been restarted"


```

### Check the upstream server's status and ensure that it is properly configured.
```shell


#!/bin/bash



# Check the upstream server's status

upstream_status=$(curl -s -o /dev/null -w "%{http_code}" ${UPSTREAM_SERVER_URL})



# Check if the upstream server is responding with 200 OK status

if [[ "$upstream_status" == "200" ]]; then

    echo "The upstream server is responding with HTTP status code 200."

else

    echo "The upstream server is not responding with HTTP status code 200."

    exit 1

fi



# Ensure that the upstream server is properly configured

upstream_config=$(curl -s ${UPSTREAM_SERVER_URL} | grep "${UPSTREAM_CONFIG_KEY}")



# Check if the upstream server's configuration is correct

if [[ -z "$upstream_config" ]]; then

    echo "The upstream server's configuration is incorrect."

    exit 1

else

    echo "The upstream server's configuration is correct."

fi



# If both upstream status and configuration are correct, exit with success status code

exit 0


```

High Nginx latency typically refers to a situation where the response time of Nginx servers is significantly slower than expected. This may occur due to a variety of reasons, such as increased traffic or a misconfiguration in the server settings. High Nginx latency can cause website downtime, slow page load times, and poor user experience. It is important to address this incident type promptly to prevent any negative impact on users or business operations.


This incident type indicates that there is a high latency issue in the execution of a Spark job. Spark is a distributed computing framework that is used for processing large datasets. High latency in this context means that the time taken to execute the Spark job is significantly longer than expected or normal. This can result in delays in processing data and can impact the performance of the application or system that is utilizing Spark.


High Latency Incident for Spark Job Execution.

This incident type involves an issue with NGINX upstream peers failing, which has triggered an alert. The details of the alert include information about the percentage of failures and how they compare to predicted values over a certain time period. The incident may be resolved or ongoing, and it may have impacted a specific service or team.


Nginx Upstream peers fails

This incident type refers to a situation where the number of 5xx errors on NGINX upstreams is higher than usual. It is an indication that there could be an issue with the NGINX server or the upstream application server. This type of incident requires immediate attention as it can impact the availability and performance of the application.


Higher than usual 5xx Errors for NGINX

This incident type is related to a high number of 4xx errors on NGINX upstreams. This error is higher than usual and requires investigation to identify the root cause. The incident description typically includes details on the percentage of anomalous errors, the query used to detect the errors, and other relevant information such as the event type, monitor state, and tags. The incident may require the attention of the software engineer or other relevant personnel to resolve the issue and prevent it from reoccurring in the future.


High 4xx Errors on NGINX

The "High error rate on NGINX" incident type refers to a situation where the error rate on the NGINX server is above 1% for the last 5 minutes. This can result in degraded performance or downtime of the affected service, impacting user experience and potentially leading to lost revenue. The incident requires immediate attention and resolution to minimize the impact on users and prevent further damage.


High error rate on NGINX incident

```shell
# Environment Variables

export SERVER_IP="PLACEHOLDER"

export PORT="PLACEHOLDER"

export NGINX_CONFIG_FILE="PLACEHOLDER"

export PATH_TO_NGINX_CONFIG_FILE="PLACEHOLDER"

export UPSTREAM_SERVER_URL="PLACEHOLDER"

export UPSTREAM_CONFIG_KEY="PLACEHOLDER"
```


### Step 1: Check CPU usage

```shell
top
```

### Step 2: Check memory usage

```shell
free -m
```

### Step 3: Check disk space usage

```shell
df -h
```

### Step 4: Check network traffic

```shell
iftop
```

### Step 5: Check Nginx status

```shell
systemctl status nginx
```

### Step 6: Check Nginx error logs

```shell
tail -f /var/log/nginx/error.log
```

### Step 7: Check Nginx access logs

```shell
tail -f /var/log/nginx/access.log
```

### Step 8: Check Nginx configuration file for errors

```shell
nginx -t
```

### Step 9: Check server load

```shell
uptime
```

### Step 10: Check for any processes consuming high CPU usage

```shell
ps aux --sort=-%cpu | head
```

### Step 11: Check for any network connectivity issues

```shell
ping ${SERVER_IP}
```

### Step 12: Check for port connectivity

```shell
telnet ${SERVER_IP} ${PORT}
```

### Step 13: Check for the response time of the server

```shell
curl -o /dev/null -s -w "Server response time: %{time_total}\n" http://${SERVER_IP}
```

### Step 14: Check for the response time of Nginx

```shell
curl -o /dev/null -s -w "Nginx response time: %{time_total}\n" http://${SERVER_IP}:${PORT}
```


### Check if Nginx configuration file exists

```shell
if [ ! -f ${NGINX_CONFIG_FILE} ]; then

  echo "Nginx configuration file does not exist."

  exit 1

fi
```

### Test Nginx configuration file

```shell
nginx -t -c ${NGINX_CONFIG_FILE}
```

### Check if Nginx configuration file test is successful

```shell
if [ $? -eq 0 ]; then

  echo "Nginx configuration file test is successful."

else

  echo "Nginx configuration file test failed."

  exit 1

fi
```

### Check the Nginx configuration file and ensure that all the directives are properly configured.

```shell


#!/bin/bash



# Set the path to the Nginx configuration file

nginx_config_file=${PATH_TO_NGINX_CONFIG_FILE}



# Check if the Nginx configuration file exists

if [ ! -f "$nginx_config_file" ]; then

  echo "Error: Nginx configuration file not found at $nginx_config_file"

  exit 1

fi



# Check the configuration file for any syntax errors

nginx -t -c "$nginx_config_file"

if [ $? -ne 0 ]; then

  echo "Error: Nginx configuration file contains syntax errors"

  exit 1

fi



# Restart Nginx to apply any changes to the configuration file

systemctl restart nginx



echo "Nginx configuration file has been checked and Nginx has been restarted"


```

### Check the upstream server's status and ensure that it is properly configured.

```shell


#!/bin/bash



# Check the upstream server's status

upstream_status=$(curl -s -o /dev/null -w "%{http_code}" ${UPSTREAM_SERVER_URL})



# Check if the upstream server is responding with 200 OK status

if [[ "$upstream_status" == "200" ]]; then

    echo "The upstream server is responding with HTTP status code 200."

else

    echo "The upstream server is not responding with HTTP status code 200."

    exit 1

fi



# Ensure that the upstream server is properly configured

upstream_config=$(curl -s ${UPSTREAM_SERVER_URL} | grep "${UPSTREAM_CONFIG_KEY}")



# Check if the upstream server's configuration is correct

if [[ -z "$upstream_config" ]]; then

    echo "The upstream server's configuration is incorrect."

    exit 1

else

    echo "The upstream server's configuration is correct."

fi



# If both upstream status and configuration are correct, exit with success status code

exit 0


```


High Nginx Latency

Overview

Parameters

Debug

Step 1: Check CPU usage

Step 2: Check memory usage

Step 3: Check disk space usage

Step 4: Check network traffic

Step 5: Check Nginx status

Step 6: Check Nginx error logs

Step 7: Check Nginx access logs

Step 8: Check Nginx configuration file for errors

Step 9: Check server load

Step 10: Check for any processes consuming high CPU usage

Step 11: Check for any network connectivity issues

Step 12: Check for port connectivity

Step 13: Check for the response time of the server

Step 14: Check for the response time of Nginx

Repair

Check if Nginx configuration file exists

Test Nginx configuration file

Check if Nginx configuration file test is successful

Check the Nginx configuration file and ensure that all the directives are properly configured.

Check the upstream server's status and ensure that it is properly configured.

Learn more

Related Runbooks

High Latency Incident for Spark Job Execution.

Nginx Upstream peers fails

Higher than usual 5xx Errors for NGINX

High 4xx Errors on NGINX

Support