---
id: 5c8778c4-0c4d-4a52-bc60-0f2094d18074
---

# High error rate on NGINX incident
---

The "High error rate on NGINX" incident type refers to a situation where the error rate on the NGINX server is above 1% for the last 5 minutes. This can result in degraded performance or downtime of the affected service, impacting user experience and potentially leading to lost revenue. The incident requires immediate attention and resolution to minimize the impact on users and prevent further damage.

### Parameters
```shell
# Environment Variables

export NGINX_PID="PLACEHOLDER"

export NETWORK_INTERFACE="PLACEHOLDER"

export TARGET_GROUP_ARN="PLACEHOLDER"

export LAUNCH_TEMPLATE_ID="PLACEHOLDER"

export INSTANCE_TYPE="PLACEHOLDER"

export NUMBER_OF_INSTANCES="PLACEHOLDER"
```

## Debug

### Check the status of the NGINX service
```shell
systemctl status nginx
```

### Check the error log for NGINX
```shell
tail -n 100 /var/log/nginx/error.log
```

### Check the access log for NGINX
```shell
tail -n 100 /var/log/nginx/access.log
```

### Check the NGINX configuration file for syntax errors
```shell
nginx -t
```

### Check the NGINX configuration file for errors in the upstream server configuration
```shell
grep -r "upstream" /etc/nginx/
```

### Check the system load average
```shell
uptime
```

### Check the CPU usage of the NGINX process
```shell
ps -p ${NGINX_PID} -o %cpu
```

### Check the memory usage of the NGINX process
```shell
ps -p ${NGINX_PID} -o %mem
```

### Check the network traffic on the NGINX server
```shell
iftop -i ${NETWORK_INTERFACE}
```

### Check the NGINX configuration for the maximum number of connections allowed
```shell
grep -r "worker_connections" /etc/nginx/
```

## Repair

### Define variables
```shell
INSTANCE_COUNT=${NUMBER_OF_INSTANCES}

INSTANCE_TYPE=${INSTANCE_TYPE}

LAUNCH_TEMPLATE_ID=${LAUNCH_TEMPLATE_ID}

TARGET_GROUP_ARN=${TARGET_GROUP_ARN}
```

### Add more instances to serve increased load
```shell
aws ec2 run-instances \

    --count $INSTANCE_COUNT \

    --instance-type $INSTANCE_TYPE \

    --launch-template LaunchTemplateId=$LAUNCH_TEMPLATE_ID \

    --user-data file://nginx_install_script.sh
```

### Wait for the new instances to start running
```shell
aws ec2 wait instance-running
```

### Get the IDs of the new instances
```shell
NEW_INSTANCE_IDS=$(aws ec2 describe-instances \

    --filters "Name=instance-state-name,Values=running" \

    --query "Reservations[].Instances[].InstanceId")
```

### Register the new instances with the target group
```shell
aws elbv2 register-targets \

    --target-group-arn $TARGET_GROUP_ARN \

    --targets Id=$NEW_INSTANCE_IDS
```

The "High error rate on NGINX" incident type refers to a situation where the error rate on the NGINX server is above 1% for the last 5 minutes. This can result in degraded performance or downtime of the affected service, impacting user experience and potentially leading to lost revenue. The incident requires immediate attention and resolution to minimize the impact on users and prevent further damage.


This incident type occurs when the number of 5xx errors on Traffic Server is higher than usual. This can indicate issues with server performance or connectivity problems. It requires investigation to identify the root cause and resolve the issue.


High 5xx Errors on Traffic Server

This incident type occurs when the number of 4xx errors on Traffic Server is at an anomalous level, higher than usual. It could be an indicator of an issue with the server or an increase in traffic that is causing errors. It requires investigation and resolution to ensure the server is functioning correctly and not impacting users.


High 4xx Errors on Traffic Server

This incident type refers to an increase in the number of errors per second on a Tomcat server, which could indicate an issue with the server itself, the host, a deployed application, or an application servlet. This could include errors generated when the Tomcat server runs out of memory, can't find a requested file or servlet, or is unable to serve a JSP due to syntax errors in the servlet codebase. This incident type requires immediate attention to diagnose and address the underlying issue.


Increase of the errors/second rate for Tomcat server

This incident type refers to a scenario where there is a high rate of statement timeouts in a Postgresql database instance. This can lead to degraded performance and potentially impact the availability of the database. It is important to quickly identify and address the underlying cause of the timeouts to ensure the stability of the system.


Postgresql high rate statement timeout incident.

This incident type involves an issue with NGINX upstream peers failing, which has triggered an alert. The details of the alert include information about the percentage of failures and how they compare to predicted values over a certain time period. The incident may be resolved or ongoing, and it may have impacted a specific service or team.


Nginx Upstream peers fails

```shell
# Environment Variables

export NGINX_PID="PLACEHOLDER"

export NETWORK_INTERFACE="PLACEHOLDER"

export TARGET_GROUP_ARN="PLACEHOLDER"

export LAUNCH_TEMPLATE_ID="PLACEHOLDER"

export INSTANCE_TYPE="PLACEHOLDER"

export NUMBER_OF_INSTANCES="PLACEHOLDER"
```


### Check the status of the NGINX service

```shell
systemctl status nginx
```

### Check the error log for NGINX

```shell
tail -n 100 /var/log/nginx/error.log
```

### Check the access log for NGINX

```shell
tail -n 100 /var/log/nginx/access.log
```

### Check the NGINX configuration file for syntax errors

```shell
nginx -t
```

### Check the NGINX configuration file for errors in the upstream server configuration

```shell
grep -r "upstream" /etc/nginx/
```

### Check the system load average

```shell
uptime
```

### Check the CPU usage of the NGINX process

```shell
ps -p ${NGINX_PID} -o %cpu
```

### Check the memory usage of the NGINX process

```shell
ps -p ${NGINX_PID} -o %mem
```

### Check the network traffic on the NGINX server

```shell
iftop -i ${NETWORK_INTERFACE}
```

### Check the NGINX configuration for the maximum number of connections allowed

```shell
grep -r "worker_connections" /etc/nginx/
```


### Define variables

```shell
INSTANCE_COUNT=${NUMBER_OF_INSTANCES}

INSTANCE_TYPE=${INSTANCE_TYPE}

LAUNCH_TEMPLATE_ID=${LAUNCH_TEMPLATE_ID}

TARGET_GROUP_ARN=${TARGET_GROUP_ARN}
```

### Add more instances to serve increased load

```shell
aws ec2 run-instances \

    --count $INSTANCE_COUNT \

    --instance-type $INSTANCE_TYPE \

    --launch-template LaunchTemplateId=$LAUNCH_TEMPLATE_ID \

    --user-data file://nginx_install_script.sh
```

### Wait for the new instances to start running

```shell
aws ec2 wait instance-running
```

### Get the IDs of the new instances

```shell
NEW_INSTANCE_IDS=$(aws ec2 describe-instances \

    --filters "Name=instance-state-name,Values=running" \

    --query "Reservations[].Instances[].InstanceId")
```

### Register the new instances with the target group

```shell
aws elbv2 register-targets \

    --target-group-arn $TARGET_GROUP_ARN \

    --targets Id=$NEW_INSTANCE_IDS
```


High error rate on NGINX incident

Overview

Parameters

Debug

Check the status of the NGINX service

Check the error log for NGINX

Check the access log for NGINX

Check the NGINX configuration file for syntax errors

Check the NGINX configuration file for errors in the upstream server configuration

Check the system load average

Check the CPU usage of the NGINX process

Check the memory usage of the NGINX process

Check the network traffic on the NGINX server

Check the NGINX configuration for the maximum number of connections allowed

Repair

Define variables

Add more instances to serve increased load

Wait for the new instances to start running

Get the IDs of the new instances

Register the new instances with the target group

Learn more

Related Runbooks

High 5xx Errors on Traffic Server

High 4xx Errors on Traffic Server

Increase of the errors/second rate for Tomcat server

Postgresql high rate statement timeout incident.

Support