---
id: 952abb82-8d45-4065-91cb-0e5fc3620a21
---
# High 4xx Errors on NGINX
---

This incident type is related to a high number of 4xx errors on NGINX upstreams. This error is higher than usual and requires investigation to identify the root cause. The incident description typically includes details on the percentage of anomalous errors, the query used to detect the errors, and other relevant information such as the event type, monitor state, and tags. The incident may require the attention of the software engineer or other relevant personnel to resolve the issue and prevent it from reoccurring in the future.

### Parameters
```shell
# Environment Variables

export ERROR_MESSAGE="PLACEHOLDER"

export SETTING_NAME="PLACEHOLDER"

export IP_ADDRESS="PLACEHOLDER"

export UPSTREAM_SERVER_URL="PLACEHOLDER"

export PATH_TO_NGINX_CONF_FILE="PLACEHOLDER"
```

## Debug

### Check the status of the NGINX service
```shell
systemctl status nginx.service
```

### Check the NGINX error log for any relevant error messages
```shell
tail -n 100 /var/log/nginx/error.log | grep "${ERROR_MESSAGE}"
```

### View NGINX access logs to identify the sources of the 4xx errors
```shell
tail -n 100 /var/log/nginx/access.log | grep " 4[0-9][0-9] "
```

### Check the configuration file for any syntax errors
```shell
nginx -t
```

### Check the NGINX configuration file for any relevant settings
```shell
cat /etc/nginx/nginx.conf | grep "${SETTING_NAME}"
```

### Check the system load and resource usage
```shell
top
```

### Check network connectivity between NGINX upstreams and clients/servers
```shell
ping ${IP_ADDRESS}
```

### Check NGINX upstream servers for any issues
```shell
curl -I ${UPSTREAM_SERVER_URL}
```

## Repair

### Define the path to the NGINX configuration file
```shell
NGINX_CONF="${PATH_TO_NGINX_CONF_FILE}"
```

### Check if the configuration file exists
```shell
if [ ! -f "$NGINX_CONF" ]; then

  echo "The NGINX configuration file does not exist."

  exit 1

fi
```

### Check the configuration file for any syntax errors
```shell
nginx -t -c "$NGINX_CONF"
```

### Restart NGINX if the configuration file is valid
```shell
if [ $? -eq 0 ]; then

  systemctl restart nginx

  echo "NGINX restarted."

else

  echo "There was an error in the NGINX configuration file."

  exit 1

fi
```


This incident type is related to a high number of 4xx errors on NGINX upstreams. This error is higher than usual and requires investigation to identify the root cause. The incident description typically includes details on the percentage of anomalous errors, the query used to detect the errors, and other relevant information such as the event type, monitor state, and tags. The incident may require the attention of the software engineer or other relevant personnel to resolve the issue and prevent it from reoccurring in the future.


This incident type occurs when the number of 5xx errors on Traffic Server is higher than usual. This can indicate issues with server performance or connectivity problems. It requires investigation to identify the root cause and resolve the issue.


High 5xx Errors on Traffic Server

This incident type occurs when the number of 4xx errors on Traffic Server is at an anomalous level, higher than usual. It could be an indicator of an issue with the server or an increase in traffic that is causing errors. It requires investigation and resolution to ensure the server is functioning correctly and not impacting users.


High 4xx Errors on Traffic Server

This incident type refers to an increase in the number of errors per second on a Tomcat server, which could indicate an issue with the server itself, the host, a deployed application, or an application servlet. This could include errors generated when the Tomcat server runs out of memory, can't find a requested file or servlet, or is unable to serve a JSP due to syntax errors in the servlet codebase. This incident type requires immediate attention to diagnose and address the underlying issue.


Increase of the errors/second rate for Tomcat server

This incident type refers to a scenario where there is a high rate of statement timeouts in a Postgresql database instance. This can lead to degraded performance and potentially impact the availability of the database. It is important to quickly identify and address the underlying cause of the timeouts to ensure the stability of the system.


Postgresql high rate statement timeout incident.

This incident type involves an issue with NGINX upstream peers failing, which has triggered an alert. The details of the alert include information about the percentage of failures and how they compare to predicted values over a certain time period. The incident may be resolved or ongoing, and it may have impacted a specific service or team.


Nginx Upstream peers fails

```shell
# Environment Variables

export ERROR_MESSAGE="PLACEHOLDER"

export SETTING_NAME="PLACEHOLDER"

export IP_ADDRESS="PLACEHOLDER"

export UPSTREAM_SERVER_URL="PLACEHOLDER"

export PATH_TO_NGINX_CONF_FILE="PLACEHOLDER"
```


### Check the status of the NGINX service

```shell
systemctl status nginx.service
```

### Check the NGINX error log for any relevant error messages

```shell
tail -n 100 /var/log/nginx/error.log | grep "${ERROR_MESSAGE}"
```

### View NGINX access logs to identify the sources of the 4xx errors

```shell
tail -n 100 /var/log/nginx/access.log | grep " 4[0-9][0-9] "
```

### Check the configuration file for any syntax errors

```shell
nginx -t
```

### Check the NGINX configuration file for any relevant settings

```shell
cat /etc/nginx/nginx.conf | grep "${SETTING_NAME}"
```

### Check the system load and resource usage

```shell
top
```

### Check network connectivity between NGINX upstreams and clients/servers

```shell
ping ${IP_ADDRESS}
```

### Check NGINX upstream servers for any issues

```shell
curl -I ${UPSTREAM_SERVER_URL}
```


### Define the path to the NGINX configuration file

```shell
NGINX_CONF="${PATH_TO_NGINX_CONF_FILE}"
```

### Check if the configuration file exists

```shell
if [ ! -f "$NGINX_CONF" ]; then

  echo "The NGINX configuration file does not exist."

  exit 1

fi
```

### Check the configuration file for any syntax errors

```shell
nginx -t -c "$NGINX_CONF"
```

### Restart NGINX if the configuration file is valid

```shell
if [ $? -eq 0 ]; then

  systemctl restart nginx

  echo "NGINX restarted."

else

  echo "There was an error in the NGINX configuration file."

  exit 1

fi
```


High 4xx Errors on NGINX

Overview

Parameters

Debug

Check the status of the NGINX service

Check the NGINX error log for any relevant error messages

View NGINX access logs to identify the sources of the 4xx errors

Check the configuration file for any syntax errors

Check the NGINX configuration file for any relevant settings

Check the system load and resource usage

Check network connectivity between NGINX upstreams and clients/servers

Check NGINX upstream servers for any issues

Repair

Define the path to the NGINX configuration file

Check if the configuration file exists

Check the configuration file for any syntax errors

Restart NGINX if the configuration file is valid

Learn more

Related Runbooks

High 5xx Errors on Traffic Server

High 4xx Errors on Traffic Server

Increase of the errors/second rate for Tomcat server

Postgresql high rate statement timeout incident.

Support