---
id: 01ec4d56-b832-4710-ae16-155029b2f148
---



# Read latency higher than average for MongoDB host.
---

This incident type involves an alert triggered when read latency is higher than usual for a MongoDB host. The alert is triggered when at least 100% of the average read latency values have been more than 2 deviations above the predicted values during the last 15 minutes. The incident requires monitoring and investigation to identify the root cause of the increased read latency and take appropriate actions to resolve it.

### Parameters
```shell
# Environment Variables

export MONGODB_HOSTNAME="PLACEHOLDER"

export USERNAME="PLACEHOLDER"

export MONGODB_ATLAS_PORT="PLACEHOLDER"

export PASSWORD="PLACEHOLDER"

export DATABASE_NAME="PLACEHOLDER"

export COLLECTION_NAME="PLACEHOLDER"

export CLIENT_IP="PLACEHOLDER"
```

## Debug

### 1. Check if MongoDB is running and accessible
```shell
ping ${MONGODB_HOSTNAME}
```

### 2. Check MongoDB connection status
```shell
mongo ${MONGODB_HOSTNAME}:${MONGODB_ATLAS_PORT} -u ${USERNAME} -p ${PASSWORD} --authenticationDatabase admin --eval "db.runCommand({ping:1})"
```

### 3. Check if there are any MongoDB operations currently running
```shell
mongo ${MONGODB_HOSTNAME}:${MONGODB_ATLAS_PORT} -u ${USERNAME} -p ${PASSWORD} --authenticationDatabase admin --eval "db.currentOp()"
```

### 4. Check MongoDB latency
```shell
mongostat --host ${MONGODB_HOSTNAME} --port ${MONGODB_ATLAS_PORT}
```

### 5. Check the status of MongoDB indexes
```shell
mongo ${MONGODB_HOSTNAME}:${MONGODB_ATLAS_PORT}/${DATABASE_NAME} -u ${USERNAME} -p ${PASSWORD} --authenticationDatabase admin --eval "db.${COLLECTION_NAME}.getIndexes()"
```

### 6. Check MongoDB disk usage
```shell
mongo ${MONGODB_HOSTNAME}:${MONGODB_ATLAS_PORT}/${DATABASE_NAME} -u ${USERNAME} -p ${PASSWORD} --authenticationDatabase admin --eval "db.stats()"
```
### 7. Check the read latency for the MongoDB host
```shell
mongo ${MONGODB_HOSTNAME}:${MONGODB_ATLAS_PORT}/${DATABASE_NAME} -u ${USERNAME} -p ${PASSWORD} --authenticationDatabase admin --eval "db.currentOp({'secs_running': {$gte: 1}}).inprog.forEach(function(op) { printjson(op.opid + ': ' + op.secs_running + 'secs'); });"

### Network latency issues between the MongoDB host and the client.
```shell
#!/bin/bash
# Define the MongoDB host and client IP addresses

MONGODB_HOST=${MONGODB_HOSTNAME}

CLIENT_IP=${CLIENT_IP}

# Check if the MongoDB host is reachable from the client

ping -c 3 $MONGODB_HOST > /dev/null

if [ $? -eq 0 ]; then

    echo "MongoDB host is reachable from the client."

else

    echo "MongoDB host is not reachable from the client."

fi

# Check the network latency between the MongoDB host and the client

ping -c 3 $MONGODB_HOST | tail -1| awk '{print $4}' | cut -d '/' -f 2 > /dev/null

if [ $? -eq 0 ]; then

    latency=$(ping -c 3 $MONGODB_HOST | tail -1| awk '{print $4}' | cut -d '/' -f 2)

    echo "Network latency between the MongoDB host and the client is $latency ms."

else

    echo "Unable to determine network latency between the MongoDB host and the client."

fi

```

## Repair

### Optimize the MongoDB configuration settings to reduce read latency.
```shell
#!/bin/bash

# Define the MongoDB configuration file path

CONFIG_FILE="/etc/mongod.conf"

# Set the read concern level to "majority"

sed -i 's/readConcernLevel.*/readConcernLevel: majority/g' $CONFIG_FILE

# Increase the number of connections allowed to the database

sed -i 's/maxConnectionsPerHost.*/maxConnectionsPerHost: 100/g' $CONFIG_FILE

# Enable the query optimizer

sed -i 's/enableLocalhostAuthBypass.*/enableLocalhostAuthBypass: true/g' $CONFIG_FILE

# Restart the MongoDB service to apply the changes

systemctl restart mongodb.service


```

This incident type involves an alert triggered when read latency is higher than usual for a MongoDB host. The alert is triggered when at least 100% of the average read latency values have been more than 2 deviations above the predicted values during the last 15 minutes. The incident requires monitoring and investigation to identify the root cause of the increased read latency and take appropriate actions to resolve it.


This incident type indicates that there is a high latency issue in the execution of a Spark job. Spark is a distributed computing framework that is used for processing large datasets. High latency in this context means that the time taken to execute the Spark job is significantly longer than expected or normal. This can result in delays in processing data and can impact the performance of the application or system that is utilizing Spark.


High Latency Incident for Spark Job Execution.

This incident type refers to a situation where the write latency for a MongoDB database is higher than the average. This might cause delays in write operations and can negatively impact the performance of the system. The incident might be triggered by an alert that indicates that the write latency has exceeded a certain threshold, and it requires immediate attention from a software engineer to identify and resolve the underlying issue. The incident can be resolved by identifying and fixing the root cause of the latency issue, and ensuring that the system is functioning within acceptable performance parameters.


MongoDB Write Latency Higher Than Average

This incident type describes an issue where the efficiency of queries in MongoDB is degrading. The incident may be triggered by an alert monitoring system that detects an anomaly in the performance of queries executed in the database. The incident may result in a decrease in system performance and may require the attention of a software engineer to troubleshoot and resolve the issue.


MongoDB Efficiency of queries is degrading

This incident type refers to an issue with high virtual memory usage in a MongoDB instance. It can lead to performance degradation or even service downtime. It requires immediate attention from the responsible team to investigate the root cause and take corrective measures to resolve the issue.


MongoDB virtual memory usage incident.

A MongoDB Replication Lag Incident occurs when there is a delay in the replication of data from the primary MongoDB instance to its secondary instances. This delay can cause data inconsistencies and affect the performance of the application. It is important to identify and resolve this incident as soon as possible to ensure that the application is functioning optimally.


MongoDB Replication Lag Incident

```shell
# Environment Variables

export MONGODB_HOSTNAME="PLACEHOLDER"

export USERNAME="PLACEHOLDER"

export MONGODB_ATLAS_PORT="PLACEHOLDER"

export PASSWORD="PLACEHOLDER"

export DATABASE_NAME="PLACEHOLDER"

export COLLECTION_NAME="PLACEHOLDER"

export CLIENT_IP="PLACEHOLDER"
```


### 1. Check if MongoDB is running and accessible

```shell
ping ${MONGODB_HOSTNAME}
```

### 2. Check MongoDB connection status

```shell
mongo ${MONGODB_HOSTNAME}:${MONGODB_ATLAS_PORT} -u ${USERNAME} -p ${PASSWORD} --authenticationDatabase admin --eval "db.runCommand({ping:1})"
```

### 3. Check if there are any MongoDB operations currently running

```shell
mongo ${MONGODB_HOSTNAME}:${MONGODB_ATLAS_PORT} -u ${USERNAME} -p ${PASSWORD} --authenticationDatabase admin --eval "db.currentOp()"
```

### 4. Check MongoDB latency

```shell
mongostat --host ${MONGODB_HOSTNAME} --port ${MONGODB_ATLAS_PORT}
```

### 5. Check the status of MongoDB indexes

```shell
mongo ${MONGODB_HOSTNAME}:${MONGODB_ATLAS_PORT}/${DATABASE_NAME} -u ${USERNAME} -p ${PASSWORD} --authenticationDatabase admin --eval "db.${COLLECTION_NAME}.getIndexes()"
```

### 6. Check MongoDB disk usage

```shell
mongo ${MONGODB_HOSTNAME}:${MONGODB_ATLAS_PORT}/${DATABASE_NAME} -u ${USERNAME} -p ${PASSWORD} --authenticationDatabase admin --eval "db.stats()"
```

### 7. Check the read latency for the MongoDB host

````shell
mongo ${MONGODB_HOSTNAME}:${MONGODB_ATLAS_PORT}/${DATABASE_NAME} -u ${USERNAME} -p ${PASSWORD} --authenticationDatabase admin --eval "db.currentOp({'secs_running': {$gte: 1}}).inprog.forEach(function(op) { printjson(op.opid + ': ' + op.secs_running + 'secs'); });"

### Network latency issues between the MongoDB host and the client.
```shell
#!/bin/bash
# Define the MongoDB host and client IP addresses

MONGODB_HOST=${MONGODB_HOSTNAME}

CLIENT_IP=${CLIENT_IP}

# Check if the MongoDB host is reachable from the client

ping -c 3 $MONGODB_HOST > /dev/null

if [ $? -eq 0 ]; then

    echo "MongoDB host is reachable from the client."

else

    echo "MongoDB host is not reachable from the client."

fi

# Check the network latency between the MongoDB host and the client

ping -c 3 $MONGODB_HOST | tail -1| awk '{print $4}' | cut -d '/' -f 2 > /dev/null

if [ $? -eq 0 ]; then

    latency=$(ping -c 3 $MONGODB_HOST | tail -1| awk '{print $4}' | cut -d '/' -f 2)

    echo "Network latency between the MongoDB host and the client is $latency ms."

else

    echo "Unable to determine network latency between the MongoDB host and the client."

fi

````


### Optimize the MongoDB configuration settings to reduce read latency.

```shell
#!/bin/bash

# Define the MongoDB configuration file path

CONFIG_FILE="/etc/mongod.conf"

# Set the read concern level to "majority"

sed -i 's/readConcernLevel.*/readConcernLevel: majority/g' $CONFIG_FILE

# Increase the number of connections allowed to the database

sed -i 's/maxConnectionsPerHost.*/maxConnectionsPerHost: 100/g' $CONFIG_FILE

# Enable the query optimizer

sed -i 's/enableLocalhostAuthBypass.*/enableLocalhostAuthBypass: true/g' $CONFIG_FILE

# Restart the MongoDB service to apply the changes

systemctl restart mongodb.service


```


Read latency higher than average for MongoDB host.

Overview

Parameters

Debug

1. Check if MongoDB is running and accessible

2. Check MongoDB connection status

3. Check if there are any MongoDB operations currently running

4. Check MongoDB latency

5. Check the status of MongoDB indexes

6. Check MongoDB disk usage

7. Check the read latency for the MongoDB host

Repair

Optimize the MongoDB configuration settings to reduce read latency.

Learn more

Related Runbooks

High Latency Incident for Spark Job Execution.

MongoDB Write Latency Higher Than Average

MongoDB Efficiency of queries is degrading

MongoDB virtual memory usage incident.

Support