---
id: 1d7fce10-c993-4305-9f9a-3afc88a04a30
---

# MySQL instance restart incident
---

This incident type refers to the restart of a MySQL instance that has caused an alert to trigger. It may be related to issues with the MySQL database or the server hosting the instance. The incident requires investigation and resolution to ensure the proper functioning of the affected services.

### Parameters
```shell
# Environment Variables

export SERVER_IP_ADDRESS="PLACEHOLDER"

export SERVICE_NAME="PLACEHOLDER"

export PATH_TO_MYSQL_LOG_FILE="PLACEHOLDER"

export OLD_VALUE="PLACEHOLDER"

export PATH_TO_MYSQL_CONFIG_FILE="PLACEHOLDER"

export MAX_CONNECTIONS="PLACEHOLDER" 
```

## Debug

### Check MySQL service status
```shell
systemctl status mysql
```

### Check MySQL logs for any errors
```shell
tail -n 100 /var/log/mysql/error.log
```

### Check the uptime of the server to see if there were any recent reboots
```shell
uptime
```

### Check if there are any resource constraints that could have caused the restart
```shell
top
```

### Check the MySQL configuration file for any issues
```shell
cat /etc/mysql/my.cnf
```

### Check the system logs for any other errors or warnings
```shell
dmesg | tail -n 50
```

### Check the disk usage to see if there are any disk space issues
```shell
df -h
```

### Check the network connectivity to the server
```shell
ping ${SERVER_IP_ADDRESS}
```

### Check if there are any other services running on the same machine that could have caused the restart
```shell
ps aux | grep -i ${SERVICE_NAME}
```

## Repair

### Set the log file path
```shell
LOG_FILE=${PATH_TO_MYSQL_LOG_FILE}
```

### Check if the log file exists
```shell
if [ ! -f "$LOG_FILE" ]; then

  echo "ERROR: Log file $LOG_FILE not found"

  exit 1

fi
```

### Search the log file for errors
```shell
ERRORS=$(grep -i "error" "$LOG_FILE")
```

### Check if there are any errors
```shell
if [ -n "$ERRORS" ]; then

  echo "ERRORS FOUND:"

  echo "$ERRORS"

else

  echo "No errors found in log file"

fi
```

### Review the configuration settings of the MySQL instance to ensure that they are not causing the restart.
```shell


#!/bin/bash



# Set the path to the MySQL configuration file

MYSQL_CONF=${PATH_TO_MYSQL_CONFIG_FILE}



# Check if the configuration file exists

if [ ! -f $MYSQL_CONF ]; then

    echo "Error: MySQL configuration file not found at $MYSQL_CONF"

    exit 1

fi



# Check the MySQL configuration for any settings that may cause the restart

if grep -q "max_connections" $MYSQL_CONF; then

    # If the max_connections setting is found, reduce it to prevent overloading the server

    sed -i 's/max_connections=${OLD_VALUE}/max_connections=${MAX_CONNECTIONS}/' $MYSQL_CONF

    echo "max_connections setting updated in MySQL configuration file."

fi



# Restart the MySQL instance to apply the new configuration

systemctl restart mysql



echo "MySQL instance has been restarted with updated configuration settings."


```

This incident type refers to the restart of a MySQL instance that has caused an alert to trigger. It may be related to issues with the MySQL database or the server hosting the instance. The incident requires investigation and resolution to ensure the proper functioning of the affected services.


This incident type refers to a situation where the Tomcat server's JDBC connection pool usage surpasses its configured limits. This can lead to a shortage of available connections for incoming requests, resulting in slow response times or even server crashes. It is important to monitor and manage the JDBC connection pool usage to prevent such incidents from occurring.


Tomcat High JDBC Connection Pool Usage Incident.

The Redis too many masters incident occurs when there are too many master nodes in a Redis cluster, leading to connection issues and potential data loss. This can happen due to misconfiguration, network issues, or other factors, and requires immediate attention to prevent further damage.


Redis too many masters incident

This incident type refers to a sudden and abnormal drop in the SELECT query rate on a MySQL server. This can potentially cause disruption in the service and affect the user experience. The cause of the drop could be due to a variety of factors such as database issues, server overload, or network problems. It requires immediate attention and investigation to identify the root cause and resolve the issue to prevent further impact.


Unusual drop in SELECT query rate on MySQL server

This incident type occurs when the number of concurrent connections to a MySQL database server exceeds the configured limit of 80%. This can lead to performance degradation or even database server crashes, making the application or system that relies on the database server unavailable for the end-users. This type of incident typically requires immediate attention and resolution to restore normal operation of the affected service.


MySQL too many connections incident

A MySQL slow queries incident is when a MySQL server is experiencing slow queries. This can result in degraded performance and responsiveness of the system. It is important to resolve this incident quickly to minimize impact on users.


MySQL slow queries incident

```shell
# Environment Variables

export SERVER_IP_ADDRESS="PLACEHOLDER"

export SERVICE_NAME="PLACEHOLDER"

export PATH_TO_MYSQL_LOG_FILE="PLACEHOLDER"

export OLD_VALUE="PLACEHOLDER"

export PATH_TO_MYSQL_CONFIG_FILE="PLACEHOLDER"

export MAX_CONNECTIONS="PLACEHOLDER" 
```


### Check MySQL service status

```shell
systemctl status mysql
```

### Check MySQL logs for any errors

```shell
tail -n 100 /var/log/mysql/error.log
```

### Check the uptime of the server to see if there were any recent reboots

```shell
uptime
```

### Check if there are any resource constraints that could have caused the restart

```shell
top
```

### Check the MySQL configuration file for any issues

```shell
cat /etc/mysql/my.cnf
```

### Check the system logs for any other errors or warnings

```shell
dmesg | tail -n 50
```

### Check the disk usage to see if there are any disk space issues

```shell
df -h
```

### Check the network connectivity to the server

```shell
ping ${SERVER_IP_ADDRESS}
```

### Check if there are any other services running on the same machine that could have caused the restart

```shell
ps aux | grep -i ${SERVICE_NAME}
```


### Set the log file path

```shell
LOG_FILE=${PATH_TO_MYSQL_LOG_FILE}
```

### Check if the log file exists

```shell
if [ ! -f "$LOG_FILE" ]; then

  echo "ERROR: Log file $LOG_FILE not found"

  exit 1

fi
```

### Search the log file for errors

```shell
ERRORS=$(grep -i "error" "$LOG_FILE")
```

### Check if there are any errors

```shell
if [ -n "$ERRORS" ]; then

  echo "ERRORS FOUND:"

  echo "$ERRORS"

else

  echo "No errors found in log file"

fi
```

### Review the configuration settings of the MySQL instance to ensure that they are not causing the restart.

```shell


#!/bin/bash



# Set the path to the MySQL configuration file

MYSQL_CONF=${PATH_TO_MYSQL_CONFIG_FILE}



# Check if the configuration file exists

if [ ! -f $MYSQL_CONF ]; then

    echo "Error: MySQL configuration file not found at $MYSQL_CONF"

    exit 1

fi



# Check the MySQL configuration for any settings that may cause the restart

if grep -q "max_connections" $MYSQL_CONF; then

    # If the max_connections setting is found, reduce it to prevent overloading the server

    sed -i 's/max_connections=${OLD_VALUE}/max_connections=${MAX_CONNECTIONS}/' $MYSQL_CONF

    echo "max_connections setting updated in MySQL configuration file."

fi



# Restart the MySQL instance to apply the new configuration

systemctl restart mysql



echo "MySQL instance has been restarted with updated configuration settings."


```


MySQL instance restart incident

Overview

Parameters

Debug

Check MySQL service status

Check MySQL logs for any errors

Check the uptime of the server to see if there were any recent reboots

Check if there are any resource constraints that could have caused the restart

Check the MySQL configuration file for any issues

Check the system logs for any other errors or warnings

Check the disk usage to see if there are any disk space issues

Check the network connectivity to the server

Check if there are any other services running on the same machine that could have caused the restart

Repair

Set the log file path

Check if the log file exists

Search the log file for errors

Check if there are any errors

Review the configuration settings of the MySQL instance to ensure that they are not causing the restart.

Learn more

Related Runbooks

Tomcat High JDBC Connection Pool Usage Incident.

Redis too many masters incident

Unusual drop in SELECT query rate on MySQL server

MySQL too many connections incident

Support