---
id: e03fc2ac-31e4-4961-90fb-96987207a512
---

# Unresponsive MySQL service on pod with database lock.
---

This incident type refers to a situation where the MySQL service running on a pod becomes unresponsive and is unable to process new read and write requests. The incident could be caused by a database lock, which prevents other processes from accessing the database. The exact cause of the lock situation may need to be determined to resolve the issue.

### Parameters
```shell
export POD_NAME="PLACEHOLDER"

export MYSQL_CONTAINER_NAME="PLACEHOLDER"

export MYSQL_SERVICE_NAME="PLACEHOLDER"

export PROCESS_ID="PLACEHOLDER"

export MYSQL_USER="PLACEHOLDER"

export NAMESPACE="PLACEHOLDER"

export MYSQL_PASSWORD="PLACEHOLDER"
```

## Debug

### Next Step
```shell
shell

# List all pods in the current namespace

kubectl get pods
```

### Check the status of the pod in question
```shell
kubectl describe pod ${POD_NAME}
```

### Check the logs of the MySQL container in the pod
```shell
kubectl logs ${POD_NAME} ${MYSQL_CONTAINER_NAME}
```

### Check the status of the MySQL service
```shell
kubectl get svc ${MYSQL_SERVICE_NAME}
```

### Check if the MySQL database is locked
```shell
kubectl exec ${POD_NAME} -c ${MYSQL_CONTAINER_NAME} -- mysql -u ${MYSQL_USER} -p${MYSQL_PASSWORD} -e "SHOW OPEN TABLES WHERE In_use > 0;"
```

### Inside the container, connect to the MySQL database
```shell
mysql -u ${MYSQL_USER} -p${MYSQL_PASSWORD}
```

### Inside the MySQL prompt, show the process list
```shell
SHOW PROCESSLIST;
```

### Inside the MySQL prompt, show the locked tables
```shell
SHOW OPEN TABLES WHERE In_use > 0;
```

## Repair

### Inside the MySQL prompt, kill a locked process
```shell
KILL ${PROCESS_ID};
```

### If the locks cannot be released, try restarting the MySQL service or the entire pod to clear all locks.
```shell
bash

#!/bin/bash



# Set variables

NAMESPACE=${NAMESPACE}

POD_NAME=${POD_NAME}

MYSQL_SERVICE=${MYSQL_SERVICE_NAME}



# Check for active locks

if kubectl exec -n $NAMESPACE $POD_NAME -- mysql -u ${MYSQL_USER} -p${MYSQL_PASSWORD} -e "SHOW OPEN TABLES WHERE In_use > 0" | grep -q "In_use"; then

  # Kill processes causing locks

  kubectl exec -n $NAMESPACE $POD_NAME -- mysql -u ${MYSQL_USER} -p${MYSQL_PASSWORD} -e "SHOW PROCESSLIST" | grep -v "Sleep" | awk '{print "KILL "$1";"}' | kubectl exec -n $NAMESPACE $POD_NAME -- mysql -u ${MYSQL_USER} -p${MYSQL_PASSWORD}



  # If locks persist, restart MySQL service or entire pod

  kubectl rollout restart deployment $MYSQL_SERVICE -n $NAMESPACE

  # OR

  kubectl delete pod $POD_NAME -n $NAMESPACE

fi


```

This incident type refers to a situation where the MySQL service running on a pod becomes unresponsive and is unable to process new read and write requests. The incident could be caused by a database lock, which prevents other processes from accessing the database. The exact cause of the lock situation may need to be determined to resolve the issue.


This incident type refers to a situation where the Tomcat server's JDBC connection pool usage surpasses its configured limits. This can lead to a shortage of available connections for incoming requests, resulting in slow response times or even server crashes. It is important to monitor and manage the JDBC connection pool usage to prevent such incidents from occurring.


Tomcat High JDBC Connection Pool Usage Incident.

This incident type refers to the occurrence of deadlocks in a PostgreSQL database. A deadlock happens when two or more transactions are waiting for each other to release locks on resources, resulting in a situation where none of the transactions can proceed. This can cause significant disruption to the database and the applications that rely on it, as it prevents any further progress until the deadlock is resolved.


Postgresql Deadlocks Incident

This incident type refers to a sudden and abnormal drop in the SELECT query rate on a MySQL server. This can potentially cause disruption in the service and affect the user experience. The cause of the drop could be due to a variety of factors such as database issues, server overload, or network problems. It requires immediate attention and investigation to identify the root cause and resolve the issue to prevent further impact.


Unusual drop in SELECT query rate on MySQL server

This incident type occurs when the number of concurrent connections to a MySQL database server exceeds the configured limit of 80%. This can lead to performance degradation or even database server crashes, making the application or system that relies on the database server unavailable for the end-users. This type of incident typically requires immediate attention and resolution to restore normal operation of the affected service.


MySQL too many connections incident

A MySQL slow queries incident is when a MySQL server is experiencing slow queries. This can result in degraded performance and responsiveness of the system. It is important to resolve this incident quickly to minimize impact on users.


MySQL slow queries incident

```shell
export POD_NAME="PLACEHOLDER"

export MYSQL_CONTAINER_NAME="PLACEHOLDER"

export MYSQL_SERVICE_NAME="PLACEHOLDER"

export PROCESS_ID="PLACEHOLDER"

export MYSQL_USER="PLACEHOLDER"

export NAMESPACE="PLACEHOLDER"

export MYSQL_PASSWORD="PLACEHOLDER"
```


### Next Step

```shell
shell

# List all pods in the current namespace

kubectl get pods
```

### Check the status of the pod in question

```shell
kubectl describe pod ${POD_NAME}
```

### Check the logs of the MySQL container in the pod

```shell
kubectl logs ${POD_NAME} ${MYSQL_CONTAINER_NAME}
```

### Check the status of the MySQL service

```shell
kubectl get svc ${MYSQL_SERVICE_NAME}
```

### Check if the MySQL database is locked

```shell
kubectl exec ${POD_NAME} -c ${MYSQL_CONTAINER_NAME} -- mysql -u ${MYSQL_USER} -p${MYSQL_PASSWORD} -e "SHOW OPEN TABLES WHERE In_use > 0;"
```

### Inside the container, connect to the MySQL database

```shell
mysql -u ${MYSQL_USER} -p${MYSQL_PASSWORD}
```

### Inside the MySQL prompt, show the process list

```shell
SHOW PROCESSLIST;
```

### Inside the MySQL prompt, show the locked tables

```shell
SHOW OPEN TABLES WHERE In_use > 0;
```


### Inside the MySQL prompt, kill a locked process

```shell
KILL ${PROCESS_ID};
```

### If the locks cannot be released, try restarting the MySQL service or the entire pod to clear all locks.

```shell
bash

#!/bin/bash



# Set variables

NAMESPACE=${NAMESPACE}

POD_NAME=${POD_NAME}

MYSQL_SERVICE=${MYSQL_SERVICE_NAME}



# Check for active locks

if kubectl exec -n $NAMESPACE $POD_NAME -- mysql -u ${MYSQL_USER} -p${MYSQL_PASSWORD} -e "SHOW OPEN TABLES WHERE In_use > 0" | grep -q "In_use"; then

  # Kill processes causing locks

  kubectl exec -n $NAMESPACE $POD_NAME -- mysql -u ${MYSQL_USER} -p${MYSQL_PASSWORD} -e "SHOW PROCESSLIST" | grep -v "Sleep" | awk '{print "KILL "$1";"}' | kubectl exec -n $NAMESPACE $POD_NAME -- mysql -u ${MYSQL_USER} -p${MYSQL_PASSWORD}



  # If locks persist, restart MySQL service or entire pod

  kubectl rollout restart deployment $MYSQL_SERVICE -n $NAMESPACE

  # OR

  kubectl delete pod $POD_NAME -n $NAMESPACE

fi


```


Unresponsive MySQL service on pod with database lock.

Overview

Parameters

Debug

Next Step

Check the status of the pod in question

Check the logs of the MySQL container in the pod

Check the status of the MySQL service

Check if the MySQL database is locked

Inside the container, connect to the MySQL database

Inside the MySQL prompt, show the process list

Inside the MySQL prompt, show the locked tables

Repair

Inside the MySQL prompt, kill a locked process

If the locks cannot be released, try restarting the MySQL service or the entire pod to clear all locks.

Learn more

Related Runbooks

Tomcat High JDBC Connection Pool Usage Incident.

Postgresql Deadlocks Incident

Unusual drop in SELECT query rate on MySQL server

MySQL too many connections incident

Support