---
id: 08ed846e-6d98-4f99-a30f-1edf011a1cd0
---

# High number of blocked jobs in Jenkins queue.
---

This incident type indicates that there is a high number of blocked jobs in the Jenkins queue. This can cause delays in job execution and can be indicative of a larger issue within the system. It is important to investigate the root cause of the blocked jobs and take appropriate action to prevent this from happening in the future.

### Parameters
```shell
# Environment Variables

export JENKINS_NAMESPACE="PLACEHOLDER"

export JENKINS_POD_NAME="PLACEHOLDER"

export JENKINS_SERVICE_NAME="PLACEHOLDER"

export JENKINS_DEPLOYMENT_NAME="PLACEHOLDER"

export JENKINS_CONTAINER_NAME="PLACEHOLDER"

export DESIRED_NUMBER_OF_EXECUTORS="PLACEHOLDER"

export MINIMUM_DISK_SPACE_GB="PLACEHOLDER"
```

## Debug

### Get the list of pods running in the Jenkins namespace
```shell
kubectl get pods -n ${JENKINS_NAMESPACE}
```

### Check the logs of the Jenkins pod
```shell
kubectl logs ${JENKINS_POD_NAME} -n ${JENKINS_NAMESPACE}
```

### Check the resource usage of the Jenkins pod
```shell
kubectl top pod ${JENKINS_POD_NAME} -n ${JENKINS_NAMESPACE}
```

### Check the status of the Jenkins service
```shell
kubectl get svc ${JENKINS_SERVICE_NAME} -n ${JENKINS_NAMESPACE}
```

### Check the status of the Jenkins deployment
```shell
kubectl get deployment ${JENKINS_DEPLOYMENT_NAME} -n ${JENKINS_NAMESPACE}
```

### Check the Kubernetes events related to the Jenkins deployment
```shell
kubectl get events --sort-by=.metadata.creationTimestamp -n ${JENKINS_NAMESPACE} | grep ${JENKINS_DEPLOYMENT_NAME}
```

### Check the Jenkins queue status
```shell
kubectl exec -it ${JENKINS_POD_NAME} -n ${JENKINS_NAMESPACE} -- sh -c "java -jar /var/jenkins_home/war/WEB-INF/jenkins-cli.jar -s http://localhost:8080/ list-queue-items"
```

### Check the Jenkins build executor status
```shell
kubectl exec -it ${JENKINS_POD_NAME} -n ${JENKINS_NAMESPACE} -- sh -c "java -jar /var/jenkins_home/war/WEB-INF/jenkins-cli.jar -s http://localhost:8080/ list-builds"
```

### Check the Jenkins system logs
```shell
kubectl exec -it ${JENKINS_POD_NAME} -n ${JENKINS_NAMESPACE} -- sh -c "tail -f /var/log/jenkins/jenkins.log"
```

### The Jenkins server may be overloaded with too many queued jobs, causing some of them to be blocked and unable to execute.
```shell


#!/bin/bash



# Set the namespace, pod name, and container name for Jenkins

NAMESPACE=${JENKINS_NAMESPACE}

POD_NAME=${JENKINS_POD_NAME}

CONTAINER_NAME=${JENKINS_CONTAINER_NAME}



# Get the number of queued jobs in Jenkins

QUEUED_JOBS=$(kubectl exec -n $NAMESPACE $POD_NAME -c $CONTAINER_NAME -- jenkins-cli.sh -s http://localhost:8080/ list-queue-items | wc -l)



# Check if the number of queued jobs is above a certain threshold

if [ $QUEUED_JOBS -gt 100 ]; then

  echo "There are $QUEUED_JOBS queued jobs in Jenkins, which is above the threshold of 100. Jenkins may be overloaded."

  # Get the system load average for the past 1 minute

  LOAD_AVG=$(kubectl exec -n $NAMESPACE $POD_NAME -c $CONTAINER_NAME -- uptime | awk -F '[a-z]:' '{ print $2 }' | awk -F ',' '{ print $1 }')

  echo "The system load average for the past 1 minute is $LOAD_AVG."

  # Check if the system load average is above a certain threshold

  if (( $(echo "$LOAD_AVG > 1.0" | bc -l) )); then

    echo "The system load average is above the threshold of 1.0. This may be contributing to the Jenkins overload."

  fi

else

  echo "There are $QUEUED_JOBS queued jobs in Jenkins, which is below the threshold of 100. Jenkins is not overloaded."

fi


```

### There could be an issue with the configuration of the Jenkins server or job settings that is causing certain jobs to block others in the queue.
```shell


#!/bin/bash



# Get the list of pods running in the Jenkins namespace

PODS=$(kubectl get pods -n ${JENKINS_NAMESPACE} -o jsonpath='{.items[*].metadata.name}')



# Loop through all the pods and check their logs for any errors or warnings

for POD in $PODS; do

    echo "Checking logs for pod $POD ..."

    LOGS=$(kubectl logs $POD -n ${JENKINS_NAMESPACE})

    if [[ $LOGS == *"ERROR"* || $LOGS == *"WARNING"* ]]; then

        echo "Found error or warning in logs for pod $POD:"

        echo "$LOGS"

    else

        echo "No errors or warnings found in logs for pod $POD"

    fi

done



# Check the configuration of the Jenkins jobs

JOBS=$(kubectl get jobs -n ${JENKINS_NAMESPACE} -o jsonpath='{.items[*].metadata.name}')

for JOB in $JOBS; do

    echo "Checking configuration for job $JOB ..."

    CONFIG=$(kubectl get job $JOB -n ${JENKINS_NAMESPACE} -o yaml)

    if [[ $CONFIG == *"spec:"* ]]; then

        echo "Configuration for job $JOB looks good:"

        echo "$CONFIG"

    else

        echo "Problem with configuration for job $JOB:"

        echo "$CONFIG"

    fi

done


```

## Repair

### Increase the number of Jenkins executors to alleviate the job backlog and prevent jobs from being blocked in the queue.
```shell


#!/bin/bash



# Set variables

JENKINS_DEPLOYMENT=${JENKINS_DEPLOYMENT_NAME}

NUM_EXECUTORS=${DESIRED_NUMBER_OF_EXECUTORS}



# Scale up Jenkins deployment

kubectl scale deployment ${JENKINS_DEPLOYMENT} --replicas=${NUM_EXECUTORS}


```

### Identify and resolve any issues with the Jenkins infrastructure, such as network or disk space problems, that may be causing the queue to become blocked.
```shell


#!/bin/bash



# Set the namespace where Jenkins is deployed

NAMESPACE="${JENKINS_NAMESPACE}"



# Set the name of the Jenkins deployment

DEPLOYMENT_NAME="${JENKINS_DEPLOYMENT_NAME}"



# Set the minimum disk space threshold in gigabytes

MIN_DISK_SPACE_GB="${MINIMUM_DISK_SPACE_GB}"



# Check the disk space of the Jenkins deployment

disk_space=$(kubectl exec -it -n $NAMESPACE $DEPLOYMENT_NAME df -h | awk '/^\/dev\/sda1/ {print $4}')



# Convert the disk space to gigabytes

disk_space_gb=$(echo $disk_space | sed 's/G//')



# Check if the disk space is below the minimum threshold

if [ $(echo "$disk_space_gb < $MIN_DISK_SPACE_GB" | bc -l) -eq 1 ]; then

    # If the disk space is below the minimum threshold, scale up the Jenkins deployment to a larger size

    kubectl scale deployment -n $NAMESPACE $DEPLOYMENT_NAME --replicas=2

    echo "Jenkins deployment scaled up to 2 replicas to resolve disk space issue."

else

    echo "Jenkins deployment has sufficient disk space."

fi


```

This incident type indicates that there is a high number of blocked jobs in the Jenkins queue. This can cause delays in job execution and can be indicative of a larger issue within the system. It is important to investigate the root cause of the blocked jobs and take appropriate action to prevent this from happening in the future.


In software engineering, a large number of suspended threads in Tomcat is an incident type that occurs when a high number of threads are created but are blocked from executing. This can happen due to a variety of reasons, such as slow database queries, network latency, or resource contention. When this occurs, the system may become unresponsive or slow, causing a degradation in performance or even a complete system failure. It is important to investigate and resolve this issue as quickly as possible to ensure the system can continue to function properly.


Tomcat Large Number of Suspended Threads Incident.

This incident type refers to a situation where the Tomcat server's JDBC connection pool usage surpasses its configured limits. This can lead to a shortage of available connections for incoming requests, resulting in slow response times or even server crashes. It is important to monitor and manage the JDBC connection pool usage to prevent such incidents from occurring.


Tomcat High JDBC Connection Pool Usage Incident.

This incident type refers to a situation where Spark jobs are failing due to resource contentions in the cluster. When multiple Spark jobs are trying to access the same resources or data at the same time, it can cause a bottleneck that leads to job failures. This can happen when the resources in the cluster are not properly allocated or when the number of jobs running simultaneously exceeds the cluster's capacity to handle them. The result is that Spark jobs fail, leading to disruptions in data processing and analysis.


Spark job failures due to cluster resource contentions.

This incident type refers to a failure in one or more Spark executors during the execution of a job. Spark executors are worker processes that run computations and store data in memory or on disk. When an executor fails, it can cause the entire job to fail or result in degraded performance. This type of incident can occur for a variety of reasons, such as hardware or network issues, memory errors, or software bugs.


Spark executor failure during job execution.

This incident type indicates that there is a high latency issue in the execution of a Spark job. Spark is a distributed computing framework that is used for processing large datasets. High latency in this context means that the time taken to execute the Spark job is significantly longer than expected or normal. This can result in delays in processing data and can impact the performance of the application or system that is utilizing Spark.


High Latency Incident for Spark Job Execution.

```shell
# Environment Variables

export JENKINS_NAMESPACE="PLACEHOLDER"

export JENKINS_POD_NAME="PLACEHOLDER"

export JENKINS_SERVICE_NAME="PLACEHOLDER"

export JENKINS_DEPLOYMENT_NAME="PLACEHOLDER"

export JENKINS_CONTAINER_NAME="PLACEHOLDER"

export DESIRED_NUMBER_OF_EXECUTORS="PLACEHOLDER"

export MINIMUM_DISK_SPACE_GB="PLACEHOLDER"
```


### Get the list of pods running in the Jenkins namespace

```shell
kubectl get pods -n ${JENKINS_NAMESPACE}
```

### Check the logs of the Jenkins pod

```shell
kubectl logs ${JENKINS_POD_NAME} -n ${JENKINS_NAMESPACE}
```

### Check the resource usage of the Jenkins pod

```shell
kubectl top pod ${JENKINS_POD_NAME} -n ${JENKINS_NAMESPACE}
```

### Check the status of the Jenkins service

```shell
kubectl get svc ${JENKINS_SERVICE_NAME} -n ${JENKINS_NAMESPACE}
```

### Check the status of the Jenkins deployment

```shell
kubectl get deployment ${JENKINS_DEPLOYMENT_NAME} -n ${JENKINS_NAMESPACE}
```

### Check the Kubernetes events related to the Jenkins deployment

```shell
kubectl get events --sort-by=.metadata.creationTimestamp -n ${JENKINS_NAMESPACE} | grep ${JENKINS_DEPLOYMENT_NAME}
```

### Check the Jenkins queue status

```shell
kubectl exec -it ${JENKINS_POD_NAME} -n ${JENKINS_NAMESPACE} -- sh -c "java -jar /var/jenkins_home/war/WEB-INF/jenkins-cli.jar -s http://localhost:8080/ list-queue-items"
```

### Check the Jenkins build executor status

```shell
kubectl exec -it ${JENKINS_POD_NAME} -n ${JENKINS_NAMESPACE} -- sh -c "java -jar /var/jenkins_home/war/WEB-INF/jenkins-cli.jar -s http://localhost:8080/ list-builds"
```

### Check the Jenkins system logs

```shell
kubectl exec -it ${JENKINS_POD_NAME} -n ${JENKINS_NAMESPACE} -- sh -c "tail -f /var/log/jenkins/jenkins.log"
```

### The Jenkins server may be overloaded with too many queued jobs, causing some of them to be blocked and unable to execute.

```shell


#!/bin/bash



# Set the namespace, pod name, and container name for Jenkins

NAMESPACE=${JENKINS_NAMESPACE}

POD_NAME=${JENKINS_POD_NAME}

CONTAINER_NAME=${JENKINS_CONTAINER_NAME}



# Get the number of queued jobs in Jenkins

QUEUED_JOBS=$(kubectl exec -n $NAMESPACE $POD_NAME -c $CONTAINER_NAME -- jenkins-cli.sh -s http://localhost:8080/ list-queue-items | wc -l)



# Check if the number of queued jobs is above a certain threshold

if [ $QUEUED_JOBS -gt 100 ]; then

  echo "There are $QUEUED_JOBS queued jobs in Jenkins, which is above the threshold of 100. Jenkins may be overloaded."

  # Get the system load average for the past 1 minute

  LOAD_AVG=$(kubectl exec -n $NAMESPACE $POD_NAME -c $CONTAINER_NAME -- uptime | awk -F '[a-z]:' '{ print $2 }' | awk -F ',' '{ print $1 }')

  echo "The system load average for the past 1 minute is $LOAD_AVG."

  # Check if the system load average is above a certain threshold

  if (( $(echo "$LOAD_AVG > 1.0" | bc -l) )); then

    echo "The system load average is above the threshold of 1.0. This may be contributing to the Jenkins overload."

  fi

else

  echo "There are $QUEUED_JOBS queued jobs in Jenkins, which is below the threshold of 100. Jenkins is not overloaded."

fi


```

### There could be an issue with the configuration of the Jenkins server or job settings that is causing certain jobs to block others in the queue.

```shell


#!/bin/bash



# Get the list of pods running in the Jenkins namespace

PODS=$(kubectl get pods -n ${JENKINS_NAMESPACE} -o jsonpath='{.items[*].metadata.name}')



# Loop through all the pods and check their logs for any errors or warnings

for POD in $PODS; do

    echo "Checking logs for pod $POD ..."

    LOGS=$(kubectl logs $POD -n ${JENKINS_NAMESPACE})

    if [[ $LOGS == *"ERROR"* || $LOGS == *"WARNING"* ]]; then

        echo "Found error or warning in logs for pod $POD:"

        echo "$LOGS"

    else

        echo "No errors or warnings found in logs for pod $POD"

    fi

done



# Check the configuration of the Jenkins jobs

JOBS=$(kubectl get jobs -n ${JENKINS_NAMESPACE} -o jsonpath='{.items[*].metadata.name}')

for JOB in $JOBS; do

    echo "Checking configuration for job $JOB ..."

    CONFIG=$(kubectl get job $JOB -n ${JENKINS_NAMESPACE} -o yaml)

    if [[ $CONFIG == *"spec:"* ]]; then

        echo "Configuration for job $JOB looks good:"

        echo "$CONFIG"

    else

        echo "Problem with configuration for job $JOB:"

        echo "$CONFIG"

    fi

done


```


### Increase the number of Jenkins executors to alleviate the job backlog and prevent jobs from being blocked in the queue.

```shell


#!/bin/bash



# Set variables

JENKINS_DEPLOYMENT=${JENKINS_DEPLOYMENT_NAME}

NUM_EXECUTORS=${DESIRED_NUMBER_OF_EXECUTORS}



# Scale up Jenkins deployment

kubectl scale deployment ${JENKINS_DEPLOYMENT} --replicas=${NUM_EXECUTORS}


```

### Identify and resolve any issues with the Jenkins infrastructure, such as network or disk space problems, that may be causing the queue to become blocked.

```shell


#!/bin/bash



# Set the namespace where Jenkins is deployed

NAMESPACE="${JENKINS_NAMESPACE}"



# Set the name of the Jenkins deployment

DEPLOYMENT_NAME="${JENKINS_DEPLOYMENT_NAME}"



# Set the minimum disk space threshold in gigabytes

MIN_DISK_SPACE_GB="${MINIMUM_DISK_SPACE_GB}"



# Check the disk space of the Jenkins deployment

disk_space=$(kubectl exec -it -n $NAMESPACE $DEPLOYMENT_NAME df -h | awk '/^\/dev\/sda1/ {print $4}')



# Convert the disk space to gigabytes

disk_space_gb=$(echo $disk_space | sed 's/G//')



# Check if the disk space is below the minimum threshold

if [ $(echo "$disk_space_gb < $MIN_DISK_SPACE_GB" | bc -l) -eq 1 ]; then

    # If the disk space is below the minimum threshold, scale up the Jenkins deployment to a larger size

    kubectl scale deployment -n $NAMESPACE $DEPLOYMENT_NAME --replicas=2

    echo "Jenkins deployment scaled up to 2 replicas to resolve disk space issue."

else

    echo "Jenkins deployment has sufficient disk space."

fi


```


High number of blocked jobs in Jenkins queue.

Overview

Parameters

Debug

Get the list of pods running in the Jenkins namespace

Check the logs of the Jenkins pod

Check the resource usage of the Jenkins pod

Check the status of the Jenkins service

Check the status of the Jenkins deployment

Check the Jenkins queue status

Check the Jenkins build executor status

Check the Jenkins system logs

The Jenkins server may be overloaded with too many queued jobs, causing some of them to be blocked and unable to execute.

There could be an issue with the configuration of the Jenkins server or job settings that is causing certain jobs to block others in the queue.

Repair

Increase the number of Jenkins executors to alleviate the job backlog and prevent jobs from being blocked in the queue.

Identify and resolve any issues with the Jenkins infrastructure, such as network or disk space problems, that may be causing the queue to become blocked.

Learn more

Related Runbooks

Tomcat Large Number of Suspended Threads Incident.

Tomcat High JDBC Connection Pool Usage Incident.

Spark job failures due to cluster resource contentions.

Spark executor failure during job execution.

Support