---
id: b57f9179-ff98-4954-90bb-8ceb03d01c17
---

# Jenkins run failure total incident
---

This incident type refers to a situation where there has been a total failure in running Jenkins jobs. Jenkins is a popular automation server used for building, testing, and deploying applications. When such an incident occurs, it can lead to significant downtime and issues in the software development and delivery process. The incident may be triggered by factors such as misconfiguration, code errors, or infrastructure issues. Resolving the issue promptly is critical to minimize the impact on the software development process.

### Parameters
```shell
export JENKINS_PORT="PLACEHOLDER"

export JENKINS_INSTANCE="PLACEHOLDER"

export JOB_NAME="PLACEHOLDER"

export BUILD_NUMBER="PLACEHOLDER"

export JENKINS_SERVER_ADDRESS="PLACEHOLDER"
```

## Debug

### 1. Check the status of the Jenkins service
```shell
systemctl status jenkins
```

### 2. Check the logs for any errors or warnings
```shell
journalctl -u jenkins --no-pager --since "2 hours ago"
```

### 3. Verify the Jenkins configuration
```shell
sudo su - jenkins -c "java -jar /usr/share/jenkins/jenkins.war --check"
```

### 4. Check the connectivity to the Jenkins instance
```shell
nc -vz ${JENKINS_INSTANCE} ${JENKINS_PORT}
```

### 5. Check the Jenkins job configuration for any errors
```shell
sudo cat /var/lib/jenkins/jobs/${JOB_NAME}/config.xml
```

### 6. Check the Jenkins build history for any failed builds
```shell
sudo cat /var/lib/jenkins/jobs/${JOB_NAME}/builds/${BUILD_NUMBER}/log
```

### Infrastructure issues: When running Jenkins on an infrastructure that is not properly set up, there may be issues with resources such as memory, disk space, or network connectivity. This could result in failed jobs and ultimately a total failure of the system.
```shell


#!/bin/bash



# Check memory usage

memory=$(free -m | awk 'NR==2{printf "%.2f%%", $3*100/$2}')

if (( $(echo "$memory > 80.00" |bc -l) )); then

    echo "WARNING: Memory usage is high. Current usage: $memory"

else

    echo "Memory usage is normal. Current usage: $memory"

fi



# Check disk space

disk=$(df -h / | awk '{print $5}' | tail -n 1 | cut -d'%' -f1)

if (( "$disk" > 80 )); then

    echo "WARNING: Disk space is low. Current usage: $disk%"

else

    echo "Disk space usage is normal. Current usage: $disk%"

fi



# Check network connectivity

ping -c 4 ${JENKINS_SERVER_ADDRESS} > /dev/null 2>&1

if [ $? -eq 0 ]; then

    echo "Jenkins server is reachable"

else

    echo "ERROR: Jenkins server is not reachable"

fi



# Check Jenkins log files for errors

if grep -q "ERROR" /var/log/jenkins/jenkins.log; then

    echo "ERROR: Jenkins log files contain errors"

else

    echo "Jenkins log files do not contain errors"

fi


```

## Repair

### Restart the Jenkins server to see if the issue resolves itself.
```shell
bash

#!/bin/bash



# Restart the Jenkins service

sudo systemctl restart jenkins



# Sleep for 10 seconds to make sure Jenkins service has restarted successfully

sleep 10



# Verify that the service has started successfully

sudo systemctl status jenkins


```

This incident type refers to a situation where there has been a total failure in running Jenkins jobs. Jenkins is a popular automation server used for building, testing, and deploying applications. When such an incident occurs, it can lead to significant downtime and issues in the software development and delivery process. The incident may be triggered by factors such as misconfiguration, code errors, or infrastructure issues. Resolving the issue promptly is critical to minimize the impact on the software development process.


Tomcat Server Crash incident type refers to a situation where the Tomcat Server, an open-source web server and servlet container, becomes unresponsive and stops functioning. This can result in a complete or partial system outage, leading to a disruption in the delivery of web applications and services. The root cause of the crash can be attributed to a variety of factors such as misconfiguration, network or hardware issues, software bugs, or excessive server load. It is critical to address this incident quickly to minimize the impact on users and restore normal system functionality.


Tomcat Server Crash

This incident type refers to a situation where the Tomcat server's JDBC connection pool usage surpasses its configured limits. This can lead to a shortage of available connections for incoming requests, resulting in slow response times or even server crashes. It is important to monitor and manage the JDBC connection pool usage to prevent such incidents from occurring.


Tomcat High JDBC Connection Pool Usage Incident.

This incident type refers to an increase in the number of errors per second on a Tomcat server, which could indicate an issue with the server itself, the host, a deployed application, or an application servlet. This could include errors generated when the Tomcat server runs out of memory, can't find a requested file or servlet, or is unable to serve a JSP due to syntax errors in the servlet codebase. This incident type requires immediate attention to diagnose and address the underlying issue.


Increase of the errors/second rate for Tomcat server

This incident type involves the failure of the YARN ResourceManager, which can impact the performance of Spark jobs. The ResourceManager is responsible for managing resources in a Hadoop cluster, and when it fails, it can prevent Spark jobs from running properly. This incident requires investigation to determine the root cause of the failure, and recovery steps to restore the ResourceManager and prevent similar failures from occurring in the future.


YARN ResourceManager Failure Impacting Spark Jobs.

This incident type refers to a situation where Spark tasks are failing due to out of memory errors. Spark is a distributed computing system used for big data processing. When the data volume exceeds the allocated memory, the Spark tasks fail, and the system generates an out of memory error. This type of incident can cause data processing delays or even system downtime, which can impact the overall performance of the application.


Spark tasks failing due to out of memory errors.

```shell
export JENKINS_PORT="PLACEHOLDER"

export JENKINS_INSTANCE="PLACEHOLDER"

export JOB_NAME="PLACEHOLDER"

export BUILD_NUMBER="PLACEHOLDER"

export JENKINS_SERVER_ADDRESS="PLACEHOLDER"
```


### 1. Check the status of the Jenkins service

```shell
systemctl status jenkins
```

### 2. Check the logs for any errors or warnings

```shell
journalctl -u jenkins --no-pager --since "2 hours ago"
```

### 3. Verify the Jenkins configuration

```shell
sudo su - jenkins -c "java -jar /usr/share/jenkins/jenkins.war --check"
```

### 4. Check the connectivity to the Jenkins instance

```shell
nc -vz ${JENKINS_INSTANCE} ${JENKINS_PORT}
```

### 5. Check the Jenkins job configuration for any errors

```shell
sudo cat /var/lib/jenkins/jobs/${JOB_NAME}/config.xml
```

### 6. Check the Jenkins build history for any failed builds

```shell
sudo cat /var/lib/jenkins/jobs/${JOB_NAME}/builds/${BUILD_NUMBER}/log
```

### Infrastructure issues: When running Jenkins on an infrastructure that is not properly set up, there may be issues with resources such as memory, disk space, or network connectivity. This could result in failed jobs and ultimately a total failure of the system.

```shell


#!/bin/bash



# Check memory usage

memory=$(free -m | awk 'NR==2{printf "%.2f%%", $3*100/$2}')

if (( $(echo "$memory > 80.00" |bc -l) )); then

    echo "WARNING: Memory usage is high. Current usage: $memory"

else

    echo "Memory usage is normal. Current usage: $memory"

fi



# Check disk space

disk=$(df -h / | awk '{print $5}' | tail -n 1 | cut -d'%' -f1)

if (( "$disk" > 80 )); then

    echo "WARNING: Disk space is low. Current usage: $disk%"

else

    echo "Disk space usage is normal. Current usage: $disk%"

fi



# Check network connectivity

ping -c 4 ${JENKINS_SERVER_ADDRESS} > /dev/null 2>&1

if [ $? -eq 0 ]; then

    echo "Jenkins server is reachable"

else

    echo "ERROR: Jenkins server is not reachable"

fi



# Check Jenkins log files for errors

if grep -q "ERROR" /var/log/jenkins/jenkins.log; then

    echo "ERROR: Jenkins log files contain errors"

else

    echo "Jenkins log files do not contain errors"

fi


```


### Restart the Jenkins server to see if the issue resolves itself.

```shell
bash

#!/bin/bash



# Restart the Jenkins service

sudo systemctl restart jenkins



# Sleep for 10 seconds to make sure Jenkins service has restarted successfully

sleep 10



# Verify that the service has started successfully

sudo systemctl status jenkins


```


Jenkins run failure total incident

Overview

Parameters

Debug

1. Check the status of the Jenkins service

2. Check the logs for any errors or warnings

3. Verify the Jenkins configuration

4. Check the connectivity to the Jenkins instance

5. Check the Jenkins job configuration for any errors

6. Check the Jenkins build history for any failed builds

Infrastructure issues: When running Jenkins on an infrastructure that is not properly set up, there may be issues with resources such as memory, disk space, or network connectivity. This could result in failed jobs and ultimately a total failure of the system.

Repair

Restart the Jenkins server to see if the issue resolves itself.

Learn more

Related Runbooks

Tomcat Server Crash

Tomcat High JDBC Connection Pool Usage Incident.

Increase of the errors/second rate for Tomcat server

YARN ResourceManager Failure Impacting Spark Jobs.

Support