Runbook

Kubernetes - Unexpected Image Pulls Incident

Back to Runbooks

Overview

The Unexpected Image Pulls Incident refers to an alert triggered by unexpected pulls of container images, which can indicate a compromise in the supply chain. This type of incident can occur when a container image is downloaded from an untrusted or malicious source, or when a legitimate image has been tampered with and modified to include malicious code. Such incidents can pose a serious security risk, as they can allow attackers to gain unauthorized access to systems and steal sensitive data or compromise system integrity. Prompt detection and response to this type of incident is critical to prevent further compromise and protect the security of the system.

Parameters

1export NAMESPACE="PLACEHOLDER"
2export POD_NAME="PLACEHOLDER"
3export DEPLOYMENT="PLACEHOLDER"

Debug

List all pods in the cluster

kubectl get pods --all-namespaces

Check the logs of a specific pod

kubectl logs ${POD_NAME} -n ${NAMESPACE}

Check which image a pod is using

kubectl describe pod ${POD_NAME} -n ${NAMESPACE} | grep -i image

Check if there are any image pull errors in the events for a pod

kubectl describe pod ${POD_NAME} -n ${NAMESPACE} | grep -i error

Check if there are any image pull errors in the events for the entire namespace

kubectl describe namespace ${NAMESPACE} | grep -i error

Check if there are any image pull errors in the events for the entire cluster

kubectl get events --sort-by='.metadata.creationTimestamp' | grep -i error

Repair

Remove the compromised container images .

1bash
2#!/bin/bash
3
4# Define the namespace and deployment name
5NAMESPACE=${NAMESPACE}
6DEPLOYMENT=${DEPLOYMENT}
7
8# Get the current image in use
9CURRENT_IMAGE=$(kubectl get deployment $DEPLOYMENT -n $NAMESPACE -o=jsonpath='{.spec.template.spec.containers[0].image}')
10
11# Remove the current deployment
12kubectl delete deployment $DEPLOYMENT -n $NAMESPACE
13
14# Remove the current image from the cluster
15kubectl image prune -a --force --filter "until=24h" --filter "reference=$CURRENT_IMAGE"

Learn more

Related Runbooks

Check out these related runbooks to help you debug and resolve similar issues.