Runbook
Kubernetes - Unexpected Image Pulls Incident
Back to Runbooks
Overview
The Unexpected Image Pulls Incident refers to an alert triggered by unexpected pulls of container images, which can indicate a compromise in the supply chain. This type of incident can occur when a container image is downloaded from an untrusted or malicious source, or when a legitimate image has been tampered with and modified to include malicious code. Such incidents can pose a serious security risk, as they can allow attackers to gain unauthorized access to systems and steal sensitive data or compromise system integrity. Prompt detection and response to this type of incident is critical to prevent further compromise and protect the security of the system.
Parameters
1export NAMESPACE="PLACEHOLDER"2export POD_NAME="PLACEHOLDER"3export DEPLOYMENT="PLACEHOLDER"
Debug
List all pods in the cluster
kubectl get pods --all-namespaces
Check the logs of a specific pod
kubectl logs ${POD_NAME} -n ${NAMESPACE}
Check which image a pod is using
kubectl describe pod ${POD_NAME} -n ${NAMESPACE} | grep -i image
Check if there are any image pull errors in the events for a pod
kubectl describe pod ${POD_NAME} -n ${NAMESPACE} | grep -i error
Check if there are any image pull errors in the events for the entire namespace
kubectl describe namespace ${NAMESPACE} | grep -i error
Check if there are any image pull errors in the events for the entire cluster
kubectl get events --sort-by='.metadata.creationTimestamp' | grep -i error
Repair
Remove the compromised container images .
1bash2#!/bin/bash34# Define the namespace and deployment name5NAMESPACE=${NAMESPACE}6DEPLOYMENT=${DEPLOYMENT}78# Get the current image in use9CURRENT_IMAGE=$(kubectl get deployment $DEPLOYMENT -n $NAMESPACE -o=jsonpath='{.spec.template.spec.containers[0].image}')1011# Remove the current deployment12kubectl delete deployment $DEPLOYMENT -n $NAMESPACE1314# Remove the current image from the cluster15kubectl image prune -a --force --filter "until=24h" --filter "reference=$CURRENT_IMAGE"
Learn more
Related Runbooks
Check out these related runbooks to help you debug and resolve similar issues.