Problem
In Kubernetes environments, container images often need to be rebuilt when new dependencies are required. A common scenario is adding Python libraries to worker or scheduler containers.
In some cases, pushing the image immediately to a registry is not ideal due to testing requirements, restricted access, or the need for fast validation inside the cluster.
Diagnosis
Step 1: Confirm image changes are already applied
At this stage, the container image has already been rebuilt and includes all required changes (for example, new Python libraries). Once validated locally, the next step is to extract the image for distribution across the Kubernetes nodes.
Solution
Step 1: Extract the rebuilt image
Save the rebuilt image into a TAR file using versioning instead of date-based tags:
docker save example/airflow:worker-v1 -o airflow-worker-v1.tar
Verify the file size:
ls -lh airflow-worker-v1.tar
Step 2: Identify Kubernetes nodes
kubectl get nodes -o wide
Step 3: Copy the image to each node
scp airflow-worker-v1.tar user@node1-ip:/tmp/
scp airflow-worker-v1.tar user@node2-ip:/tmp/
scp airflow-worker-v1.tar user@node3-ip:/tmp/
Step 4: Load the image on each node
If Docker is used as the container runtime:
ssh user@node1-ip
docker load -i /tmp/airflow-worker-v1.tar
rm /tmp/airflow-worker-v1.tar
exit
If containerd or CRI is used:
sudo ctr -n k8s.io images import /tmp/airflow-worker-v1.tar
Verify:
sudo crictl images | grep example
Step 5: Update Kubernetes workloads
kubectl set image statefulset/airflow-worker airflow-worker=example/airflow:worker-v1 -n airflow
Step 6: Configure imagePullPolicy
kubectl patch statefulset airflow-worker -n airflow -p '{"spec":{"template":{"spec":{"containers":[{"name":"airflow-worker","imagePullPolicy":"IfNotPresent"}]}}}}'
Step 7: Push the image to a registry (optional)
docker push example/airflow:worker-v1
docker push example/airflow:worker-latest
Results
- Kubernetes successfully used the local image
- No image pull from the registry was required
- New dependencies were validated in the cluster
- Workloads started correctly on all nodes
Lessons Learned
- Kubernetes can run workloads using local images
imagePullPolicy: IfNotPresentis essential- All nodes must have the image available
- Ideal for controlled testing before publishing images