Garbage collection is the process by which Kubernetes removes dependent objects after you delete a resource. It also handles automatic clean-up of redundant items in your cluster. Finished containers and disused images are periodically removed to prevent large numbers of unnecessary resources accumulating over time.
Several garbage collection strategies can be deployed depending on the type of object that’s being deleted. Garbage collection of unused resources is configurable by Kubelet settings on your cluster’s worker nodes. In this article, we’ll explain the different forms of garbage collection and how you can customize clean-up settings.
Basic Garbage Collection
Kubernetes garbage collection handles the removal of dependent objects, old containers, and unused images. Kubelet runs cleaning operations periodically while its service is active. It will delete stopped containers, such as those from finished Jobs, and remove unused images once a certain disk utilization threshold is reached.
Garbage collection of cluster resources in response to object deletions uses a cascading mechanism. This ensures an object’s children are removed alongside it, preventing orphans from existing. You can control this process by requesting a specific cascade order when you initiate deletion.
Parent-child relationships are expressed by Kubernetes owner references. Owners are responsible for the objects nested within them. This creates a dependency graph which informs the garbage collection process.
Kubernetes uses owner reference data to establish which resources should be removed before a target is deleted. As an example, Pods that are part of a ReplicaSet will have an owner reference that defines this link. Deleting the ReplicaSet will automatically remove the Pods within it too.
Owner references are set on objects in their
metadata.ownerReference manifest field. You shouldn’t normally need to inspect or manually change this field as owner references are automatically managed by Kubernetes controllers.
Children can block their parent’s deletion by setting the
metadata.blockOwnerDeletion field to the boolean value
true. This will prevent parents in the object’s
ownerReference chain from being deleted until the object has itself been removed.
Cascading Deletion Mechanisms
The owner reference chain means there’s two ways in which object deletions can proceed:
- Background deletion: The target object is deleted first, then its dependents.
- Foreground deletion: The dependents are deleted first, then the target object.
Consider a deletion command such as the following:
$ kubectl delete deployment/demo-app
The background deletion method will immediately delete the
demo-app Deployment object. This will leave its Pods running in your cluster. They’ll be cleaned up automatically in the background as part of the garbage collection process.
Foreground deletion begins by marking the
demo-app Deployment as a “deletion in progress.” It then deletes all the dependent objects that exist within it. The
demo-app Deployment will remain visible in your cluster until its Pods have been removed. The Deployment will be cleaned up once the dependents are gone.
Background cascading is the default mechanism. It offers an immediate result that removes the target object from your cluster without delay. Foreground deletion can be more desirable in some scenarios though, such as when you want a Deployment or ReplicaSet to be deleted only after its Pods have been destroyed. This avoids the brief period where the Pods are effectively orphaned, with their owner ReplicaSet already gone.
You can opt-in to foreground deletion by passing the
--cascade=foreground flag to the
kubectl delete command:
$ kubectl delete deployment/demo-app --cascade=foreground
The Pods will be deleted, then the
Disabling Garbage Collection of Dependents
There is a third cascade option: ignore dependent garbage collection altogether. This will orphan any dependents of the object, leaving them in your cluster but with their owner reference removed.
$ kubectl delete deployment/demo-app --cascade=orphan
This command immediately deletes the
demo-app Deployment but leaves its Pods intact. They’ll keep running in your cluster until deleted separately, as part of another command. Although this behavior is rarely desirable, it can be helpful if you decide Pods no longer need to be in a Deployment or ReplicaSet.
Garbage Collection and Finalizers
Finalizers also affect how garbage collection operates. They prevent deletions from occurring until certain conditions are met.
Many apparently “stuck” deletions are caused by pending finalizers. A finalizer that’s not marked itself as finished will prevent the object being removed from your cluster. Once the finalizer has cleared, the Kubernetes control plane will proceed with the deletion.
Some finalizers are included with Kubernetes, such as the
pv-protection safeguard against deleting actively used volumes. Others may be added by third-party controllers you add to your cluster. Overriding a finalizer isn’t recommended as it could create an incorrect state that leads to unexpected behavior.
Customizing Garbage Collection of Unused Resources
Garbage collection of redundant containers and images occurs automatically on your worker nodes. Five parameters are available to tune the deletion thresholds:
maximum-dead-containers– The maximum number of old containers that can continue to exist after garbage collection is run. The default of
-1removes the limit.
maximum-dead-containers-per-container– Set the maximum number of retained old containers on a per-container basis. This controls how many previous versions of a container stay in the cluster, after a new replacement is started.
minimum-container-ttl-duration– Containers become eligible for garbage collection this many minutes after they stop. The default value of
0disables the delay.
image-gc-high-threshold– When disk usage reaches this percentage Kubelet will try to cull unused images…
image-gc-low-threshold– …to bring the usage down to this level.
These parameters are set as Kubelet launch flags. Flags are typically placed into
KUBELET_KUBEADM_ARGS="--image-gc-high-threshold=60 --image-gc-low-threshold=50 --minimum-container-ttl-duration=5"
This configuration will permit automatic deletion of containers that have been redundant for at least five minutes. Each garbage collection pass will also try to prune old images once 60% disk utilization is reached, aiming to reduce consumption down to 50%.
You should restart the Kubelet process after you make these changes:
$ sudo systemctl daemon-reload $ sudo systemctl restart kubelet
As the settings are part of Kubelet, you’ll need to apply them to each of your worker nodes in your cluster if you want configuration to be consistent across them all.
Understanding the different forms of Kubernetes garbage collection helps you decide on the correct strategy when deleting objects from your cluster. Background and foreground cascades affect the order in which dependent resources are cleaned up. Relationships between objects are identified by their owner reference chains. Kubernetes also cleans up unused containers and images in the background. The thresholds for this behavior can be customized using Kubelet settings.
For best results you should stick to the mechanisms that Kubernetes provides. The project warns against using third-party garbage collection tools as these can create state inconsistencies that cause problems in your cluster.