Kubernetes object deletions aren't as straightforward as they seem on the surface. Deleting an object is an involved process that includes conditional checks to determine whether safe removal is possible. This is achieved by API objects called Finalizers.

In this article, we'll look at what Finalizers are, how they're managed, and challenges they can cause when you want to delete an object. Having a better understanding of the deletion process can help you debug problems where resources don't seem to terminate in a timely manner.

What Are Finalizers?

Finalizers are a mechanism for enforcing certain conditions be met before an object can be deleted. When you run a command like

        kubectl delete namespace/example
    

, Kubernetes checks the Finalizers defined on the referenced object. These are listed in its metadata.finalizers field. Each Finalizer gets a chance to postpone the deletion until it's completed its actions.

The actual deletion process ends up looking like this:

1. Issue a deletion command. - Kubernetes marks the object as pending deletion. This leaves the resource in the read-only "Terminating" state.
2. Run each of the actions associated with the object's Finalizers. - Each time a Finalizer action completes, that Finalizer is detached from the object, so it'll no longer appear in the metadata.finalizers field.
3. Kubernetes keeps monitoring the Finalizers attached to the object. - The object will be deleted once the metadata.finalizers field is empty, because all Finalizers were removed by the completion of their actions.

Finalizers are commonly used to run clean-up and garbage collection procedures before an object is removed from the cluster. You can add your own Finalizers using the Kubernetes API; built-in Finalizers are also applied automatically to some types of object.

As an example, PersistentVolume resources come with a

        kubernetes.io/pv-protection
    

Finalizer that prevents accidental deletion of volumes in active use by Pods. The Finalizer enforces that the PersistentVolume cannot be removed from the cluster until there's no Pods using it. Issuing a deletion command while there's still an active Pod will cause the volume to be marked as Terminating; it will stay in this state for as long as the Pod needs the volume, then get deleted automatically as soon as possible afterwards.

Finalizer Challenges

Long-running Finalizers that wait for a condition involving other resources can cause deletions to appear stuck in the Terminating state. You may also run into issues where a Finalizer blocks the deletion of dependent objects which prevents the parent from successfully terminating.

These problems regularly cause confusion - developers and operators tend to see deletions as simple procedures when the process is actually nuanced and variable. The prerequisites for successful deletion depend on the resource's relations and their Finalizers, as well as the target object itself.

When an object's been Terminating for an excessive time, check its Finalizers by inspecting the metadata.finalizers field in its YAML:

kubectl get pod example-pod --namespace example -o json | jq

Once you know which finalizers are defined, you can begin identifying the ones likely to block a deletion. Viewing the object's events and condition changes can aid debugging by showing actions that have occurred since the deletion command was issued. Conditions are shown in the YAML's spec.status.conditions field; events are visible when running kubectl describe pod example-pod.

You can manually remove an object's Finalizers by patching the spec.finalizers field to null. This technique shouldn't be used unless absolutely necessary. Finalizers are safeguards meant to protect your cluster; overriding them could lead to orphaned objects and broken dependency chains.

kubectl patch pod example-pod -p '{"metadata: {"finalizers": null}}'

Owners and Propagation Policies

A related topic is object owners and deletion propagation policies. Owner references define the relationships between objects. They're used to remove entire object trees when a parent is deleted. As an example, if you delete a Deployment, Kubernetes must also destroy the Pods within that Deployment.

Owner references are defined via the metadata.ownerReferences field on objects. Each reference includes the kind and name of the object to parent the current resource to.

When owner references are used, deleting a parent automatically removes all its children. This is called cascading deletion. It's possible to disable the cascade by adding the --cascade=orphan flag to kubectl delete. Kubernetes will allow the object's children to remain in the cluster, leaving them available but orphaned.

Kubernetes also supports different deletion "propagation policies." These define whether the parent or its children are deleted first. The default Foreground policy deletes the children and then the parent, ensuring no orphaning occurs. Background inverts the order so the parent is removed first. The third policy, Orphan, instructs Kubernetes to ignore owner references altogether.

The kubectl delete command doesn't support propagation policies. You must make a direct API request if you want to change the policy for a deletion operation:

curl -X DELETE 
    
    
             localhost/api/v1/namespaces/default/deployments/example 
    
    
             -d '{"apiVersion": "v1", "kind": "DeleteOptions", "propagationPolicy": "Background"}'
    
    
             -H "Content-Type: application/json"

Finalizers are respected when a deletion is propagated or cascaded to related objects. In the case of the Foreground policy, this means all the Finalizers on all the children will need to complete before the parent can terminate. For the Background policy, children will remain live until their parent's Finalizers have finished.

Implementing Finalizers

You can implement your own Finalizers using the Kubernetes API and Go SDK. Finalizers are created by registering hooks in the Reconcile method of a controller.

The method should check whether the object to reconcile has a value in its DeletionTimestamp field. This means it's pending deletion and is in the Terminating state. Choose an identifier for your finalizer and check whether the object includes the value in its metadata.finalizers field. If it does, you should run any necessary actions and then detach the Finalizer from the object. An example implementation is included in the Kubebuilder guide to writing your own Kubernetes object types using CRDs (custom resource definitions).

Finalizers are always implemented as code in a controller method. The metadata.finalizers field acts in a similar capacity to annotations and labels, listing the Finalizers to apply to that object without directly defining the code to execute.

Conclusion

Finalizers control the lifecycle of a Kubernetes object after deletion is initiated. They're used to implement garbage collection, notify controllers of impending removals, and prevent accidental deletion of objects that are still being referenced by other resources.

Because Finalizers can block object deletions for arbitrarily long time periods, they're a common source of frustration when ops teams don't understand why an object's "stuck" terminating. In this situation it's best to inspect the affected resources, see which Finalizers are active, and investigate inter-object relationships which might be acting as blocking dependencies. Force removing a Finalizer should be your last resort if you must immediately delete a Terminating object or you've exhausted all your other options.