Custom Resource Definitions (CRDs) are Kubernetes API extensions which can define new object types. Pods, ReplicaSets, ConfigMaps, and Ingresses are examples of common built-in resources. CRDs let you add entirely new types to this list, then manage them using familiar Kubernetes tools such as Kubectl.

The CRD mechanism is intentionally abstract and can be used in myriad ways to store data and build new functionality. You'll find custom resources in many popular community tools: Cert-Manager defines objects that represent SSL Certificates and Issuers while Helm represents Charts as their own CRD.

What Makes a Resource?

Kubernetes resources define the types of data you can store in your cluster. They're accessed via the Kubernetes API which provides endpoints for creating, listing, and modifying items within each resource type.

You can add custom resources to store your own arbitrary data inside your cluster. Items you create will be stored by the etcd control plane component, alongside instances of built-in resources. Custom resources are automatically surfaced by the API so you don't need to set up your own tooling to create item instances.

CRDs act as simple data structures by default. Although CRDs in the wild often have their own behaviors, this isn't provided by the CRD mechanism. Custom Kubernetes controllers and operators can be used to implement functionality around custom resources. Without a controller, a CRD's items will always exist as static in-cluster data that you can interact with via CRUD endpoints provided by the Kubernetes API.

CRDs are dynamic components which can be created and removed at any time. Some object types that are included with Kubernetes are implemented as CRDs too, affording more modularity within the cluster's core.

Creating a CRD

CRDs are themselves a type of Kubernetes object. You create them in the same way as any other resource, by writing a YAML file and applying it to your cluster:

apiVersion: apiextensions.k8s.io/v1
    
    
            kind: CustomResourceDefinition
    
    
            metadata:
    
    
             name: custom-apps.crds.example.com
    
    
            spec:
    
    
             group: crds.example.com
    
    
             versions:
    
    
             - name: v1
    
    
             served: true
    
    
             storage: true
    
    
             schema:
    
    
             openAPIV3Schema:
    
    
             type: object
    
    
             properties:
    
    
             spec:
    
    
             type: object
    
    
             properties:
    
    
             app-name:
    
    
             type: string
    
    
             app-version:
    
    
             type: string
    
    
             release-count:
    
    
             type: integer
    
    
             scope: Namespaced
    
    
             names:
    
    
             plural: custom-apps
    
    
             singular: custom-app
    
    
             kind: CustomApp

Use Kubectl to add the CustomApp CRD to your cluster:

$ kubectl apply -f custom-apps-crd.yaml
    
    
            customresourcedefinition.apiextensions.k8s.io/custom-apps.crds.example.com created

The YAML file defines a CRD that could be used to store data about apps. CRDs need a metadata.name and spec.group field in a precise format: the group takes the form of a subdomain which the CRD belongs to. The same subdomain must be included in the CRD's metadata.name . The value of names.plural is prepended as a new subdomain component to build the final metadata.name.

The spec.names field defines how you'll refer to the CRD when using the Kubernetes API and Kubectl commands. In this example you'll be able to run kubectl get custom-apps and kubectl get custom-app/example-app to interact with objects that use the CRD. When you're creating a new object as a YAML file, you should set kind: CustomApp to make it an instance of the CRD.

The CRD is configured as a namespace-level object type by the scope field. You can use Cluster as an alternative value for this field to create objects that exist at the cluster-level, outside any namespace.

The data associated with your custom objects is defined in the spec.versions.schema.openAPIV3Schema field. Each listed "version" creates a new version of the CRD's API that you can reference with the apiVersion field in your resource YAML files. The CRD's data is configured using OpenAPI properties; here, each "custom app" in your cluster should have app-name, app-version, and release-count properties set in its YAML spec.

Using Your CRD

It can take a few minutes for a new CRD's API endpoints to be provisioned. Check the progress by retrieving the CRD's details with Kubectl:

$ kubectl describe crd custom-apps.crds.example.com
    
    
            ...
    
    
            Status:
    
    
             Accepted Names:
    
    
             Kind: CustomApp
    
    
             List Kind: CustomAppList
    
    
             Plural: custom-apps
    
    
             Singular: custom-app
    
    
             Conditions:
    
    
             Last Transition Time: 2022-04-04T13:29:24Z
    
    
             Message: no conflicts found
    
    
             Reason: NoConflicts
    
    
             Status: True
    
    
             Type: NamesAccepted
    
    
             Last Transition Time: 2022-04-04T13:29:24Z
    
    
             Message: the initial names have been accepted
    
    
             Reason: InitialNamesAccepted
    
    
             Status: True
    
    
             Type: Established
    
    
            ...

The CRD is ready to use when you see Type: Established near the end of the command's output. Kubernetes will be accepting requests to the CRD's API endpoint. In this case, the API base URL will be as follows:

/apis/custom-apps.crds.example.com/v1/namespaces/*/custom-apps/...

You can now use Kubectl to view objects that have been created with the CRD:

$ kubectl get custom-apps
    
    
            No resources found in default namespace.

Although no objects exist yet, Kubernetes now knows it has a resource type called custom-apps.

To create a "custom app" object, write a new YAML file with kind: CustomApp. The apiVersion must be set to the group name and API version provided by the CRD. Within the spec section, include the properties you defined in the CRD's schema.

apiVersion: crds.example.com/v1
    
    
            kind: CustomApp
    
    
            metadata:
    
    
             name: demo-app-1
    
    
            spec:
    
    
             app-name: demo-app
    
    
             app-version: 1.1.0
    
    
             release-count: 5

Use Kubectl to add the object to your cluster:

$ kubectl apply -f custom-app.yaml
    
    
            customapp.crds.example.com/demo-app created

Now you can retrieve the object's details using familiar Kubectl commands:

$ kubectl describe custom-app/demo-app-1
    
    
            Name: demo-app
    
    
            Namespace: default
    
    
            Labels: <none>
    
    
            Annotations: <none>
    
    
            API Version: crds.example.com/v1
    
    
            Kind: CustomApp
    
    
            ...
    
    
            Spec:
    
    
             App - Name: demo-app
    
    
             App - Version: 1.1.0
    
    
             Release - Count: 5
    
    
            ...
    
    
            Events: <none>

You have a functioning custom resource which is now storing some data inside your Kubernetes cluster. You can remove the CRD by deleting it in Kubectl; this will automatically clean up all the objects that use it.

$ kubectl delete crd custom-apps.crds.example.com
    
    
            customresourcedefinition.apiextensions.k8s.io "custom-apps.crds.example.com" deleted

Building Declarative APIs With CRDs

This CRD's not adding any functionality to the cluster. It stores data, provides an API to interact with it, and can be referenced by other objects. CRDs become more powerful when they're paired with a custom controller that can take responsibility for managing them.

Controllers track resources and take actions in response to their events. Writing a controller for your CRDs lets you turn them into declarative APIs that cause real change inside your cluster. Your objects can represent the desired state, instead of the precise current state.

Cert-Manager uses this model to automatically acquire SSL certificates when new CertificateRequest objects appear in your cluster. Within the Kubernetes core, Nodes pull and run container images in response to the appearance of Pods. Controllers let you attach behavior to your own CRDs, so adding a "custom app" could cause its configuration to be fetched from an external service. You can start creating controllers by using the Go SDK to integrate your own code with the Kubernetes runtime.

When to Use CRDs?

CRDs are best used for handling data that's internal to your organization, team, or project. They're designed to represent clearly defined schemas, either as static values or declarative APIs backed by a custom controller implementation.

Advanced features let you implement validation routines for the fields in your schema. You can also use finalizers to handle object deletions and adopt the versioning system to handle changes to your CRD definitions.

CRDs sometimes overlap with Kubernetes ConfigMaps. These are built-in objects for storing generic config data associated with your applications. A ConfigMap is appropriate when you'll be consuming the values in a specific place in your cluster, such as a Pod that accesses database settings as environment variables injected from a ConfigMap.

CRDs are intended to be used when data needs to be a first-class citizen in your cluster. You can create multiple instances of the CRD's resource type, directly interact with them using Kubectl, and build your own structured schemas that guide users towards inputting correct values. They can be a better choice when the data exists independently of any other item in your cluster.

Summary

Kubernetes Custom Resource Definitions (CRDs) define new object types that you can use with the Kubernetes API and Kubectl. Each CRD get its own versioned API, has a structured schema, and can be used to implement new in-cluster functionality when backed by a companion controller.

CRDs can sometimes seem complicated and reserved for advanced situations. This doesn't have to be the case. Simple CRDs for storing static values in your cluster are easy to create, as illustrated by the "custom app" example above. They can be used to hold standalone cluster data so it receives first-class treatment within Kubernetes.

It's also important to recognize where CRDs aren't a great fit. Built-in objects like ConfigMaps and Secrets are designed to accommodate most forms of configuration that will be directly used by your application's Pods. Writing a CRD that defines your app's config file schema is usually unnecessary and trickier to maintain over time, as you won't benefit from ConfigMap features like automatic rolling updates and environment variable injection.