Custom Resource Definitions (CRDs) are Kubernetes API extensions which can define new object types. Pods, ReplicaSets, ConfigMaps, and Ingresses are examples of common built-in resources. CRDs let you add entirely new types to this list, then manage them using familiar Kubernetes tools such as Kubectl.
The CRD mechanism is intentionally abstract and can be used in myriad ways to store data and build new functionality. You’ll find custom resources in many popular community tools: Cert-Manager defines objects that represent SSL Certificates and Issuers while Helm represents Charts as their own CRD.
What Makes a Resource?
Kubernetes resources define the types of data you can store in your cluster. They’re accessed via the Kubernetes API which provides endpoints for creating, listing, and modifying items within each resource type.
You can add custom resources to store your own arbitrary data inside your cluster. Items you create will be stored by the etcd control plane component, alongside instances of built-in resources. Custom resources are automatically surfaced by the API so you don’t need to set up your own tooling to create item instances.
CRDs act as simple data structures by default. Although CRDs in the wild often have their own behaviors, this isn’t provided by the CRD mechanism. Custom Kubernetes controllers and operators can be used to implement functionality around custom resources. Without a controller, a CRD’s items will always exist as static in-cluster data that you can interact with via CRUD endpoints provided by the Kubernetes API.
CRDs are dynamic components which can be created and removed at any time. Some object types that are included with Kubernetes are implemented as CRDs too, affording more modularity within the cluster’s core.
Creating a CRD
CRDs are themselves a type of Kubernetes object. You create them in the same way as any other resource, by writing a YAML file and applying it to your cluster:
apiVersion: apiextensions.k8s.io/v1 kind: CustomResourceDefinition metadata: name: custom-apps.crds.example.com spec: group: crds.example.com versions: - name: v1 served: true storage: true schema: openAPIV3Schema: type: object properties: spec: type: object properties: app-name: type: string app-version: type: string release-count: type: integer scope: Namespaced names: plural: custom-apps singular: custom-app kind: CustomApp
Use Kubectl to add the CustomApp CRD to your cluster:
$ kubectl apply -f custom-apps-crd.yaml customresourcedefinition.apiextensions.k8s.io/custom-apps.crds.example.com created
The YAML file defines a CRD that could be used to store data about apps. CRDs need a
spec.group field in a precise format: the group takes the form of a subdomain which the CRD belongs to. The same subdomain must be included in the CRD’s
metadata.name . The value of
names.plural is prepended as a new subdomain component to build the final
spec.names field defines how you’ll refer to the CRD when using the Kubernetes API and Kubectl commands. In this example you’ll be able to run
kubectl get custom-apps and
kubectl get custom-app/example-app to interact with objects that use the CRD. When you’re creating a new object as a YAML file, you should set
kind: CustomApp to make it an instance of the CRD.
The CRD is configured as a namespace-level object type by the
scope field. You can use
Cluster as an alternative value for this field to create objects that exist at the cluster-level, outside any namespace.
The data associated with your custom objects is defined in the
spec.versions.schema.openAPIV3Schema field. Each listed “version” creates a new version of the CRD’s API that you can reference with the
apiVersion field in your resource YAML files. The CRD’s data is configured using OpenAPI properties; here, each “custom app” in your cluster should have
release-count properties set in its YAML
Using Your CRD
It can take a few minutes for a new CRD’s API endpoints to be provisioned. Check the progress by retrieving the CRD’s details with Kubectl:
$ kubectl describe crd custom-apps.crds.example.com ... Status: Accepted Names: Kind: CustomApp List Kind: CustomAppList Plural: custom-apps Singular: custom-app Conditions: Last Transition Time: 2022-04-04T13:29:24Z Message: no conflicts found Reason: NoConflicts Status: True Type: NamesAccepted Last Transition Time: 2022-04-04T13:29:24Z Message: the initial names have been accepted Reason: InitialNamesAccepted Status: True Type: Established ...
The CRD is ready to use when you see
Type: Established near the end of the command’s output. Kubernetes will be accepting requests to the CRD’s API endpoint. In this case, the API base URL will be as follows:
You can now use Kubectl to view objects that have been created with the CRD:
$ kubectl get custom-apps No resources found in default namespace.
Although no objects exist yet, Kubernetes now knows it has a resource type called
To create a “custom app” object, write a new YAML file with
kind: CustomApp. The
apiVersion must be set to the group name and API version provided by the CRD. Within the
spec section, include the properties you defined in the CRD’s schema.
apiVersion: crds.example.com/v1 kind: CustomApp metadata: name: demo-app-1 spec: app-name: demo-app app-version: 1.1.0 release-count: 5
Use Kubectl to add the object to your cluster:
$ kubectl apply -f custom-app.yaml customapp.crds.example.com/demo-app created
Now you can retrieve the object’s details using familiar Kubectl commands:
$ kubectl describe custom-app/demo-app-1 Name: demo-app Namespace: default Labels: <none> Annotations: <none> API Version: crds.example.com/v1 Kind: CustomApp ... Spec: App - Name: demo-app App - Version: 1.1.0 Release - Count: 5 ... Events: <none>
You have a functioning custom resource which is now storing some data inside your Kubernetes cluster. You can remove the CRD by deleting it in Kubectl; this will automatically clean up all the objects that use it.
$ kubectl delete crd custom-apps.crds.example.com customresourcedefinition.apiextensions.k8s.io "custom-apps.crds.example.com" deleted
Building Declarative APIs With CRDs
This CRD’s not adding any functionality to the cluster. It stores data, provides an API to interact with it, and can be referenced by other objects. CRDs become more powerful when they’re paired with a custom controller that can take responsibility for managing them.
Controllers track resources and take actions in response to their events. Writing a controller for your CRDs lets you turn them into declarative APIs that cause real change inside your cluster. Your objects can represent the desired state, instead of the precise current state.
Cert-Manager uses this model to automatically acquire SSL certificates when new CertificateRequest objects appear in your cluster. Within the Kubernetes core, Nodes pull and run container images in response to the appearance of Pods. Controllers let you attach behavior to your own CRDs, so adding a “custom app” could cause its configuration to be fetched from an external service. You can start creating controllers by using the Go SDK to integrate your own code with the Kubernetes runtime.
When to Use CRDs?
CRDs are best used for handling data that’s internal to your organization, team, or project. They’re designed to represent clearly defined schemas, either as static values or declarative APIs backed by a custom controller implementation.
Advanced features let you implement validation routines for the fields in your schema. You can also use finalizers to handle object deletions and adopt the versioning system to handle changes to your CRD definitions.
CRDs sometimes overlap with Kubernetes ConfigMaps. These are built-in objects for storing generic config data associated with your applications. A ConfigMap is appropriate when you’ll be consuming the values in a specific place in your cluster, such as a Pod that accesses database settings as environment variables injected from a ConfigMap.
CRDs are intended to be used when data needs to be a first-class citizen in your cluster. You can create multiple instances of the CRD’s resource type, directly interact with them using Kubectl, and build your own structured schemas that guide users towards inputting correct values. They can be a better choice when the data exists independently of any other item in your cluster.
Kubernetes Custom Resource Definitions (CRDs) define new object types that you can use with the Kubernetes API and Kubectl. Each CRD get its own versioned API, has a structured schema, and can be used to implement new in-cluster functionality when backed by a companion controller.
CRDs can sometimes seem complicated and reserved for advanced situations. This doesn’t have to be the case. Simple CRDs for storing static values in your cluster are easy to create, as illustrated by the “custom app” example above. They can be used to hold standalone cluster data so it receives first-class treatment within Kubernetes.
It’s also important to recognize where CRDs aren’t a great fit. Built-in objects like ConfigMaps and Secrets are designed to accommodate most forms of configuration that will be directly used by your application’s Pods. Writing a CRD that defines your app’s config file schema is usually unnecessary and trickier to maintain over time, as you won’t benefit from ConfigMap features like automatic rolling updates and environment variable injection.