How I Finally Understood CRDs

Open a terminal and look

One day I got curious and typed this into a terminal.

kubectl get crds

The scroll went on for a while. Hundreds of lines. ArgoCD Application, KEDA ScaledObject, cert-manager Certificate — all names I look at every day. But staring at that list, something felt off.

I've been working with Kubernetes for years. I've run clusters, migrated them. And yet if you asked me to explain how Kubernetes actually gets extended, I couldn't do it. I'd been using a tool daily without understanding how it works.

Kubernetes is a language, not a platform

When people think Kubernetes, Pod, Deployment, and Service come to mind first. I sat at that level for a long time. It's just the surface.

Go a bit deeper and Kubernetes looks less like a platform and more like a language. CRDs are how you register new words in it.

The structure is genuinely strange when you think about it. The moment you define something like DatabaseCluster, MLModel, or TenantConfig, it becomes a first-class API object. You can query it with kubectl. It's stored in etcd. RBAC applies to it. You can watch it like any native resource.

The abstraction that lived in my head becomes part of Kubernetes itself. Anyone who's designed a data model knows how big a claim that is. Normally, adding a new domain object means hand-wiring storage, an API endpoint, permission checks, and a query interface. Here you get all of it from one declaration.

Writing an actual CRD

Theory didn't land for me, so I wrote a DatabaseCluster CRD myself.

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: databaseclusters.infra.example.com
spec:
  group: infra.example.com
  scope: Namespaced
  names:
    plural: databaseclusters
    singular: databasecluster
    kind: DatabaseCluster
    shortNames:
      - dbc
  versions:
    - name: v1alpha1
      served: true
      storage: true
      subresources:
        status: {}
      additionalPrinterColumns:
        - name: Replicas
          type: integer
          jsonPath: .spec.replicas
        - name: Region
          type: string
          jsonPath: .spec.region
        - name: Phase
          type: string
          jsonPath: .status.phase
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              required: ["engine", "replicas", "region"]
              properties:
                engine:
                  type: string
                  enum: ["postgres", "mysql", "mariadb"]
                replicas:
                  type: integer
                  minimum: 1
                  maximum: 9
                region:
                  type: string
                storageGB:
                  type: integer
                  minimum: 10
                  default: 20
            status:
              type: object
              properties:
                phase:
                  type: string
                endpoint:
                  type: string

A few things are worth a second look.

group and names give you a brand new API endpoint at infra.example.com/v1alpha1. No server restart. That part felt unreal at first — adding an API usually means a deploy.

scope decides the blast radius: Namespaced means it lives per namespace, Cluster means cluster-wide. This is where your multi-tenancy design forks.

The status entry under subresources is the important one. Users define only spec (the desired state), and status (the actual state) can only be touched by the controller.

A CRD doesn't run anything

This trips a lot of people up: a CRD by itself does nothing.

It only defines intent. CRDs define the vocabulary; operators and controllers do the work. Without a controller, kubectl get dbc shows you an object sitting there and nothing else happens.

# my-database.yaml
apiVersion: infra.example.com/v1alpha1
kind: DatabaseCluster
metadata:
  name: production-postgres
  namespace: databases
spec:
  engine: postgres
  replicas: 3
  region: ap-northeast-2
  storageGB: 100

Apply it:

kubectl apply -f my-database.yaml
kubectl get dbc -n databases

# NAME                 REPLICAS   REGION           PHASE     AGE
# production-postgres  3         ap-northeast-2   <none>    10s

Phase is <none> because there's no controller to fill in status yet. The declaration was accepted; nobody's around to make it real. Once you see that, it's obvious why the operator pattern always shows up alongside CRDs.

Why the status subresource matters

For a while I didn't get why subresources needed status: {}. It works fine without it, so I left it out for a stretch. Turns out it's the line between people who use CRDs and people who design platforms.

With the status subresource in place, users can only modify spec, and only the controller can update status. That separation is enforced at the API level.

It's not a convention, it's enforced architecture. Users declare the desired state, controllers report the actual state. What matters is that the responsibility boundary is pinned down by an API spec instead of a code convention. Anyone who's operated systems knows the difference. Conventions get broken. Things the API blocks don't.

The kubectl trick nobody mentions

Add additionalPrinterColumns and kubectl output changes completely.

kubectl get dbc

# NAME                 REPLICAS   REGION           PHASE      AGE
# production-postgres  3         ap-northeast-2   Ready      2d
# staging-mysql       1         ap-northeast-2   Pending    5m

kubectl becomes the dashboard, no UI required. Sounds minor, but whether an operator can read state at a glance feeds straight into how fast incidents get handled.

Mistakes I keep seeing

A few failure modes come up over and over.

Ship a CRD without a schema and anything goes in, which makes debugging hell later. You've just deferred validation to runtime. Dropping the status subresource is the same species of mistake, except here users can overwrite system state.

Changing schemas on a whim is another common one. Existing resources break, so touching a schema without a versioning strategy is dangerous. And deploying a CRD with no controller logic gives you declarations with no behavior — nothing happens.

Most CRD problems aren't Kubernetes problems, they're API design problems. It's the same set of tradeoffs as designing an OpenAPI spec. Loosening the schema is easier today, and you pay for it later in debugging and broken compatibility.

When it finally clicked

Once CRDs made sense, the whole cloud native ecosystem looked different. ArgoCD, KEDA, cert-manager — I could see they all run on the same principle.

They weren't "extending" Kubernetes. They were adding vocabulary to the Kubernetes language. Not plugins, not hacks. Just new nouns. Tools that looked unrelated turned out to share a root.

That's when it made sense why Kubernetes ended up with such a thick ecosystem. There's one standardized path for extending the API, so hundreds of independently built tools all behave the same way. A single standard interface holds the consistency of the whole ecosystem up.

It felt like finally seeing the real face of a tool I'd been handling every day. Using a tool and understanding why it was designed that way are two different seats, and the view is not the same.