What really happens when you kubectl apply

This post is the short version of my video. Watch it here: What really happens when you kubectl apply

In the CNCF’s latest survey, 82% of companies that use containers run Kubernetes in production. Most of us still use it like a magic box: we type kubectl apply, and somehow the app is running.

Here’s what’s inside the box. All terminal output below comes from a real cluster: one control plane node and two workers.

Desired state is the whole idea

Kubernetes fits in one sentence: you describe what you want, and Kubernetes figures out how to get there.

You don’t say “start three containers”. You say “I want three copies of this app running”. Controllers then loop forever: observe the current state, compare it with the desired state, act to close the gap. That loop is called reconciliation, and everything below is a version of it.

The pieces:

Control plane: the API server (the front door for every request), etcd (where all cluster state lives), the scheduler (picks a node for each new pod) and the controller manager (runs the reconciliation loops).
Worker nodes: the kubelet (makes sure the right containers run on its node), a container runtime like containerd, and kube-proxy (service networking).

And the objects you’ll use most: a Pod wraps one or more containers with a shared network and its own IP, a ReplicaSet keeps a fixed number of identical pods running, and a Deployment manages ReplicaSets so you can roll forward and back.

kubectl apply, slowed down

Here’s a minimal Deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello
spec:
  replicas: 3
  selector:
    matchLabels:
      app: hello
  template:
    metadata:
      labels:
        app: hello
    spec:
      containers:
      - name: hello
        image: traefik/whoami:v1.10.3
        ports:
        - containerPort: 80

1. kubectl turns YAML into JSON. JSON is what the API speaks. This shows you exactly what it will send:

kubectl create -f deployment.yaml --dry-run=client -o json

2. It POSTs it to the API server. With -v=6 you can see the real request:

kubectl apply -f deployment.yaml -v=6 2>&1 | grep -E 'verb="POST"|created'

I1001 19:05:05.979225   80725 round_trippers.go:632] "Response" verb="POST" url="https://127.0.0.1:54812/apis/apps/v1/namespaces/default/deployments?fieldManager=kubectl-client-side-apply&fieldValidation=Strict" status="201 Created" milliseconds=5
deployment.apps/hello createdoutput

3. The API server runs a pipeline. Authentication (who are you?), authorization, usually RBAC (are you allowed to create deployments here?), admission controllers (fill in defaults, enforce policies, or reject), and schema validation.

4. It writes to etcd. That’s it. 201 Created means a record was written to a database. Nothing is running yet.

You can look inside etcd yourself:

kubectl -n kube-system exec etcd-demo-control-plane -- etcdctl \
    --cacert=/etc/kubernetes/pki/etcd/ca.crt \
    --cert=/etc/kubernetes/pki/etcd/server.crt \
    --key=/etc/kubernetes/pki/etcd/server.key \
    get /registry --prefix --keys-only |
  grep -E "(deployments|replicasets|pods)/default/hello"

/registry/deployments/default/hello
/registry/pods/default/hello-d87759d4d-kghk7
/registry/pods/default/hello-d87759d4d-vntqx
/registry/pods/default/hello-d87759d4d-xdr5z
/registry/replicasets/default/hello-d87759d4doutput

Every object in Kubernetes is just a key in etcd.

Nobody polls. Everybody watches.

So who created that ReplicaSet and those pods? The components don’t poll; they watch, and the API server streams every change to them.

The deployment controller sees a new Deployment and creates a ReplicaSet.
The ReplicaSet controller sees that and creates three pods, with no node yet.
The scheduler filters nodes (enough CPU and memory? do taints and affinity allow it?), scores the rest and writes a binding: this pod goes to demo-worker2. The control plane node is tainted, so regular pods never land there.
On that node, the kubelet sees a pod assigned to it. Through CRI it asks containerd to pull the image, the CNI plugin wires the network and assigns the pod IP, volumes get mounted (through CSI for storage), the containers start, probes run, and the kubelet reports Running.

The names tell the ownership story:

kubectl get deployment,replicaset,pod

NAME                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/hello   3/3     3            3           5s

NAME                              DESIRED   CURRENT   READY   AGE
replicaset.apps/hello-d87759d4d   3         3         3       5s

NAME                        READY   STATUS    RESTARTS   AGE
pod/hello-d87759d4d-kghk7   1/1     Running   0          5s
pod/hello-d87759d4d-vntqx   1/1     Running   0          5s
pod/hello-d87759d4d-xdr5z   1/1     Running   0          5soutput

Deployment hello, ReplicaSet hello plus a hash, pods with a random suffix. And kubectl describe pod shows the same flow in its events: scheduled, pulled, created, started.

Services: a stable address for things that move

Pods come and go, and their IPs change. A Service gives you one stable IP and DNS name, and load-balances across every pod that matches its selector. ClusterIP is internal, NodePort opens a port on every node, and LoadBalancer asks your provider for an external IP. For routing by host or path you add an Ingress or the Gateway API.

for i in 1 2 3 4; do curl -s http://172.100.100.200 | grep Hostname; done

Hostname: hello-d87759d4d-xdr5z
Hostname: hello-d87759d4d-xdr5z
Hostname: hello-d87759d4d-vntqx
Hostname: hello-d87759d4d-xdr5zoutput

Reconciliation, live

Self-healing. Delete a pod, and a new one shows up seconds later. The ReplicaSet saw two pods instead of three and fixed it.

Scaling. kubectl scale deployment hello --replicas=5, and five pods are running a few seconds later.

Rolling update and rollback. Change the image and Kubernetes swaps pods gradually, so the app never goes down. The old ReplicaSet scales to zero, the new one takes over:

kubectl get replicaset

NAME              DESIRED   CURRENT   READY   AGE
hello-d87759d4d   0         0         0       19s
hello-fd98cb9f8   5         5         5       3soutput

One command takes you back:

kubectl rollout undo deployment/hello

deployment.apps/hello rolled backoutput

kubectl get replicaset

NAME              DESIRED   CURRENT   READY   AGE
hello-d87759d4d   5         5         5       22s
hello-fd98cb9f8   0         0         0       6soutput

A rollback is just another rollout.

ConfigMaps and Secrets

Don’t bake configuration into the image. A ConfigMap holds plain key-value config, a Secret holds sensitive values, and a Deployment can use both as environment variables or as mounted files. Change the config, and you don’t rebuild the image.

One thing to remember: a Secret is base64-encoded, not encrypted. Anyone who can read it can decode it in one command. Lock it down with RBAC, turn on encryption at rest, or use an external secret manager.

From the internet to a pod

With Cloudflare in front: DNS answers with Cloudflare’s edge IP, not yours. The edge terminates TLS, filters attacks and caches what it can, then forwards to your origin, the external IP of your load balancer. The load balancer picks a node, kube-proxy’s rules route to a pod behind the Service, and the pod answers. In production you’d usually put an Ingress controller or the Gateway API behind that load balancer, or use a Cloudflare Tunnel so there’s no public IP at all.

The whole thing in one paragraph

You declare the desired state. kubectl turns it into JSON and posts it to the API server, which stores it in etcd. Controllers and the scheduler reconcile it, and the kubelet makes it real with containerd, CNI and CSI. Deployments manage ReplicaSets, ReplicaSets manage pods, Services give them a stable address, ConfigMaps and Secrets configure them, and a load balancer brings the traffic in.