Session 06 demos – Deployments, rollouts and probes

Demo 0 and Demos 1–5 belong to Block A (labels, ReplicaSets, rolling updates, probes), Demos 6–7 to Block B (scaling, restarts, Jobs). Everything runs on the pinboard kind cluster in the demo namespace, so nothing collides with the students’ pinboard namespace. Files next to this one:

File	Used in
(none)	Demo 0 — labels and selectors, created with `kubectl run`
`api-deploy-v1.yaml`	Demos 1–6 — 3 replicas, probes, `maxSurge 1 / maxUnavailable 0` (+ a Service)
`api-deploy-v2.yaml`	Demo 2 — image `1.1`, theme `amber`
`api-deploy-broken.yaml`	Demo 4 — image `1.2-broken`, crashes on start
`api-deploy-noprobe.yaml`	Demo 5 — no readiness probe, `FAIL_READY_AFTER=20`
`job-migrate.yaml`	Demo 7 — a one-shot “migration” Job
`cronjob.yaml`	Demo 7 — the same, every minute

Before the session

Cluster and images (≈ 3 min, do it before the room fills up):

cd ~/docker-kubernetes-training/demos/session06

kind get clusters                      # expect: pinboard
docker image ls | grep pinboard-api    # expect 1.0, 1.1, 1.2-broken

If a tag is missing, build it from src/:

docker build -t pinboard-api:1.0 --build-arg APP_VERSION=1.0 ../../src/pinboard-api
docker build -t pinboard-api:1.1 --build-arg APP_VERSION=1.1 ../../src/pinboard-api
# 1.2-broken = 1.0 with CRASH_ON_START baked in as an ENV (one extra layer)
printf 'FROM pinboard-api:1.0\nENV CRASH_ON_START=true\n' \
  | docker build -t pinboard-api:1.2-broken -

Load them into the cluster and prepare the namespace:

kind load docker-image pinboard-api:1.0 pinboard-api:1.1 pinboard-api:1.2-broken --name pinboard
docker pull busybox:1.37 && kind load docker-image busybox:1.37 --name pinboard
kubectl create namespace demo --dry-run=client -o yaml | kubectl apply -f -
kubectl config set-context --current --namespace=demo

Terminal layout — three panes, font size up:

Pane	Contents
1 (big, left)	where you type
2 (top right)	`watch -n 1 kubectl -n demo get deploy,rs,pods`
3 (bottom right)	the curl loop (started in Demo 2)

kubectl get -w watches one resource type at a time — kubectl get deploy,rs,pods -w is rejected. watch(1) is the pragmatic way to see the three levels at once; mention this, students will hit it.

Every -f below is relative to demos/session06. The manifests carry namespace: demo themselves, so apply -f needs no flag; the ad-hoc commands spell out -n demo anyway, so a shared screen never lies about the namespace.

Demo 0: labels and selectors (≈ 4 min)

What to show / say

Labels are the only thing holding Kubernetes together: a Service finds Pods by label, a ReplicaSet owns Pods by label, and every kubectl query you will ever type is a selector. They are not names and they are not comments.

Steps

Three throwaway Pods, labelled the way the recommended scheme says:

kubectl -n demo run web-1 --image=busybox:1.37 --restart=Never \
  --labels='app.kubernetes.io/name=pinboard-web,tier=front' -- sleep 600
kubectl -n demo run api-1 --image=busybox:1.37 --restart=Never \
  --labels='app.kubernetes.io/name=pinboard-api,tier=back' -- sleep 600
kubectl -n demo run api-2 --image=busybox:1.37 --restart=Never \
  --labels='app.kubernetes.io/name=pinboard-api,tier=back' -- sleep 600

pod/web-1 created
pod/api-1 created
pod/api-2 created

What is actually on them:

kubectl -n demo get pods --show-labels

NAME    READY   STATUS    RESTARTS   AGE   LABELS
api-1   1/1     Running   0          12s   app.kubernetes.io/name=pinboard-api,tier=back
api-2   1/1     Running   0          11s   app.kubernetes.io/name=pinboard-api,tier=back
web-1   1/1     Running   0          13s   app.kubernetes.io/name=pinboard-web,tier=front

👉 “No run= label any more — modern kubectl run only sets what you ask for. And app.kubernetes.io/name is a prefixed key: the prefix is a DNS domain, which is how tools avoid colliding on a label called name.”

Equality-based selection — the everyday form:

kubectl -n demo get pods -l app.kubernetes.io/name=pinboard-api
kubectl -n demo get pods -l tier=front --show-labels

NAME    READY   STATUS    RESTARTS   AGE
api-1   1/1     Running   0          40s
api-2   1/1     Running   0          39s

NAME    READY   STATUS    RESTARTS   AGE   LABELS
web-1   1/1     Running   0          41s   app.kubernetes.io/name=pinboard-web,tier=front

👉 “-l is the same selector a Service uses. When a Service has no endpoints, this is the first command you type — and nine times out of ten it returns nothing.”

Set-based selectors, and negation. Note the quotes — the shell would eat them:

kubectl -n demo get pods -l 'tier in (front,back)'
kubectl -n demo get pods -l 'app.kubernetes.io/name=pinboard-api,tier!=front'
kubectl -n demo get pods -l '!tier'

NAME    READY   STATUS    RESTARTS   AGE
api-1   1/1     Running   0          70s
api-2   1/1     Running   0          69s
web-1   1/1     Running   0          71s

NAME    READY   STATUS    RESTARTS   AGE
api-1   1/1     Running   0          70s
api-2   1/1     Running   0          69s

No resources found in demo namespace.

👉 “A comma is an AND, never an OR. in (…), notin (…), !key and key are the set-based forms — matchExpressions in YAML. Deployments and Services accept equality selectors; matchExpressions works in Deployments, Jobs and NetworkPolicy.”

Labels are mutable — and that is how canaries and quarantines work:
```
kubectl -n demo label pod api-2 tier=front --overwrite
kubectl -n demo label pod api-2 track=canary
kubectl -n demo get pods -L tier -L track
```
```
NAME    READY   STATUS    RESTARTS   AGE   TIER    TRACK
api-1   1/1     Running   0          2m    back
api-2   1/1     Running   0          2m    front   canary
web-1   1/1     Running   0          2m    front
```
👉 -L promotes a label to its own column — much easier to read than --show-labels once objects carry six labels. And say the punchline: “if api-2 had been a Pod owned by a ReplicaSet, changing the label it is selected by would have orphaned it — the ReplicaSet would immediately start a replacement, and the orphan keeps running. That is the standard trick for pulling one broken Pod out of a Service to debug it without losing capacity.”

Clean up before Demo 1:

kubectl -n demo delete pod web-1 api-1 api-2

Demo 1: Deployment → ReplicaSet → Pod (≈ 5 min)

What to show / say

A Deployment does not create Pods; it creates a ReplicaSet, and that creates Pods — three controllers, three names, one desired state.

Steps

Start pane 2 first, so the objects appear while you talk:
```
watch -n 1 kubectl -n demo get deploy,rs,pods
```

Show the manifest for ten seconds — only the four lines that matter:

grep -n -E "replicas|maxSurge|maxUnavailable|readinessProbe" api-deploy-v1.yaml

Apply it:

kubectl apply -f api-deploy-v1.yaml

deployment.apps/pinboard-api created
service/pinboard-api created

👉 point at pane 2 while the Pods appear:

NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/pinboard-api   3/3     3            3           18s

NAME                                      DESIRED   CURRENT   READY   AGE
replicaset.apps/pinboard-api-7d9c8f6b45   3         3         3       18s

NAME                                 READY   STATUS    RESTARTS   AGE
pod/pinboard-api-7d9c8f6b45-4kq2r    1/1     Running   0          18s
pod/pinboard-api-7d9c8f6b45-9xnzt    1/1     Running   0          18s
pod/pinboard-api-7d9c8f6b45-vc7hd    1/1     Running   0          18s

👉 point at the shared name prefix: pinboard-api → -7d9c8f6b45 (the pod-template-hash of the ReplicaSet) → -4kq2r (the Pod’s own suffix).

Prove the ownership chain rather than asserting it:

kubectl -n demo get rs -o jsonpath='{.items[0].metadata.ownerReferences[0].kind}{"\n"}'
kubectl -n demo get pod -l app.kubernetes.io/name=pinboard-api \
  -o jsonpath='{.items[0].metadata.ownerReferences[0].kind}{"\n"}'

Deployment
ReplicaSet

Delete a Pod and let the room watch it come back:
```
POD=$(kubectl -n demo get pod -l app.kubernetes.io/name=pinboard-api -o name | head -1)
kubectl -n demo delete "$POD"
```
👉 pane 2: READY 2/3 for a second or two, then 3/3 again with one new Pod name. Say it out loud: “nobody fixed that — the ReplicaSet controller did, within a reconcile loop.”

The one-liner students will use forever:

kubectl -n demo rollout status deploy/pinboard-api

deployment "pinboard-api" successfully rolled out

Demo 2: A rolling update you can watch (1.0 → 1.1) (≈ 8 min)

What to show / say

maxSurge: 1 / maxUnavailable: 0 means the Service never has fewer than three healthy backends — a version change is visible in the responses, not in the errors.

Steps

Start the traffic loop in pane 3 (busybox inside the cluster, so it goes through the Service and round-robins over the Pods):

kubectl -n demo run curlloop --image=busybox:1.37 --restart=Never --rm -it -- \
  sh -c 'while true; do wget -q -T 2 -O - http://pinboard-api:8080/api/info \
    | sed -e "s/.*hostname\":\"\([^\"]*\)\".*theme\":\"\([^\"]*\)\".*/\1 \2/"; sleep 0.3; done'

pinboard-api-7d9c8f6b45-4kq2r emerald
pinboard-api-7d9c8f6b45-vc7hd emerald
pinboard-api-7d9c8f6b45-9xnzt emerald

👉 point at the hostnames cycling — one Service, three Pods.

In pane 1, roll out 1.1:
```
kubectl apply -f api-deploy-v2.yaml
```

👉 pane 2 during the ~20 s of the rollout: a second ReplicaSet appears, the old one drains one Pod at a time, total Pods peak at 4 (that is maxSurge: 1):

NAME                                      DESIRED   CURRENT   READY   AGE
replicaset.apps/pinboard-api-7d9c8f6b45   1         1         1       6m
replicaset.apps/pinboard-api-6b4f7c9d82   3         3         2       12s

👉 pane 3 — the payoff. The theme flips to amber without a single failed request, and for a few seconds both versions answer:
```
pinboard-api-7d9c8f6b45-vc7hd emerald
pinboard-api-6b4f7c9d82-t8w5x amber
pinboard-api-7d9c8f6b45-9xnzt emerald
pinboard-api-6b4f7c9d82-hn2qc amber
...
pinboard-api-6b4f7c9d82-t8w5x amber
```
Say: “that overlap is normal and it is why your app must tolerate N and N+1 running at the same time.”

Follow the rollout in text as well:

kubectl -n demo rollout status deploy/pinboard-api

Waiting for deployment "pinboard-api" rollout to finish: 2 of 3 updated replicas are available...
deployment "pinboard-api" successfully rolled out

Optional, if a browser is on screen: kubectl -n demo port-forward svc/pinboard-api 8080:8080 and reload http://localhost:8080/api/info. 👉 mention that port-forward pins one Pod, which is why the loop above uses a Pod inside the cluster instead.

Leave the loop running — Demos 3 to 6 use it.

Demo 3: `rollout history` and `undo` (≈ 4 min)

What to show / say

Revisions are just the old ReplicaSets kept around; undo is not magic, it scales one back up.

Steps

kubectl -n demo rollout history deploy/pinboard-api

deployment.apps/pinboard-api
REVISION  CHANGE-CAUSE
1         <none>
2         <none>

👉 CHANGE-CAUSE is <none> because nobody annotated it. Fix that in one line:

kubectl -n demo annotate deploy/pinboard-api kubernetes.io/change-cause="theme amber, image 1.1"
kubectl -n demo rollout history deploy/pinboard-api --revision=2 | head -12

Go back:
```
kubectl -n demo rollout undo deploy/pinboard-api
kubectl -n demo rollout status deploy/pinboard-api
```
👉 pane 3: the theme returns to emerald; pane 2: the old ReplicaSet scales from 0 back to 3 — no image is pulled, nothing is rebuilt.
```
kubectl -n demo rollout history deploy/pinboard-api
```
```
REVISION  CHANGE-CAUSE
2         theme amber, image 1.1
3         <none>
```
👉 revision 1 is gone and a new revision 3 was appended — undo rolls forward to an old spec. revisionHistoryLimit: 5 caps how far back you can go.

Roll forward again so the next demo starts from 1.1:

kubectl -n demo rollout undo deploy/pinboard-api

Demo 4: A broken image, and the way out (≈ 6 min)

What to show / say

The safety net is maxUnavailable: 0 + readiness: a bad release stalls instead of causing an outage. And it stalls loudly.

Steps

kubectl apply -f api-deploy-broken.yaml

👉 pane 3: nothing changes. The loop keeps printing amber from the 1.1 Pods. Let that sink in before showing pane 2.

👉 pane 2 after ~60 s:

NAME                                 READY   STATUS             RESTARTS      AGE
pod/pinboard-api-5f7b9d4c61-2mnpq    0/1     CrashLoopBackOff   3 (22s ago)   64s
pod/pinboard-api-6b4f7c9d82-hn2qc    1/1     Running            0             7m
pod/pinboard-api-6b4f7c9d82-t8w5x    1/1     Running            0             7m
pod/pinboard-api-6b4f7c9d82-x4dlv    1/1     Running            0             7m

Only one new Pod was ever created (surge 1), and the old three are intact.

Diagnose it the way Session 09 will formalise (get → describe → logs):

kubectl -n demo logs -l app.kubernetes.io/name=pinboard-api --tail=1 --prefix

[pod/pinboard-api-5f7b9d4c61-2mnpq/api] time=2026-05-12T09:31:44.112Z level=ERROR msg="CRASH_ON_START is set — exiting with status 1 (this is on purpose)"
[pod/pinboard-api-6b4f7c9d82-hn2qc/api] time=2026-05-12T09:24:02.907Z level=INFO msg="pinboard-api starting" version=1.1 theme=amber store=memory addr=:8080

👉 for a crash-looping Pod kubectl logs shows the last terminated container — you do not have to catch it while it is up.

The rollout gives up on its own:

kubectl -n demo rollout status deploy/pinboard-api --timeout=90s

Waiting for deployment "pinboard-api" rollout to finish: 1 out of 3 new replicas have been updated...
error: deployment "pinboard-api" exceeded its progress deadline

👉 this exit code is what a CI pipeline should fail on.

Get out:

kubectl -n demo rollout undo deploy/pinboard-api
kubectl -n demo rollout status deploy/pinboard-api

deployment "pinboard-api" successfully rolled out

👉 pane 3 never printed one error. That is the demo.

Demo 5: No readiness probe = outage (≈ 7 min)

What to show / say

Readiness is how a Pod says “don’t send me traffic”. Delete the probe and the Service cheerfully load-balances onto broken Pods.

Steps

Show the diff first — it is three lines:
```
diff <(grep -v '^ *#' api-deploy-v1.yaml) <(grep -v '^ *#' api-deploy-noprobe.yaml) | head -20
```
👉 readinessProbe removed, FAIL_READY_AFTER=20 added. /healthz still answers 200, so the liveness probe will never restart anything.

Point pane 3 at the readiness endpoint instead of /api/info. Stop the old loop (Ctrl-C) and start:

kubectl -n demo run curlloop --image=busybox:1.37 --restart=Never --rm -it -- \
  sh -c 'while true; do wget -q -T 2 -O - http://pinboard-api:8080/readyz || echo "FAILED"; sleep 1; done'

```
kubectl apply -f api-deploy-noprobe.yaml
kubectl -n demo rollout status deploy/pinboard-api
```
👉 the rollout finishes in seconds — with no readiness probe a Pod counts as Ready as soon as the container process starts.

Wait ~25 s, then 👉 pane 3:

ready
ready
not ready: FAIL_READY_AFTER elapsed
not ready: FAIL_READY_AFTER elapsed

and 👉 pane 1:

kubectl -n demo get pods
kubectl -n demo get endpointslices -l kubernetes.io/service-name=pinboard-api -o wide

NAME                             READY   STATUS    RESTARTS   AGE
pinboard-api-84cd6f9b77-6hkkd    1/1     Running   0          75s
...
NAME                 ADDRESSTYPE   PORTS   ENDPOINTS                          READY
pinboard-api-p9v2k   IPv4          8080    10.244.1.9,10.244.2.7,10.244.1.11  true,true,true

Say it plainly: “every Pod is telling us it is broken, every Pod is 1/1 Ready, and the Service is still sending it traffic. Kubernetes is not clairvoyant.”

Put the probe back, keeping the same failure:

kubectl apply -f api-deploy-v1.yaml
kubectl -n demo set env deploy/pinboard-api FAIL_READY_AFTER=20
kubectl -n demo rollout status deploy/pinboard-api

Wait ~30 s and repeat the two commands from step 4:

NAME                             READY   STATUS    RESTARTS   AGE
pinboard-api-6c58d7f9c4-b7rlt    0/1     Running   0          48s
pinboard-api-6c58d7f9c4-jm4sw    0/1     Running   0          48s
pinboard-api-6c58d7f9c4-nz8gq    0/1     Running   0          48s

NAME                 ADDRESSTYPE   PORTS   ENDPOINTS   READY
pinboard-api-p9v2k   IPv4          8080    <unset>     false,false,false

👉 0/1 Ready, the EndpointSlice marks every address not-ready, and the loop now fails to connect at all. Kubernetes has taken the app out of rotation and is shouting about it. Same bug, two completely different operational outcomes.

Back to a healthy 1.0 (apply drops the env var, because it is not in the file):

kubectl apply -f api-deploy-v1.yaml
kubectl -n demo rollout status deploy/pinboard-api

Demo 6: `scale` and `rollout restart` (≈ 4 min)

What to show / say

Two commands you will type every week, and what makes them different.

Steps

```
kubectl -n demo scale deploy/pinboard-api --replicas=5
```
👉 pane 2: two Pods added, same ReplicaSet, no new revision.

kubectl -n demo rollout history deploy/pinboard-api | tail -3

👉 scaling is not a revision — the Pod template did not change.

kubectl -n demo scale deploy/pinboard-api --replicas=3
kubectl -n demo rollout restart deploy/pinboard-api
kubectl -n demo rollout status deploy/pinboard-api

👉 pane 2: a new ReplicaSet, a normal rolling update, zero errors in pane 3.

Where the restart comes from:
```
kubectl -n demo get deploy pinboard-api -o jsonpath='{.spec.template.metadata.annotations}{"\n"}'
```
```
{"kubectl.kubernetes.io/restartedAt":"2026-05-12T09:48:03Z"}
```
👉 it just stamps an annotation into the Pod template — the template changed, so the Deployment controller does what it always does. Use it after a Secret or ConfigMap change (Session 08).

Demo 7: Job and CronJob (≈ 6 min)

What to show / say

Deployments are for things that should never finish. Jobs are for things that must finish exactly once; CronJobs are Jobs with a schedule.

Steps

kubectl apply -f job-migrate.yaml
kubectl -n demo get job,pod -l app.kubernetes.io/name=pinboard-migrate

NAME                         STATUS    COMPLETIONS   DURATION   AGE
job.batch/pinboard-migrate   Running   0/1           5s         5s

NAME                          READY   STATUS    RESTARTS   AGE
pod/pinboard-migrate-tq6zm    1/1     Running   0          5s

kubectl -n demo logs job/pinboard-migrate -f

==> connecting to database
==> applying 001_notes.sql
==> applying 002_notes_author.sql
==> migration complete

kubectl -n demo get job pinboard-migrate
kubectl -n demo get pod -l app.kubernetes.io/name=pinboard-migrate

NAME               STATUS      COMPLETIONS   DURATION   AGE
pinboard-migrate   Complete    1/1           11s        40s

NAME                       READY   STATUS      RESTARTS   AGE
pinboard-migrate-tq6zm     0/1     Completed   0          40s

👉 the Pod stays in Completed so you can still read its logs — that is a feature, not a leak. ttlSecondsAfterFinished: 600 cleans it up later. 👉 also point at restartPolicy: OnFailure in the manifest: Always is rejected for Jobs, because “always restart” and “run once” are contradictions.

Now the scheduled version:

kubectl apply -f cronjob.yaml
kubectl -n demo get cronjob

NAME              SCHEDULE    TIMEZONE   SUSPEND   ACTIVE   LAST SCHEDULE   AGE
pinboard-readyz   * * * * *   <none>     False     0        <none>          8s

Wait for the top of the next minute (fill it with the concurrencyPolicy / startingDeadlineSeconds slide), then:

kubectl -n demo get jobs -l app.kubernetes.io/name=pinboard-readyz
kubectl -n demo logs -l app.kubernetes.io/name=pinboard-readyz --tail=2

NAME                       STATUS     COMPLETIONS   DURATION   AGE
pinboard-readyz-29061234   Complete   1/1           2s         35s

checking pinboard-api ...
ready

Trigger one by hand instead of waiting — the trick nobody knows:

kubectl -n demo create job readyz-now --from=cronjob/pinboard-readyz
kubectl -n demo logs job/readyz-now

Stop the schedule without deleting it:

kubectl -n demo patch cronjob pinboard-readyz -p '{"spec":{"suspend":true}}'

Reset

Lab 06 expects a clean demo namespace (the students work in pinboard), the three images loaded, and no leftover CronJob firing every minute during the lab.

# stop the traffic loop first (Ctrl-C in pane 3), then:
kubectl -n demo delete pod curlloop --ignore-not-found
kubectl -n demo delete pod web-1 api-1 api-2 --ignore-not-found   # Demo 0, if you skipped its cleanup
kubectl -n demo delete -f cronjob.yaml -f job-migrate.yaml --ignore-not-found
kubectl -n demo delete job readyz-now --ignore-not-found
kubectl delete -f api-deploy-v1.yaml --ignore-not-found     # Deployment + Service
kubectl -n demo get all

No resources found in demo namespace.

Keep the demo namespace itself — Session 07 reuses it:

kubectl get ns demo
kubectl config set-context --current --namespace=default

Session 06 demos – Deployments, rollouts and probes

Before the session

Demo 0: labels and selectors (≈ 4 min)

What to show / say

Steps

Demo 1: Deployment → ReplicaSet → Pod (≈ 5 min)

What to show / say

Steps

Demo 2: A rolling update you can watch (1.0 → 1.1) (≈ 8 min)

What to show / say

Steps

Demo 3: rollout history and undo (≈ 4 min)

What to show / say

Steps

Demo 4: A broken image, and the way out (≈ 6 min)

What to show / say

Steps

Demo 5: No readiness probe = outage (≈ 7 min)

What to show / say

Steps

Demo 6: scale and rollout restart (≈ 4 min)

What to show / say

Steps

Demo 7: Job and CronJob (≈ 6 min)

What to show / say

Steps

Reset

Demo 3: `rollout history` and `undo` (≈ 4 min)

Demo 6: `scale` and `rollout restart` (≈ 4 min)