Session 06 · 2h

Workloads: Deployments, rollouts and probes

Stop running Pods by hand. Let a controller own them, ship 1.0 → 1.1 with nobody noticing, and undo it when it goes wrong.

Labels & selectors Deployments & rollouts Probes Other controllers Lab 06

Session 06

What we'll cover today

1 · Labels, selectors, controllers

How Kubernetes finds objects, the recommended app.kubernetes.io/* labels, and ReplicaSet vs Deployment.

2 · Rollouts & probes

maxSurge/maxUnavailable, rollout status/history/undo, and the three probes that make it safe.

3 · Other controllers

DaemonSet, StatefulSet, Job and CronJob at a glance — plus scaling and graceful termination.

4 · Lab 06

Make the Pinboard API resilient: 3 replicas, probes, a zero-downtime 1.0 → 1.1, then a rollback from a broken image.

Labels, selectors, controllers

Kubernetes has no folders. Objects find each other by matching key/value labels — and that is the whole trick.

The glue

Labels — and what they are not

Labels are identifying key/value pairs used to select objects
Annotations are non-identifying metadata — never selected on (checksums, URLs, tool state)
Services, ReplicaSets, NetworkPolicies and HPAs all work by selecting labels
Labels are mutable; change one and membership changes instantly

metadata:
  name: pinboard-api
  namespace: pinboard
  labels:
    app.kubernetes.io/name: pinboard-api
    app.kubernetes.io/component: api
    app.kubernetes.io/part-of: pinboard
  annotations:
    kubernetes.io/change-cause: "bump to 1.1"

Relabel a running Pod and its ReplicaSet will consider it lost — and immediately create a replacement.

Matching

Selectors: equality and set-based

kubectl get pods -l app.kubernetes.io/name=pinboard-api
kubectl get pods -l 'app.kubernetes.io/component!=db'
kubectl get pods -l 'app.kubernetes.io/component in (api,web)'
kubectl get pods -l app.kubernetes.io/part-of=pinboard --show-labels
kubectl delete pods -l app.kubernetes.io/component=api    # careful

selector:                       # Deployments/ReplicaSets: matchLabels + matchExpressions
  matchLabels:
    app.kubernetes.io/name: pinboard-api
  matchExpressions:
    - { key: app.kubernetes.io/component, operator: In, values: [api] }

Services only understand equality selectors; Deployments, ReplicaSets and Jobs understand both forms. Point out that a Deployment's spec.selector is immutable after creation — get it wrong and you delete and recreate the Deployment. Show the multi-label select live if the cluster is up, then --show-labels so the class sees what they are matching on.

Convention

The recommended labels

Label	Pinboard value	Why
app.kubernetes.io/name	`pinboard-api`	The application itself — our selector key
app.kubernetes.io/component	`api` · `web` · `db`	The role inside the system
app.kubernetes.io/part-of	`pinboard`	Groups the whole app — one query shows everything
app.kubernetes.io/version	`"1.1"`	Human-readable release; not the selector
app.kubernetes.io/managed-by	`kubectl` · `helm`	Who owns this object

Select only on stable labels. Never put version in matchLabels — the selector is immutable, so every release would need a new Deployment.

These are the shared vocabulary that lets dashboards, Helm, Argo CD and k9s group your resources without configuration. We use them in every manifest for the rest of the course, including labs/solutions/final/. Stress the version trap on the last line: it is the single most common labelling mistake, and it only bites at the first upgrade.

The controller under the hood

ReplicaSet: keep N Pods alive

Watches Pods matching its selector, counts them, creates or deletes to reach replicas
Owns its Pods through metadata.ownerReferences
Delete a Pod → a replacement appears in seconds
Node dies → its Pods are recreated elsewhere
It cannot change the Pod template safely — no rollout logic at all

The self-healing you were promised

This is the Session 05 reconciliation loop with one concrete goal: observed replicas == desired replicas.

You will almost never write a ReplicaSet by hand — but every kubectl get rs line you see for the rest of the course is one.

Demonstrate if you have the cluster: kubectl delete pod <one-of-three> and run kubectl get pods -w — the replacement is Running before the room finishes reading. Mention that ReplicationController is the pre-1.2 ancestor and only exists in old blog posts. Then set up the next slide: the missing piece is what happens when the image tag changes.

Layering

Deployment = ReplicaSets over time

Deployment

You edit this — the desired template

→

ReplicaSet 1.0

Scaled 3 → 0 during the rollout

→

ReplicaSet 1.1

New template hash, scaled 0 → 3

→

Pods

Owned by whichever RS created them

	ReplicaSet	Deployment
Keeps N Pods running	yes	yes — through a ReplicaSet
Change the image safely	no	rolling update, one step at a time
History & rollback	no	`rollout history` / `undo`

The manifest

The Pinboard API Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: pinboard-api
  namespace: pinboard
spec:
  replicas: 3
  selector:
    matchLabels:                              # immutable — choose it once
      app.kubernetes.io/name: pinboard-api
  template:                                   # ← the Pod from Session 05
    metadata:
      labels:
        app.kubernetes.io/name: pinboard-api

template.spec is the Pod spec you already wrote — containers, resources, probes
The template's labels must satisfy spec.selector, or the API rejects it

Show them the shortcut: kubectl create deployment … --dry-run=client -o yaml generates exactly this skeleton. Walk from the outside in and stop at the template — everything below it is last session's Pod, unchanged. That continuity is the point: they already know how to fill it. Warn again that the selector cannot be edited later; kubectl apply will fail with a field-immutable error and the only fix is delete and recreate. The full file is labs/solutions/final/20-api-deployment.yaml.

Rollouts

Shipping pinboard-api:1.0 → 1.1 while people are using it — and getting back if you were wrong.

Mechanics

How a rolling update actually steps

3 × 1.0

Steady state, all Ready

→

+1 surge

One 1.1 Pod created — 4 exist

→

wait Ready

Readiness probe passes, endpoint added

→

−1 old

An old Pod is terminated gracefully

↺

3 × 1.1

Repeat until the new RS owns all replicas

spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1          # at most 4 Pods exist at once
      maxUnavailable: 0    # never fewer than 3 Ready — capacity is never reduced

Both knobs accept a count or a percentage of replicas. The defaults are 25%/25%, which quietly means you can run at 75% capacity mid-rollout. Our maxUnavailable: 0 trades speed for safety — the rollout cannot start a step until the previous new Pod is Ready, so it is slower but never dips. Say plainly that step 3 is the one that depends on the readiness probe: without it, "Ready" is a lie and step 4 kills a working Pod for a broken one.

Choosing

RollingUpdate vs Recreate

RollingUpdate (default)

Zero downtime when readiness is honest
Old and new versions run at the same time
Needs backward-compatible APIs and DB schemas
What Pinboard uses

Recreate

Kill everything, then start the new version
Guaranteed downtime — plan a window
Right when versions genuinely cannot coexist: a breaking migration, or a single ReadWriteOnce volume

Rolling updates mean N and N+1 serve traffic together for a minute. Your migrations must be additive — expand, deploy, contract.

Driving it

status · history · undo

kubectl set image deployment/pinboard-api api=pinboard-api:1.1 -n pinboard
kubectl rollout status deployment/pinboard-api -n pinboard   # blocks until done
kubectl get rs -n pinboard                                   # old RS at 0, new at 3

kubectl rollout history deployment/pinboard-api -n pinboard
# REVISION  CHANGE-CAUSE
# 1         initial 1.0
# 2         bump to 1.1

kubectl rollout undo deployment/pinboard-api -n pinboard
kubectl rollout undo deployment/pinboard-api --to-revision=1 -n pinboard
kubectl rollout pause deployment/pinboard-api -n pinboard    # edit safely, then resume

revisionHistoryLimit: 5 keeps five old ReplicaSets — that is exactly how many revisions you can roll back to.

In a pipeline, rollout status is your gate: it exits non-zero when progressDeadlineSeconds (default 600) expires, so a stuck rollout fails the build instead of pretending to succeed. CHANGE-CAUSE comes from the kubernetes.io/change-cause annotation — set it in the manifest or your history is a column of <none>. undo is not a time machine: it just scales the previous ReplicaSet back up, which is why it takes seconds.

Three panes, namespace demo. Pane 2: watch -n 1 kubectl -n demo get deploy,rs,pods — note that get deploy,rs,pods -w is rejected, watch is the workaround. Apply api-deploy-v1.yaml, point at the shared name prefix and the pod-template-hash, then delete one Pod and let the ReplicaSet replace it. Start the in-cluster loop (kubectl -n demo run curlloop --image=busybox:1.37 --rm -it wget-ing http://pinboard-api:8080/api/info), apply api-deploy-v2.yaml, and show the second ReplicaSet, the peak of four Pods, and both themes answering for a few seconds. Finish with rollout history, an annotate … change-cause, and rollout undo.

Probes

The cluster cannot read your logs. Probes are the only way it knows whether your container is alive and whether it is ready.

Three jobs, three probes

liveness · readiness · startup

livenessProbe

Am I broken? On failure the kubelet restarts the container. For deadlocks and wedged processes only.

readinessProbe

Should I get traffic? On failure the Pod is removed from Service endpoints. Nothing is restarted.

startupProbe

Have I booted yet? Holds the other two off until it passes — for slow starters, without loosening liveness.

Handlers: httpGet · tcpSocket · exec · grpc. Pinboard exposes /healthz (liveness) and /readyz (readiness, 503 when the store is unreachable).

The distinction that matters: readiness is about traffic, liveness is about restarts. Getting them backwards is how a database blip turns into a cluster-wide restart storm. Remind them our API's /readyz pings Postgres while /healthz only proves the process is up — that separation is deliberate and is what you want in your own services.

Configuration

Probes on the Pinboard container

periodSeconds × failureThreshold = time to react
Readiness here reacts in ~15 s; liveness in ~30 s
A startupProbe of 2 s × 30 gives 60 s to boot
Prefer a startup probe over a long initialDelaySeconds
port: http refers to the named container port

readinessProbe:        # drop from endpoints
  httpGet:
    path: /readyz
    port: http
  periodSeconds: 5
  failureThreshold: 3
livenessProbe:         # restart the container
  httpGet:
    path: /healthz
    port: http
  periodSeconds: 10
  failureThreshold: 3

Do the arithmetic out loud so the numbers stop being magic: 5 s × 3 = 15 s before a failing Pod stops receiving traffic. The startup probe on the slide is periodSeconds: 2 with failureThreshold: 30. Then the trap — never point a liveness probe at a dependency. If /healthz checked Postgres, a 30-second database hiccup would restart every API Pod simultaneously and turn a small outage into a large one. Full file: labs/solutions/final/20-api-deployment.yaml.

Why this session exists

A rolling update without readiness is an outage

No readinessProbe

"Ready" means the process started. Kubernetes adds the Pod to the Service the instant the container runs — before the HTTP listener is up.

1.0 emerald  api-7f4c
curl: (52) Empty reply from server
curl: (7) Connection refused
1.1 amber    api-9d21

With readinessProbe

The rollout waits for /readyz to answer 200 before it removes an old Pod. With maxUnavailable: 0, capacity never dips.

1.0 emerald  api-7f4c
1.0 emerald  api-7f4c
1.1 amber    api-9d21
1.1 amber    api-9d21

The failure is silent in kubectl: the rollout reports success either way. Only the users see it.

Getting it wrong

Probe pitfalls

Liveness that checks dependencies — a DB blip restarts every replica at once
Thresholds too tight — a GC pause or a slow disk becomes a restart loop
Liveness but no readiness — traffic hits a Pod that is up but not warm
An expensive endpoint — a probe every 5 s × 200 Pods is real load; keep it cheap and local
No startup probe on a slow starter — liveness kills it mid-boot, forever

Sane default: always a readiness probe. Add liveness only when you can name the failure it recovers from.

The last callout is deliberately opinionated and worth defending: a liveness probe can only ever restart a container, so if you cannot describe a hang that a restart fixes, it adds risk without benefit. The slow-starter loop is a classic — the container never finishes booting because liveness kills it at 30 s — and the fix is a startup probe, not a bigger initialDelaySeconds. Session 09 revisits all of this from the troubleshooting side.

Keep the traffic loop from the previous demo running in pane 3. Apply api-deploy-noprobe.yaml first, point the loop at /readyz, and let the room read not ready: FAIL_READY_AFTER elapsed scrolling past while every Pod still reports 1/1 Ready and stays in the EndpointSlice — that is the outage. Restore the probe, then kubectl apply -f api-deploy-broken.yaml and say nothing for ten seconds: pane 3 keeps printing amber. Only then show pane 2 — one new Pod in CrashLoopBackOff, three healthy 1.1 Pods untouched. Diagnose with kubectl -n demo logs -l app.kubernetes.io/name=pinboard-api --tail=1 --prefix, let rollout status --timeout=90s exceed its progress deadline, and finish with kubectl -n demo rollout undo deploy/pinboard-api.

Beyond Deployments

Four more controllers, one scaling command, and the ten seconds your Pod gets to say goodbye.

Pick the right one

Controllers at a glance

Deployment

Stateless, interchangeable replicas. Random names, rolling updates, any Pod can serve any request. pinboard-api, pinboard-web.

DaemonSet

One Pod per node, automatically on new nodes. Log collectors, CNI agents, node exporters. kube-proxy is one.

StatefulSet

Stable names (pinboard-db-0), stable storage per Pod, ordered start and rollout. Session 08.

Job & CronJob

Run to completion, once or on a schedule. Migrations, backups, batch imports — not servers.

Give the two-question decision rule: does it need to run everywhere (DaemonSet), does each replica need its own identity and disk (StatefulSet), does it finish (Job), otherwise Deployment. Note that DaemonSets have their own rolling update strategy with maxUnavailable, and that a StatefulSet updates in reverse ordinal order — highest Pod first — which surprises people the first time.

Run to completion

Jobs and CronJobs for Pinboard

restartPolicy must be OnFailure or Never — never Always
backoffLimit caps retries; ttlSecondsAfterFinished cleans finished Pods up
CronJob wraps a Job template: schedule: "* * * * *", plus concurrencyPolicy: Forbid
Schedules use the cluster's timezone unless you set timeZone

apiVersion: batch/v1
kind: Job
metadata:
  name: pinboard-migrate
  namespace: pinboard
spec:
  backoffLimit: 2
  ttlSecondsAfterFinished: 300
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: postgres:17-alpine

The migration Job is the pattern they will actually need: run it before or alongside the rollout, let backoffLimit stop an endless retry, and let the TTL controller clean up so kubectl get pods is not full of Completed rows. In Lab 06 they write a CronJob heartbeat that fires every minute — a safe way to watch scheduled Pods appear, complete and be cleaned up; once the API has a Service in Lab 07 the same CronJob can probe /readyz instead.

Everyday operations

Scaling and restarting

kubectl scale deployment/pinboard-api --replicas=5 -n pinboard
kubectl get deploy pinboard-api -n pinboard
# NAME           READY   UP-TO-DATE   AVAILABLE   AGE
# pinboard-api   5/5     5            5           12m

kubectl rollout restart deployment/pinboard-api -n pinboard   # new ConfigMap? restart
kubectl scale deployment/pinboard-api --replicas=3 -n pinboard

rollout restart

Stamps a new annotation on the Pod template, so a normal rolling update replaces every Pod — safely, one at a time. The way to pick up a changed ConfigMap or Secret.

Careful with drift

scale changes the live object, not your YAML. The next apply puts replicas back — unless an HPA owns it (Session 09).

READY, UP-TO-DATE and AVAILABLE mean different things and reading them is a skill: UP-TO-DATE counts Pods on the newest template, AVAILABLE counts Pods that have stayed Ready long enough. rollout restart is the honest replacement for kubectl delete pod — same effect, no downtime. Flag the imperative/declarative drift now, because it is the first thing GitOps fixes in Session 10.

Shutting down

terminationGracePeriodSeconds

Pod marked Terminating — removed from endpoints
preStop hook runs, if any
SIGTERM to PID 1 in each container
Grace period counts down (default 30 s)
Still alive? SIGKILL — no cleanup, dropped connections

Steps 1 and 3 are concurrent. A short preStop: sleep 5 lets endpoint removal propagate before you stop accepting requests.

spec:
  terminationGracePeriodSeconds: 15
  containers:
    - name: api
      image: pinboard-api:1.1
      # SHUTDOWN_GRACE=10s — the API drains
      # in-flight requests, then exits 0

Session 03's exec-form lesson pays off here: shell form means PID 1 is /bin/sh and SIGTERM never reaches your process.

This closes the zero-downtime story from the other end: readiness gets traffic to healthy Pods, graceful termination gets traffic safely off dying ones. Make the arithmetic explicit — our grace period is 15 s and the app drains in 10 s, so it always exits cleanly and never sees SIGKILL. Remind them where the exec-form trap comes from and that kubectl delete pod --grace-period=0 --force is a last resort, not a habit.

Point them at the two panes they need: a curl loop against the port-forwarded Service and kubectl get pods -w. Tell them to do step 7 properly: re-run the rollout from 20-api-deployment-noready.yaml (readiness probe removed, FAIL_READY_AFTER=15) and watch it go green in twenty seconds while every Pod answers 503 on /readyz — then apply the manifest with the probe back and watch the same change stall instead. That contrast is the whole lesson. Remind them to kind load docker-image pinboard-api:1.1 and pinboard-api:1.2-broken before starting, or every step ends in ImagePullBackOff.

Session 06

Key takeaways

Labels are the wiring

Selectors are how controllers and Services find Pods. Use the app.kubernetes.io/* set and never select on version.

A Deployment plays ReplicaSets

One per template hash — which is why maxSurge/maxUnavailable tune the rollout and undo takes seconds.

Readiness is the contract

Without it a rolling update is an outage. Liveness restarts, readiness routes, startup buys boot time.

Next up

Session 07 — Services, cluster DNS and Ingress: how anyone reaches these Pods in the first place.

Tie the session back to its promise: we now have self-healing, safe releases and instant rollback for Pinboard. Then name the gap deliberately — we have been reaching the Pods through port-forward, which is a debugging tool, and the Pod IPs change on every rollout. That is exactly the problem Services solve next session. Ask them to leave the Deployment running.

Questions?

Up next — Session 07: Networking — Services, DNS, Ingress. Keep the Deployment running.

Workloads: Deployments, rollouts and probes

What we'll cover today

1 · Labels, selectors, controllers

2 · Rollouts & probes

3 · Other controllers

4 · Lab 06

Labels, selectors, controllers

Labels — and what they are not

Selectors: equality and set-based

The recommended labels

ReplicaSet: keep N Pods alive

The self-healing you were promised

Deployment = ReplicaSets over time

Deployment

ReplicaSet 1.0

ReplicaSet 1.1

Pods

The Pinboard API Deployment

Rollouts

How a rolling update actually steps

3 × 1.0

+1 surge

wait Ready

−1 old

3 × 1.1

RollingUpdate vs Recreate

RollingUpdate (default)

Recreate

status · history · undo

Ship 1.0 → 1.1 without dropping a request

Probes

liveness · readiness · startup

livenessProbe

readinessProbe

startupProbe

Probes on the Pinboard container

A rolling update without readiness is an outage

No readinessProbe

With readinessProbe

Probe pitfalls

Break it, watch it, undo it

Beyond Deployments

Controllers at a glance

Deployment

DaemonSet

StatefulSet

Job & CronJob

Jobs and CronJobs for Pinboard

Scaling and restarting

rollout restart

Careful with drift

terminationGracePeriodSeconds

Make the Pinboard API resilient

Key takeaways

Labels are the wiring

A Deployment plays ReplicaSets

Readiness is the contract

Next up

Questions?