Session 09 demos – observability, troubleshooting, autoscaling, security basics

Demos 1–6 belong to Block A (metrics, the troubleshooting method, kubectl debug), Demos 7–11 to Block B (HPA, PDB, quotas, RBAC, PSA). Everything runs in the demo namespace of the pinboard kind cluster. Pick from 6, 10 and 11 if you are short on time; do not skip 2–4, they are the heart of the session. Files next to this one:

File	Used in
`hpa.yaml`	Demo 7 — `autoscaling/v2`, cpu 50 %, 1–6 replicas
`load-generator.yaml`	Demo 7 — hammers `/api/burn` through the Service
`pdb.yaml`	Demo 8 — `minAvailable: 2` vs `kubectl drain`
`quota.yaml`	Demo 9 — namespace ceiling, and the rejection it causes
`limitrange.yaml`	Demo 9 — the defaults that make the quota usable
`rbac-reader.yaml`	Demo 10 — ServiceAccount + Role + RoleBinding + a Pod using it
`pod-psa-violation.yaml`	Demo 11 — a root Pod for `restricted` to refuse

Before the session

Namespace, images, workload:

cd ~/docker-kubernetes-training/demos/session09

docker pull busybox:1.37 && kind load docker-image busybox:1.37 --name pinboard
kind load docker-image pinboard-api:1.0 --name pinboard
kubectl create namespace demo --dry-run=client -o yaml | kubectl apply -f -
kubectl config set-context --current --namespace=demo
kubectl apply -f ../session06/api-deploy-v1.yaml     # Deployment + Service, 3 replicas
kubectl -n demo rollout status deploy/pinboard-api

Install metrics-server now (Demo 1 shows the commands, but kubectl top needs ~60 s of scraped data before it answers; installing early avoids dead air). If you prefer to install it live, do Demo 1 first and Demo 2 while it warms up.
Optional: k9s version — if it is not installed, skip Demo 6 rather than installing it in front of the room.
Three panes: 1 for typing, 2 watch -n 1 kubectl -n demo get pods, 3 free (used for kubectl get events -w and the HPA watch).

Demo 1: metrics-server and `kubectl top` (≈ 6 min)

What to show / say

kubectl top is not built in. It is an API served by an add-on, and on kind it needs one flag — which is a useful lesson about kubelet TLS.

Steps

Show the failure first, so the fix means something:

kubectl top nodes

error: Metrics API not available

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
kubectl -n kube-system rollout status deploy/metrics-server --timeout=60s

Waiting for deployment "metrics-server" rollout to finish: 0 of 1 updated replicas are available...
error: timed out waiting for the condition

👉 it will not become Ready. Diagnose it with the method the session is about:

kubectl -n kube-system logs deploy/metrics-server --tail=3

E0512 09:22:14.331  scraper.go:149] "Failed to scrape node" err="Get \"https://172.18.0.3:10250/metrics/resource\": tls: failed to verify certificate: x509: cannot validate certificate for 172.18.0.3 because it doesn't contain any IP SANs" node="pinboard-worker"

👉 kind’s kubelets serve self-signed certificates. On a managed cluster this is handled for you; here you tell metrics-server to skip verification:

kubectl -n kube-system patch deployment metrics-server --type=json \
  -p '[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]'
kubectl -n kube-system rollout status deploy/metrics-server

deployment "metrics-server" successfully rolled out

👉 say the quiet part: --kubelet-insecure-tls is fine for a laptop cluster and is not what you do in production; there you fix the kubelet certificates.

Wait ~60 s for the first scrape, then:

kubectl top nodes
kubectl top pods -n demo

NAME                     CPU(cores)   CPU(%)   MEMORY(bytes)   MEMORY(%)
pinboard-control-plane   224m         2%       842Mi           10%
pinboard-worker          61m          0%       412Mi           5%
pinboard-worker2         55m          0%       398Mi           5%

NAME                            CPU(cores)   MEMORY(bytes)
pinboard-api-7d9c8f6b45-4kq2r   1m           9Mi
pinboard-api-7d9c8f6b45-9xnzt   1m           9Mi
pinboard-api-7d9c8f6b45-vc7hd   1m           9Mi

👉 9 MiB for a whole Go API — that is the distroless image from Session 02. 👉 and the warning: metrics-server keeps ~1 minute of data in memory. It exists to feed the HPA, not to be your monitoring. That job belongs to Prometheus, Loki/ELK for logs, OpenTelemetry for traces.

Demo 2: The method — drill 1, CrashLoopBackOff (≈ 6 min)

What to show / say

Type the five steps in the same order every single time: get → describe → logs → events → exec/debug. Narrate the order out loud; the method is the deliverable, the bug is just an excuse.

Steps

Break it:

kubectl -n demo set env deploy/pinboard-api CRASH_ON_START=true

get — what is the state, and how long has it been that way?
```
kubectl -n demo get pods
```
```
NAME                             READY   STATUS             RESTARTS      AGE
pinboard-api-6d4b7f8c95-lm2v7    0/1     CrashLoopBackOff   3 (28s ago)   84s
pinboard-api-7d9c8f6b45-4kq2r    1/1     Running            0             11m
...
```
👉 CrashLoopBackOff is not an error, it is a back-off timer: the container exits, kubelet waits 10 s, 20 s, 40 s … up to 5 min before trying again. 👉 RESTARTS 3 with AGE 84s — it is restarting, so the image pulled fine and the scheduler was happy. That already rules out half the failure modes.

describe — the Pod’s own account of what happened:

kubectl -n demo describe pod -l app.kubernetes.io/name=pinboard-api | grep -A12 'Last State'

Last State:     Terminated
  Reason:       Error
  Exit Code:    1
  Started:      Tue, 12 May 2026 09:31:41 +0000
  Finished:     Tue, 12 May 2026 09:31:41 +0000

👉 exit code 1 = the application decided to quit. (137 = SIGKILL/OOM, 143 = SIGTERM, 127 = command not found — worth putting on the whiteboard.)

logs — the application’s account:

kubectl -n demo logs -l app.kubernetes.io/name=pinboard-api --tail=1 --prefix | head -2

[pod/pinboard-api-6d4b7f8c95-lm2v7/api] level=ERROR msg="CRASH_ON_START is set — exiting with status 1 (this is on purpose)"

👉 for a crash-looping Pod kubectl logs returns the last terminated container; --previous gets the one before that.

events — the cluster’s account:

kubectl -n demo get events --sort-by=.lastTimestamp | tail -5

2m    Normal    Pulled      pod/pinboard-api-6d4b7f8c95-lm2v7   Container image "pinboard-api:1.0" already present on machine
90s   Warning   BackOff     pod/pinboard-api-6d4b7f8c95-lm2v7   Back-off restarting failed container api

👉 --sort-by=.lastTimestamp because the default order is useless. Events expire after one hour — an empty event list means “nothing recent”, not “nothing happened”.

Fix, and confirm:
```
kubectl -n demo set env deploy/pinboard-api CRASH_ON_START-
kubectl -n demo rollout status deploy/pinboard-api
```
👉 the trailing - on set env removes a variable. And 👉 point at pane 2: the three healthy 1.0 Pods never went away — maxUnavailable: 0 again.

Demo 3: Drill 2 — OOMKilled (≈ 5 min)

What to show / say

The failure that looks like a crash but is not the application’s fault. Same five steps, different answer at step 3.

Steps

kubectl -n demo set resources deploy/pinboard-api --limits=memory=16Mi
kubectl -n demo get pods

NAME                             READY   STATUS      RESTARTS      AGE
pinboard-api-59fbb7c4d8-h4x7d    0/1     OOMKilled   2 (14s ago)   42s

If it survives on your machine, drop to --limits=memory=8Mi. The Go runtime reserves its heap arenas up front, so the exact tipping point varies.

```
kubectl -n demo describe pod -l app.kubernetes.io/name=pinboard-api | grep -A6 'Last State'
```
```
Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137
```
👉 137 = 128 + 9 = killed by SIGKILL. Nothing to find in the logs — the process was not asked to stop, it was shot. Say it: “the kernel’s cgroup OOM killer did this, because the container crossed its memory limit. Requests schedule; limits kill.”
```
kubectl -n demo logs -l app.kubernetes.io/name=pinboard-api --tail=2 --prefix | head -2
```
👉 the last lines are perfectly normal startup logs. That absence is the signature of an OOM kill.

kubectl -n demo set resources deploy/pinboard-api --limits=memory=64Mi
kubectl -n demo rollout status deploy/pinboard-api
kubectl top pods -n demo

👉 close the loop: real usage is ~9 MiB, so 64 MiB is a sane limit and 16 was not. kubectl top is how you pick these numbers instead of guessing.

Demo 4: Drill 3 — ImagePullBackOff (≈ 4 min)

What to show / say

The failure where logs has nothing to give you, because no container ever existed. This is where describe and events earn their keep.

Steps

kubectl -n demo set image deploy/pinboard-api api=pinboard-api:9.9
kubectl -n demo get pods

NAME                             READY   STATUS             RESTARTS   AGE
pinboard-api-7cc9f6b5d4-t9wxq    0/1     ImagePullBackOff   0          35s

kubectl -n demo logs -l app.kubernetes.io/name=pinboard-api --tail=1 --prefix | head -1

Error from server (BadRequest): container "api" in pod "pinboard-api-7cc9f6b5d4-t9wxq" is waiting to start: trying and failing to pull image

👉 no logs, ever — step 3 of the method is a dead end here. Go to step 4.

kubectl -n demo describe pod -l app.kubernetes.io/name=pinboard-api | grep -B2 -A6 Failed

Warning  Failed     kubelet  Failed to pull image "pinboard-api:9.9": failed to pull and unpack image
  "docker.io/library/pinboard-api:9.9": failed to resolve reference: pull access denied, repository does not exist or may require authorization

👉 read the resolved name aloud: docker.io/library/pinboard-api:9.9. A bare name always means Docker Hub — this is the Session 01 slide coming back. The three real causes: typo in the tag, image never loaded into kind (kind load docker-image), or a private registry with no imagePullSecret.

kubectl -n demo set image deploy/pinboard-api api=pinboard-api:1.0
kubectl -n demo rollout status deploy/pinboard-api

Demo 5: `kubectl debug` — a shell where there is no shell (≈ 6 min)

What to show / say

The API image is distroless: no shell, no ps, no curl. That is correct and it is inconvenient. Ephemeral containers solve it without changing the image.

Steps

Show the problem:

POD=$(kubectl -n demo get pod -l app.kubernetes.io/name=pinboard-api -o name | head -1)
kubectl -n demo exec -it "$POD" -- sh

OCI runtime exec failed: exec failed: unable to start container process: exec: "sh": executable file not found in $PATH

Attach a debug container to the running Pod, sharing the API container’s process namespace:
```
kubectl -n demo debug -it "$POD" --image=busybox:1.37 --target=api -- sh
```
Inside:
```
ps aux
```
```
PID   USER     COMMAND
1     65532    /pinboard-api
14    root     sh
```
👉 --target=api is what makes PID 1 of the other container visible. Without it you get a container in the same Pod (same network, same volumes) but its own process namespace.
```
wget -qO- http://localhost:8080/api/info
nslookup pinboard-api
```
```
{"app":"pinboard-api","greeting":"Welcome to Pinboard","hostname":"pinboard-api-7d9c8f6b45-4kq2r",...}
```
👉 localhost reaches the API because containers in a Pod share one network namespace. This is how you prove “is it the app or is it the network?” in thirty seconds. exit when done.
Show where it went — it is on the Pod, and it cannot be removed:
```
kubectl -n demo get "$POD" -o jsonpath='{.spec.ephemeralContainers[*].name}{"\n"}'
```
```
debugger-x8k2p
```
👉 ephemeral containers cannot be restarted or deleted; they go away when the Pod does. They also have no resource requests, so they do not affect scheduling.
The node-level version, for “is the node sick?”:
```
kubectl debug node/pinboard-worker -it --image=busybox:1.37 -- sh
```
Inside:
```
ls /host/var/log/containers | head -3
chroot /host df -h /
```
👉 this Pod runs in the host namespaces with the node’s filesystem mounted at /host — disk pressure, kubelet logs, dmesg. It is a privileged Pod created in your current namespace, named node-debugger-<node>-xxxxx: powerful, audited, and something you should RBAC away from most people. exit does not delete it, so clean it up by name:
```
kubectl -n demo get pods | grep node-debugger
kubectl -n demo delete pod node-debugger-pinboard-worker-x8k2p   # the real name
```

Demo 6: Events and k9s (optional, ≈ 4 min)

What to show / say

Two ways to keep the whole picture on screen instead of typing get pods forever.

Steps

In pane 3, leave this running whenever you demo anything:
```
kubectl -n demo get events -w
```
The sorted, cluster-wide version when something happened while you were away:
```
kubectl get events -A --sort-by=.lastTimestamp | tail -10
```
If k9s is installed:
```
k9s -n demo
```
👉 30 seconds, no more: :pods, l for logs, d for describe, s for shell, Ctrl-D to delete, : to jump anywhere. Say what it is: a keyboard-driven front end over the same API, doing nothing you cannot do with kubectl. 👉 name the alternatives once — Headlamp, the Lens-likes, kubectl plugins via krew — and move on.

Demo 7: HPA with a load generator (≈ 8 min)

What to show / say

Autoscaling is a controller reading one number and doing arithmetic. Show the number, show the arithmetic, then show the Pods.

Steps

kubectl apply -f hpa.yaml
kubectl -n demo get hpa

NAME           REFERENCE                 TARGETS       MINPODS   MAXPODS   REPLICAS   AGE
pinboard-api   Deployment/pinboard-api   cpu: 2%/50%   1         6         3          15s

👉 2%/50% — 1m of CPU against a 50m request. If this column says <unknown>, metrics-server is down or the container has no requests.cpu.

In pane 3, watch it live:
```
kubectl -n demo get hpa,pods -w
```
(or watch -n 2 kubectl -n demo get hpa,deploy)
Apply load:
```
kubectl apply -f load-generator.yaml
```
👉 within ~30 s the TARGETS column climbs past 50 % and replicas follow:
```
NAME           REFERENCE                 TARGETS         REPLICAS
pinboard-api   Deployment/pinboard-api   cpu: 612%/50%   3
pinboard-api   Deployment/pinboard-api   cpu: 612%/50%   6
```
👉 do the arithmetic out loud: ceil(3 × 612 / 50) = 37, capped at maxReplicas: 6. The HPA is not gentle — it jumps straight to what the formula says.

The controller narrates itself:

kubectl -n demo describe hpa pinboard-api | tail -8

Events:
  Type    Reason             Age   From                       Message
  Normal  SuccessfulRescale  62s   horizontal-pod-autoscaler  New size: 6; reason: cpu resource utilization (percentage of request) above target

Remove the load and wait:
```
kubectl -n demo scale deploy/load-generator --replicas=0
```
👉 scale-down is deliberately slow: the stabilizationWindowSeconds: 60 in hpa.yaml (default 300) plus the percent policy. Say why — flapping between 3 and 6 replicas every minute is worse than being briefly over-provisioned. 👉 while waiting, name the other two: VPA changes requests instead of replica counts (and fights the HPA on CPU — don’t combine them on the same metric), and the cluster autoscaler / Karpenter adds nodes when Pods stay Pending. Three different problems.

Tidy up before Demo 8, which needs a stable 3 replicas:

kubectl delete -f hpa.yaml -f load-generator.yaml
kubectl -n demo scale deploy/pinboard-api --replicas=3

Demo 8: PodDisruptionBudget and `kubectl drain` (≈ 6 min)

What to show / say

Maintenance is the disruption you can control. A PDB is the contract between the person draining a node and the app owner.

Steps

kubectl -n demo get pods -o wide | awk '{print $1, $7}'

NAME                            NODE
pinboard-api-7d9c8f6b45-4kq2r   pinboard-worker
pinboard-api-7d9c8f6b45-9xnzt   pinboard-worker2
pinboard-api-7d9c8f6b45-vc7hd   pinboard-worker

Drain a node without a PDB first, to show the baseline:

kubectl drain pinboard-worker --ignore-daemonsets --delete-emptydir-data

node/pinboard-worker cordoned
evicting pod demo/pinboard-api-7d9c8f6b45-4kq2r
evicting pod demo/pinboard-api-7d9c8f6b45-vc7hd
node/pinboard-worker drained

👉 both Pods went at once, and for a moment only one replica was serving. Nothing stopped it. cordoned = the node is marked unschedulable; drain = cordon + evict.

```
kubectl uncordon pinboard-worker
kubectl -n demo get pods -o wide | awk '{print $1, $7}'
```
👉 the replacements landed on worker2 and stay there — uncordoning does not move anything back. Rebalancing is not automatic.

Now the contract:

kubectl apply -f pdb.yaml
kubectl -n demo get pdb

NAME           MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
pinboard-api   2               N/A               1                     5s

👉 ALLOWED DISRUPTIONS: 1 — that is the whole object in one number: 3 healthy Pods, keep 2, so one may go at a time.

Drain the node that now holds everything:

kubectl drain pinboard-worker2 --ignore-daemonsets --delete-emptydir-data

evicting pod demo/pinboard-api-7d9c8f6b45-8plkw
error when evicting pods/"pinboard-api-7d9c8f6b45-2n4gq" -n "demo" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
evicting pod demo/pinboard-api-7d9c8f6b45-2n4gq
node/pinboard-worker2 drained

👉 that retry line is the demo. The eviction API said no, drain waited for a replacement Pod to become Ready elsewhere, then took the next one. The drain still finished — it just took the safe path.

The failure mode worth warning about:

kubectl -n demo patch pdb pinboard-api -p '{"spec":{"minAvailable":3}}'
kubectl uncordon pinboard-worker2
kubectl drain pinboard-worker --ignore-daemonsets --delete-emptydir-data --timeout=30s

error when evicting pods/"..." (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.
...
There are pending pods in node "pinboard-worker" when an error occurred: drain did not complete within 30s

👉 minAvailable equal to the replica count means no Pod may ever be evicted: the drain hangs forever and your node upgrade stalls at 3 a.m. Rule of thumb: minAvailable: replicas - 1, or maxUnavailable: 1.

kubectl -n demo delete -f pdb.yaml
kubectl uncordon pinboard-worker pinboard-worker2
kubectl get nodes

NAME                     STATUS   ROLES           AGE   VERSION
pinboard-control-plane   Ready    control-plane   4h    v1.33.1
pinboard-worker          Ready    <none>          4h    v1.33.1
pinboard-worker2         Ready    <none>          4h    v1.33.1

👉 every node Ready with no SchedulingDisabled — always finish a drain demo here, or the next demo will mysteriously not schedule.

Demo 9: ResourceQuota and LimitRange (≈ 5 min)

What to show / say

Two objects that stop one team from eating a cluster — and the interaction between them that everybody trips over once.

Steps

kubectl apply -f quota.yaml
kubectl -n demo describe quota demo-quota

Name:                   demo-quota
Resource                Used   Hard
--------                ----   ----
count/deployments.apps  1      5
limits.memory           192Mi  1Gi
pods                    3      6
requests.cpu            150m   1
requests.memory         96Mi   512Mi

👉 existing Pods are counted, never retro-actively evicted. A quota applies to what you create next.

The surprise:

kubectl -n demo run nolimits --image=busybox:1.37 --restart=Never -- sleep 60

Error from server (Forbidden): pods "nolimits" is forbidden: failed quota: demo-quota:
must specify limits.memory for: nolimits; requests.cpu for: nolimits; requests.memory for: nolimits

👉 the moment a quota limits a resource, every container must declare it. Suddenly nobody in the namespace can kubectl run anything. This is where teams file the “Kubernetes is broken” ticket.

The cure:

kubectl apply -f limitrange.yaml
kubectl -n demo run nolimits --image=busybox:1.37 --restart=Never -- sleep 60
kubectl -n demo get pod nolimits -o jsonpath='{.spec.containers[0].resources}{"\n"}'

pod/nolimits created
{"limits":{"cpu":"200m","memory":"128Mi"},"requests":{"cpu":"50m","memory":"32Mi"}}

👉 the exact same command now works: the LimitRange admission plugin injected default and defaultRequest before the quota was evaluated. Nobody edited a manifest.

And the bounds:

kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: toobig
  namespace: demo
spec:
  containers:
    - name: greedy
      image: busybox:1.37
      command: ["sleep", "60"]
      resources:
        requests: {cpu: 900m, memory: 32Mi}
        limits: {cpu: 900m, memory: 128Mi}
EOF

Error from server (Forbidden): error when creating "STDIN": pods "toobig" is forbidden:
maximum cpu usage per Container is 500m, but limit is 900m

👉 rejected by the LimitRange (max), not the quota — read the message and name which object refused. Two different admission plugins, two different messages.

kubectl -n demo delete pod nolimits --ignore-not-found
kubectl delete -f quota.yaml -f limitrange.yaml

Demo 10: RBAC in five minutes (≈ 5 min)

What to show / say

Three objects and one command. kubectl auth can-i --as answers RBAC questions without creating a single user.

Steps

kubectl apply -f rbac-reader.yaml
kubectl -n demo get sa,role,rolebinding,pod reader-shell

Ask the API server what that identity may do — impersonation, no login:

SA=system:serviceaccount:demo:pinboard-reader
kubectl auth can-i list pods       -n demo    --as=$SA
kubectl auth can-i get secrets     -n demo    --as=$SA
kubectl auth can-i delete pods     -n demo    --as=$SA
kubectl auth can-i list pods       -n default --as=$SA

yes
no
no
no

👉 four questions, four answers, no trial and error. The last one is the point of a Role: the grant stops at the namespace boundary. A ClusterRole + ClusterRoleBinding would not.

The whole permission set at once:

kubectl auth can-i --list -n demo --as=$SA

Resources                  Non-Resource URLs   Resource Names   Verbs
pods                       []                  []               [get list watch]
pods/log                   []                  []               [get list watch]
services                   []                  []               [get list watch]
configmaps                 []                  []               [get list watch]
deployments.apps           []                  []               [get list watch]
...

👉 RBAC is additive only — there is no deny verb. Something is forbidden precisely because no binding allows it.

How a Pod gets that identity:
```
kubectl -n demo exec reader-shell -- ls /var/run/secrets/kubernetes.io/serviceaccount/
kubectl -n demo exec reader-shell -- wc -c /var/run/secrets/kubernetes.io/serviceaccount/token
```
```
ca.crt
namespace
token
1042 /var/run/secrets/kubernetes.io/serviceaccount/token
```
👉 a projected, short-lived, audience-bound JWT that the kubelet rotates — not a Secret object any more. Every Pod gets the default ServiceAccount unless you name one, and default should have no permissions at all. Set automountServiceAccountToken: false on Pods that never call the API.
```
kubectl delete -f rbac-reader.yaml
```

Demo 11: Pod Security Admission (≈ 4 min)

What to show / say

Two labels on a namespace, enforced by the API server itself. No operator, no webhook, no CRD.

Steps

Show that the Pod is fine in an unlabelled namespace:

kubectl apply -f pod-psa-violation.yaml
kubectl -n demo logs rootpod
kubectl -n demo delete pod rootpod

uid=0(root) gid=0(root) groups=0(root)

👉 root inside the container, by default, with no complaint from anybody.

Turn the namespace on — the same labels the course’s 00-namespace.yaml carries:

kubectl label namespace demo \
  pod-security.kubernetes.io/enforce=restricted \
  pod-security.kubernetes.io/warn=restricted --overwrite

kubectl apply -f pod-psa-violation.yaml

Error from server (Forbidden): error when creating "pod-psa-violation.yaml": pods "rootpod" is forbidden:
violates PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "root" must set
securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "root" must set
securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true, seccompProfile

👉 read the four violations aloud — the message is a checklist of the restricted profile, and it happened at the API server, before scheduling. 👉 the three levels: privileged (anything), baseline (blocks the obviously dangerous), restricted (non-root, no privilege escalation, drop ALL, seccomp). And the three modes: enforce, audit, warn — roll out with warn+audit first, then flip enforce.

And the app that was built for it:
```
kubectl -n demo rollout restart deploy/pinboard-api
kubectl -n demo rollout status deploy/pinboard-api
```
```
deployment "pinboard-api" successfully rolled out
```
👉 api-deploy-v1.yaml already has runAsNonRoot, seccompProfile, allowPrivilegeEscalation: false, drop: ["ALL"] and a distroless non-root image, so nothing changes for it. That is the whole argument for Session 02’s base-image discipline, arriving seven sessions later.

Optional: check any manifest before it hits a strict namespace:

kubectl -n demo apply -f pod-psa-violation.yaml --dry-run=server

Reset

Lab 09 installs metrics-server itself in the students’ own cluster and works in pinboard; leave metrics-server in place here, but return demo to empty and remove the PSA labels (Session 10 applies plain manifests to this namespace).

kubectl -n demo delete -f hpa.yaml -f load-generator.yaml -f pdb.yaml \
  -f quota.yaml -f limitrange.yaml -f rbac-reader.yaml -f pod-psa-violation.yaml --ignore-not-found
kubectl -n demo delete pod nolimits toobig --ignore-not-found
kubectl delete -f ../session06/api-deploy-v1.yaml --ignore-not-found
kubectl -n demo get pods | grep node-debugger    # delete any leftovers by name
kubectl label namespace demo \
  pod-security.kubernetes.io/enforce- pod-security.kubernetes.io/warn- 2>/dev/null
kubectl uncordon pinboard-worker pinboard-worker2
kubectl -n demo get all
kubectl get nodes

No resources found in demo namespace.
NAME                     STATUS   ROLES           AGE   VERSION
pinboard-control-plane   Ready    control-plane   4h    v1.33.1
pinboard-worker          Ready    <none>          4h    v1.33.1
pinboard-worker2         Ready    <none>          4h    v1.33.1

kubectl config set-context --current --namespace=default

Session 09 demos – observability, troubleshooting, autoscaling, security basics

Before the session

Demo 1: metrics-server and kubectl top (≈ 6 min)

What to show / say

Steps

Demo 2: The method — drill 1, CrashLoopBackOff (≈ 6 min)

What to show / say

Steps

Demo 3: Drill 2 — OOMKilled (≈ 5 min)

What to show / say

Steps

Demo 4: Drill 3 — ImagePullBackOff (≈ 4 min)

What to show / say

Steps

Demo 5: kubectl debug — a shell where there is no shell (≈ 6 min)

What to show / say

Steps

Demo 6: Events and k9s (optional, ≈ 4 min)

What to show / say

Steps

Demo 7: HPA with a load generator (≈ 8 min)

What to show / say

Steps

Demo 8: PodDisruptionBudget and kubectl drain (≈ 6 min)

What to show / say

Steps

Demo 9: ResourceQuota and LimitRange (≈ 5 min)

What to show / say

Steps

Demo 10: RBAC in five minutes (≈ 5 min)

What to show / say

Steps

Demo 11: Pod Security Admission (≈ 4 min)

What to show / say

Steps

Reset

Demo 1: metrics-server and `kubectl top` (≈ 6 min)

Demo 5: `kubectl debug` — a shell where there is no shell (≈ 6 min)

Demo 8: PodDisruptionBudget and `kubectl drain` (≈ 6 min)