Session 05 · 2h
Kubernetes: architecture , kubectl, Pods
From one machine running Compose to a cluster that keeps Pinboard running on its own — and the one command you drive it with.
Why orchestration
Cluster architecture
API & kubectl
Pods in depth
Lab 05
Session 05
What we'll cover today
1 · Cluster architecture
Why orchestration, what Kubernetes is and isn't, and every process on the control plane and the nodes.
2 · The API & kubectl
The object model, desired state and reconciliation, kubectl grammar, kubeconfig, contexts and namespaces.
3 · Pods in depth
Shared network and volumes, lifecycle and status, requests/limits and QoS, init and sidecar containers.
4 · Lab 05
Create the kind cluster, load pinboard-api:1.0 and run it as a Pod in the pinboard namespace.
Frame the session as the pivot of the course: four sessions of containers on one machine, now a cluster. Warn that this is the heaviest theory day — architecture and the object model have to land before anything else makes sense — and that block B and the lab are entirely hands-on. Check the room: everyone needs Docker running and the images from Lab 02 still present.
01
Why Kubernetes
Compose gave us one host. Production needs many — and something that watches them for us.
Ten minutes on motivation before any component names. Ask the room who has run containers in production and what broke — the answers will be exactly the six cards coming up.
Motivation
What Compose could not do
Many hosts Compose schedules onto this machine. Nothing spreads Pinboard over a fleet.
Self-healing If the host dies, the containers die with it. Nobody restarts them elsewhere.
Rolling updates up -d replaces containers all at once — a visible gap in service.
Scaling --scale api=3 is manual and capped by one machine's CPU.
Secrets & config A .env file on a laptop is not a secret management story.
Traffic routing No load balancing, no service discovery beyond a single bridge network.
Anchor this in Lab 04: we ran Pinboard with --scale api=3 and nginx round-robined between the hostnames. Ask the room what happens when that laptop reboots — nothing brings the stack back, and the notes were gone. Every card here is a feature we will build over the next five sessions. Stress that Kubernetes is not "Compose, but bigger": it is a control system that continuously compares what you asked for with what exists.
Definition
What Kubernetes is — and isn't
A container orchestrator : it places containers on machines and keeps them there
A declarative API you POST desired state to
A set of controllers that reconcile reality toward that state
Extensible — CRDs, operators, CNI/CSI/CRI plugins
A PaaS — it does not build your code or run a CI pipeline
A container runtime — that is containerd, via the CRI
A database, a message bus or a logging stack
Free of operations — it moves the work, it does not delete it
Kubernetes 1.33+ is the baseline for this course. The project ships three minor releases a year and supports the last three.
The "is not" column prevents the most common disappointment: people expect git push to deploy. Kubernetes gives you the primitives; Helm, Argo CD and your pipeline give you the workflow. Mention that Kubernetes graduated from CNCF in 2018 and is now the de-facto standard, which is why the API has stayed remarkably stable — the manifests you write today will still apply in three years.
Where it runs
Local clusters vs managed clusters
Local — for learning & CI Managed — for production
kind — nodes are Docker containers; multi-node, fast, disposable. Our choice. AKS · EKS · GKE — the cloud runs and upgrades the control plane for you.minikube — VM or Docker driver, rich addonsYou still own nodes, upgrades, quotas and cost k3d (k3s in Docker) · Docker Desktop single-nodeReal LoadBalancers, real StorageClasses, cloud IAM
The API is identical everywhere. What changes is what a LoadBalancer Service or a PersistentVolumeClaim resolves to.
Say why kind wins for a classroom: kind create cluster takes about 40 seconds, gives us three nodes so scheduling is visible, and kind load docker-image means no registry credentials on a closed network. Warn that Docker Desktop's single node hides scheduling entirely. Promise that everything they write here applies unchanged on AKS/EKS/GKE — we will call out the three places where it does not, in Sessions 07, 08 and 10.
02
Cluster architecture
A control plane that decides, and nodes that do. Nine processes, and you only ever talk to one of them.
Tell them up front that they will not be asked to install any of this — but that every troubleshooting story later in the course names one of these processes, so the map is worth twelve minutes now.
The brain
Control plane components
kube-apiserver The only door into the cluster. REST over HTTPS, authn/authz, admission, validation.
etcd Consistent key-value store — the single source of truth. Back it up to rebuild.
kube-scheduler Watches for Pods with no node, scores candidates, writes the binding back to the API.
kube-controller-manager Dozens of control loops in one binary — Deployment, Node, Job…
cloud-controller-manager Cloud-specific loops: LoadBalancers, routes, node lifecycle. Not on kind.
CoreDNS (add-on) Cluster DNS. Not strictly control plane, but every cluster has it — Session 07.
Draw the arrows verbally: nothing ever writes to etcd directly, only the apiserver does. That single choke point is what makes RBAC, auditing and admission control possible. The scheduler and the controllers are just clients that watch and write, exactly like kubectl — which is why you could write your own. On kind there is one control-plane node; managed clusters run three or more and you never see them.
The muscle
Node components
kubelet The node agent. Watches the API for Pods bound to its node, asks the runtime to start them, runs probes, reports status back.
kube-proxy Programs iptables/IPVS (or is replaced by an eBPF CNI) so Service IPs reach Pod IPs.
Container runtime containerd via the CRI , using runc and OCI images — the same images you built in Session 02.
Dockershim was removed in Kubernetes 1.24. "Kubernetes dropped Docker" only ever meant the shim — your Docker-built OCI images run unchanged on containerd.
This is the slide infra people came for. Walk the chain: kubelet → CRI (gRPC) → containerd → runc → namespaces and cgroups, which is exactly the Session 01 picture with a new caller on top. Kill the dockershim myth explicitly, because it still costs teams days. Mention CRI-O as the other common runtime, and that on kind the "nodes" are Docker containers each running their own containerd.
Follow the request
What happens when you apply a Pod
01
kubectl apply YAML → JSON, POST to the apiserver
→
02
apiserver Authn, authz, admission, validate
→
03
etcd Object persisted, nodeName empty
→
04
scheduler Picks a node, writes the binding
→
05
kubelet Pulls image, CRI starts it, reports Running
Every arrow is a watch on the API, not a direct call. No component ever phones another — they all react to the same store.
This flow explains almost every failure mode we will meet later. Stuck at step 4 means Pending — no node fits the requests or a taint blocks it. Stuck at step 5 means ContainerCreating or ImagePullBackOff. Emphasise the level-triggered, watch-based design: components are decoupled, so you can restart the scheduler mid-flight and nothing is lost.
Build a cluster and look inside it
Create the three-node course cluster with kind, then prove that the control plane is itself just Pods running on Kubernetes.
What you'll see · nodes, control-plane Pods, containerd as the runtimeFiles · labs/kind-config.yamlDuration · 8 min
Type: kind create cluster --name pinboard --config labs/kind-config.yaml, then kubectl get nodes -o wide — point at the CONTAINER-RUNTIME column reading containerd://… and at the two workers. Then kubectl get pods -A and read the kube-system list out loud: apiserver, etcd, scheduler, controller-manager, coredns, kindnet, kube-proxy — the slide you just showed, alive. Finish with docker ps to reveal that the "nodes" are containers on the laptop.
03
The API and kubectl
Every object has the same four fields. Every command has the same three words.
This block is the one that turns Kubernetes from intimidating to learnable. Promise them that once they know the four fields and the kubectl grammar, every new resource type in Sessions 07–10 is just more of the same shape.
Object model
Four fields, every single time
apiVersion — group and version, e.g. v1, apps/v1kind — the type: Pod, Deployment, Servicemetadata — name, namespace, labels, annotationsspec — what you want ; you write thisstatus — what is ; the cluster writes this
apiVersion: v1
kind: Pod
metadata:
name: pinboard-api
namespace: pinboard
labels:
app.kubernetes.io/name: pinboard-api
spec:
containers:
- name: api
image: pinboard-api:1.0
imagePullPolicy: IfNotPresent
ports:
- { name: http, containerPort: 8080 }
Read the manifest aloud field by field — this is the shape of every file for the rest of the course. Warn them never to hand-edit status: if spec and status disagree, a controller is already working on it. imagePullPolicy: IfNotPresent matters on kind because the image is side-loaded and never exists in a registry; with Always the Pod would sit in ImagePullBackOff. Point out that the label is one of the recommended app.kubernetes.io/* keys we formalise next session.
The core idea
Desired state and the reconciliation loop
01
Declare You write spec and apply it
→
02
Observe Controller watches actual state
→
03
Diff Compare desired vs actual
→
04
Act Create, delete or update to close the gap
↺
05
Repeat Forever — this is self-healing
Same idea as terraform apply — except nobody has to run it. The loop never stops, so drift is corrected in seconds.
This is the single most important slide of the session. Delete a Pod owned by a Deployment and a new one appears before your prompt returns — not magic, just a loop noticing that replicas dropped from 3 to 2. Contrast level-triggered (compare full state every time) with edge-triggered (react to an event): Kubernetes is level-triggered, which is why a missed event is harmless. Tell them a bare Pod has no controller, so deleting it is permanent — the motivation for Session 06.
One command to learn
kubectl grammar
kubectl [verb] [resource] [name] [flags]
kubectl get pods # list
kubectl get pod pinboard-api -o yaml # one object, full spec+status
kubectl describe pod pinboard-api # human summary + events
kubectl logs pinboard-api -c api -f # container logs, follow
kubectl exec -it pinboard-api -- sh # a shell inside the container
kubectl port-forward pod/pinboard-api 8080:8080
kubectl delete pod pinboard-api
kubectl get pods -o wide --show-labels # node, IP, labels
kubectl get pods -l app.kubernetes.io/name=pinboard-apiVerbs: get describe create apply delete edit logs exec scale rollout. Resources take short names (po, deploy, svc, ns) and are always plural-friendly.
Tell them to install completion on day one: source <(kubectl completion bash) and alias k=kubectl. The two flags worth memorising now are -o wide and -o yaml; -o jsonpath comes in Session 08. Note that exec needs a shell in the image — our distroless API has none, which is exactly why kubectl debug exists (Session 09).
Two styles
Imperative vs declarative
Imperative — fast, unrepeatable
kubectl run pinboard-api \
--image=pinboard-api:1.0 \
-n pinboard --port=8080Great for a five-second experiment. Nothing is written down; nobody can review it.
Declarative — the real workflowkubectl apply -f pod.yaml
kubectl diff -f pod.yaml
kubectl apply -f k8s/ --dry-run=serverThe file is the source of truth: reviewable, versioned, re-appliable, and the input to GitOps.
Best of both: kubectl create deployment … --dry-run=client -o yaml > deploy.yaml — generate the skeleton, then edit and commit it.
Everything that ships goes through apply; imperative commands are for exploration and for exams. Mention that apply uses server-side apply with field managers now, so two owners editing the same field get a conflict instead of silently overwriting each other. In Lab 05 the students do both — kubectl run first, then the same Pod from a manifest — so the difference is felt, not asserted.
Who am I talking to?
kubeconfig and contexts
~/.kube/config — clusters, users, and contexts that pair themA context also carries a default namespace
$KUBECONFIG can merge several files with :kind create cluster adds and selects kind-pinboard
Every production incident story starts with "I thought I was on dev." Put the context in your shell prompt — kube-ps1, starship, k9s.
kubectl config get-contexts
kubectl config current-context
# kind-pinboard
kubectl config use-context kind-pinboard
# pin the namespace for this context
kubectl config set-context --current \
--namespace=pinboard
kubectl config view --minify
Show kubectl config get-contexts live and point at the asterisk. Explain that credentials in a managed cluster are usually an exec plugin calling az/aws/gcloud, so the file holds no long-lived secret. The set-context --current --namespace line saves the class hundreds of -n pinboard keystrokes in Lab 05 — have them run it right after the cluster is up.
Scoping
Namespaces
A name scope , not a security boundary by itself
Names must be unique per namespace ; DNS becomes svc.ns.svc.cluster.local
The hook for ResourceQuota, LimitRange, RBAC and NetworkPolicy
Cluster-scoped objects have none: Node, PersistentVolume, StorageClass, Namespace
kubectl create namespace pinboard
kubectl get ns
# default kube-system kube-public
# kube-node-lease pinboard
kubectl get pods -n pinboard
kubectl get pods -A # all namespaces
kubectl api-resources --namespaced=falseOurs is pinboard for every lab from here on.
Common mistake: expecting a namespace to isolate traffic. It does not — Pod-to-Pod traffic crosses namespaces freely until a NetworkPolicy says otherwise (Session 07). Namespaces are how you slice a cluster per team or per environment; some shops use a cluster per environment instead, and both are defensible. Show that kubectl get pods without -n silently means default, which is why the empty output confuses beginners.
Self-service docs
kubectl explain — the API reference offline
kubectl explain pod.spec
kubectl explain pod.spec.containers.resources
kubectl explain deployment.spec.strategy --recursive
kubectl api-resources | head
# NAME SHORTNAMES APIVERSION NAMESPACED KIND
# pods po v1 true Pod
# deployments deploy apps/v1 true Deployment
kubectl api-versions | grep appsexplain reads the OpenAPI schema from your cluster, so it is always right for the version you are on — including CRDs your team installed.
Teach this as the answer to "what fields can I put here?" — it beats guessing from a blog post. Demonstrate kubectl explain pod.spec.containers.livenessProbe and note we will use exactly those fields next session. Also show --recursive for the whole subtree, and mention that kubectl get --raw /openapi/v3 is where the data comes from.
A Pod from YAML, watched end to end
Side-load pinboard-api:1.0 into kind, apply the manifest, and watch the events narrate scheduling, pulling and starting in real time.
What you'll see · explain · apply · events --watch · port-forwardFiles · demos/session05/pod.yamlDuration · 10 min
Open a second terminal with kubectl get events -n pinboard --watch before applying. Then: kind load docker-image pinboard-api:1.0 --name pinboard, kubectl explain pod.spec.containers, kubectl apply -f pod.yaml, kubectl get pod -w. Read the event stream aloud — Scheduled, Pulled, Created, Started — mapping each line back to the five-step flow. Finish with kubectl port-forward pod/pinboard-api 8080:8080 and curl localhost:8080/api/info to show the theme is emerald.
04
Pods in depth
The smallest thing Kubernetes will schedule — and the only thing that ever runs your containers.
Block B, after the break. Shorter and much more concrete — everything here shows up directly in Lab 05, so keep the pace up and lean on the live cluster from the demo.
Anatomy
What a Pod actually shares
One network namespace — one IP, one port space; containers reach each other on localhostVolumes — declared on the Pod, mounted per containerLifecycle & node — scheduled together, killed togetherNot shared: filesystem root, PID namespace (opt-in), memory limits
Two containers cannot both bind :8080 in one Pod — same port space.
One app per Pod
pinboard-api and pinboard-web are separate Pods: they scale differently, fail differently and are released differently.
Extra containers belong in a Pod only when they are useless alone — a log shipper, a proxy, a config reloader.
The classic beginner error is packing api + web + db into one Pod because that is what the Compose file looked like. Give the test out loud: "would you ever scale these two to different numbers?" If yes, separate Pods. Mention the pause/infra container that holds the network namespace open, since they will spot it in docker ps on the kind nodes.
Reading the room
Pod lifecycle and the status you will see
Status What it means First move
Pending Accepted, not yet bound — no node fits, or a quota/taint blocks it describe pod → EventsContainerCreating Node chosen; pulling the image, mounting volumes, wiring the network Wait, then describe Running At least one container is up. Ready is a different column logs, check READY 1/1CrashLoopBackOff The container starts and exits repeatedly; kubelet backs off up to 5 min logs --previousImagePullBackOff Image missing, tag wrong, or registry auth failed Check tag; kind load Completed / Error Process exited 0 or non-zero and restartPolicy allowed it to stop Expected for Jobs
Make them memorise the pairing: status tells you where in the pipeline it stopped, and describe's Events tell you why . CrashLoopBackOff is not an error in itself — it is the kubelet being polite; the real error is in kubectl logs --previous. In Lab 05 they deliberately deploy pinboard-api:9.9 to earn an ImagePullBackOff and read the event themselves. Session 09 turns this table into a full troubleshooting method.
Sharing a machine
Requests, limits and QoS
requests — what the scheduler reserveslimits — what the kubelet enforcesOver CPU → throttled. Over memory → OOMKilled
QoS class When
Guaranteed requests == limits, every container Burstable requests set, limits differ or absent BestEffort neither set — evicted first
resources:
requests:
cpu: 50m # 0.05 of a core
memory: 32Mi
limits:
memory: 64Mi # no cpu limit on purposeCommon production stance: always set memory request = limit, and leave CPU unlimited so bursts are absorbed instead of throttled.
These are the numbers from labs/solutions/final/20-api-deployment.yaml — the Go API really does run in 32 Mi. Explain m as millicores: 1000m is one core, and CPU is compressible while memory is not, which is why one is throttled and the other is killed. Warn that a Guaranteed class needs limits on every container including sidecars. Session 09 breaks this deliberately with a 16 Mi limit.
More than one container
Init containers and native sidecars
initContainers run to completion, in order, before app containers startA native sidecar is an init container with restartPolicy: Always — GA since 1.29
Sidecars start before the app, keep running, and stop after it
That ordering is what the old "sidecar as a normal container" hack never got right
spec:
initContainers:
- name: wait-for-db
image: busybox:1.37
command: ["sh","-c",
"until nc -z pinboard-db 5432; do sleep 1; done"]
- name: log-shipper
image: busybox:1.37
restartPolicy: Always # ← native sidecar
command: ["sh","-c","tail -f /var/log/app.log"]
containers:
- name: api
image: pinboard-api:1.0
Use the Pinboard story: the API must not start querying before Postgres answers, so wait-for-db blocks the Pod instead of crash-looping. Then contrast: before 1.29 a log shipper as a normal container would keep a Job Pod alive forever and would not be guaranteed to start first — restartPolicy: Always on an init container fixes both. Note the same mechanism is what service meshes now use for their proxies.
Day-to-day
restartPolicy and working with a live Pod
restartPolicy: Always (default) · OnFailure · NeverIt restarts containers in place — the Pod is never rescheduled
A bare Pod on a dead node is simply gone. Nothing recreates it
That is why real workloads are owned by a controller — Session 06
Never run a bare Pod in production. It is a debugging tool.
kubectl describe pod pinboard-api -n pinboard
kubectl logs pinboard-api -n pinboard -f
kubectl logs pinboard-api -n pinboard --previous
kubectl exec -it pinboard-api -n pinboard -- sh
kubectl port-forward pod/pinboard-api 8080:8080 \
-n pinboard
curl -s localhost:8080/api/info
Four commands solve most problems: describe, logs, logs --previous, exec. Point out the restart counter in kubectl get pods — a quietly incrementing RESTARTS column is a bug nobody has noticed yet. Remind them our distroless image has no shell, so exec fails and kubectl debug is the answer; that is Session 09. Hand off to the lab here.
Pinboard meets Kubernetes
Create the three-node kind cluster, side-load the image you built in Lab 02, and run pinboard-api:1.0 as a Pod in the pinboard namespace — first imperatively, then from a manifest you keep.
You'll practise: kind create cluster · kind load docker-image · kubectl run vs apply · namespaces & contexts · port-forward · logs · describe · requests/limits · reading an ImagePullBackOff · kubectl explain
Set expectations before they start: the cluster takes about a minute to come up, and the image must be loaded with kind load or the Pod will sit in ImagePullBackOff — which is step 8 on purpose. Tell them to run kubectl config set-context --current --namespace=pinboard early. Keep the manifest they write; Session 06 turns it into a Deployment. Circulate and check that everyone can curl /api/info through port-forward before the break.
Session 05
Key takeaways
Everything goes through kube-apiserver into etcd; scheduler, controllers and kubelet just watch and react.
Declare desired state and the reconciliation loop closes the gap — continuously, without you re-running anything.
Shared IP and volumes, requests/limits for scheduling, init and native sidecar containers for ordering.
Session 06 — Deployments, rolling updates and the probes that make them safe.
Close the loop on the opening question: a bare Pod still does not survive a node failure, and that is exactly the gap Session 06 fills. Ask two quick checks before the break — "what does status Pending usually mean?" and "which component writes the nodeName?" — and make sure their kind cluster stays up, because Lab 06 builds directly on it.
Questions?
Up next — Session 06: Workloads, rollouts and probes . Leave the kind cluster running.
