Session 03 demos – Running containers well: config, storage, networking

Demos 1–2 belong to Block A (configuration, PID 1 and signals, storage), Demos 3–5 to Block B (networking, limits, restart policies and health checks). Files next to this one:

File	Used in
`Dockerfile.shell`	Demo 1 — `CMD sleep 300` (shell form)
`Dockerfile.exec`	Demo 1 — `CMD ["sleep","300"]` (exec form)

Before the session

~5 minutes:

cd ~/docker-kubernetes-training/demos/session03

docker pull busybox:1.37
docker pull alpine:3.22
docker pull postgres:17-alpine
docker build -f Dockerfile.shell -t sig:shell .
docker build -f Dockerfile.exec  -t sig:exec  .

# The Pinboard images from Lab 02 (rebuild them if they are gone):
docker build -t pinboard-api:1.0 --build-arg APP_VERSION=1.0 ../../src/pinboard-api
docker build -t pinboard-web:1.0 ../../src/pinboard-web

Terminal: two panes — left for docker run/stop, right kept on docker stats or docker logs -f.
Free ports: 8080 (web) and 8081 (api).
Clean start: docker ps -a empty, docker volume ls empty, docker network ls shows only bridge/host/none.
A browser window on http://localhost:8080, ready but not yet loaded (Demo 2).
Keep src/pinboard-web/html/index.html unmodified at the start — Demo 2 edits it live and the Reset section puts it back.

Demo 1: PID 1, signals, and why `docker stop` takes ten seconds (≈ 8 min)

What to show / say

docker stop is a polite request (SIGTERM), then a killing (SIGKILL) after 10 s. Whether your container is polite back depends on who PID 1 is and whether it listens.

Steps

Show what the two Dockerfiles actually configure:

docker run -d --name sig-shell sig:shell
docker run -d --name sig-exec  sig:exec
docker inspect --format ' →  ' sig-shell sig-exec

/sig-shell → /bin/sh ["-c","sleep 300"]
/sig-exec  → sleep ["300"]

👉 “Shell form wraps your command in /bin/sh -c. Exec form is your command. That is the entire syntactic difference — and it decides who receives the signal.”

Now stop them and watch the clock:
```
time docker stop sig-shell
time docker stop sig-exec
docker inspect --format ' exit=' sig-shell sig-exec
```
```
real    0m10.24s
real    0m10.21s
/sig-shell exit=137
/sig-exec  exit=137
```
👉 This is the moment of the demo. “Both took ten seconds. Exit code 137 = 128 + 9 = SIGKILL. The exec form did not save us — because PID 1 inside a namespace does not get the kernel’s default signal actions. A process with no SIGTERM handler simply ignores it when it is PID 1. sleep has no handler.”

Note (say it, it comes up every time). BusyBox’s sh optimises sh -c '<one simple command>' into an exec, so docker top sig-shell may show only sleep. With a real shell, or any command containing a pipe or &&, the shell stays as PID 1 and the child never sees the signal at all — the failure is then even worse, because the shell exits and the child is orphaned.

Fix number one — give PID 1 to an init that forwards signals:

docker run -d --init --name sig-init busybox:1.37 sleep 300
docker top sig-init
time docker stop sig-init
docker inspect --format 'exit=' sig-init

PID     USER    COMMAND
30211   root    /sbin/docker-init -- sleep 300
30244   root    sleep 300
real    0m0.31s
exit=143

👉 point at PID 1 being docker-init (tini) and sleep being its child — “now sleep is an ordinary process, so the kernel’s default action applies and it dies immediately. 143 = 128 + 15 = SIGTERM. Instant.”

Fix number two, the real one — an application that handles the signal. pinboard-api uses exec-form ENTRYPOINT and listens for SIGTERM:

docker run -d --name pinboard-api -p 8081:8080 -e SHUTDOWN_GRACE=10s pinboard-api:1.0
curl -s localhost:8081/ ; time docker stop pinboard-api
docker logs pinboard-api | tail -3
docker inspect --format 'exit=' pinboard-api

Pinboard API 1.0 on 2f6c1b9d40ae (store: memory)
real    0m0.28s
time=… level=INFO msg="signal received, shutting down gracefully" grace=10s
time=… level=INFO msg=bye
exit=0

👉 “Exit 0. It drained its connections and closed the database pool first. That log line is what you will look for in Session 06 when a rolling update has to happen without dropping a request — and SHUTDOWN_GRACE is what terminationGracePeriodSeconds will have to be larger than.”

Clean up:

docker rm -f sig-shell sig-exec sig-init pinboard-api

Demo 2: the writable layer is a lie — volumes, bind mounts, tmpfs (≈ 8 min)

What to show / say

Everything a container writes goes into its throwaway upper layer unless you say otherwise. Volumes, bind mounts and tmpfs are the three ways to say otherwise.

Steps

Prove the problem in twenty seconds, without a database:

docker run --rm alpine:3.22 sh -c 'echo important > /data.txt; cat /data.txt'
docker run --rm alpine:3.22 cat /data.txt

important
cat: can't open '/data.txt': No such file or directory

👉 “New container, new writable layer. This is what bit us at the end of Lab 01.”

Postgres with a named volume, using the explicit --mount syntax:

docker run -d --name pinboard-db \
  -e POSTGRES_USER=pinboard -e POSTGRES_PASSWORD=pinboard-secret -e POSTGRES_DB=pinboard \
  --mount type=volume,src=pinboard-data,dst=/var/lib/postgresql/data \
  postgres:17-alpine
sleep 5
docker exec -it pinboard-db psql -U pinboard -d pinboard \
  -c "CREATE TABLE IF NOT EXISTS notes(id bigserial primary key, text text);" \
  -c "INSERT INTO notes(text) VALUES ('survives the container');" \
  -c "SELECT * FROM notes;"

👉 “--mount is verbose on purpose: type=, src=, dst= — no guessing, unlike -v a:b:c. Learn this one; it is the only syntax that works for tmpfs and for Swarm/Compose long form.”

Destroy the container — keep the data:

docker rm -f pinboard-db
docker volume ls
docker run -d --name pinboard-db \
  -e POSTGRES_USER=pinboard -e POSTGRES_PASSWORD=pinboard-secret -e POSTGRES_DB=pinboard \
  --mount type=volume,src=pinboard-data,dst=/var/lib/postgresql/data \
  postgres:17-alpine
sleep 5
docker exec -it pinboard-db psql -U pinboard -d pinboard -c "SELECT * FROM notes;"

 id |          text
----+------------------------
  1 | survives the container

👉 “Different container ID, same data. The volume outlives it — and only docker volume rm (or compose down -v) deletes it.”

Where does that actually live?
```
docker volume inspect pinboard-data
```
```
[
    {
        "CreatedAt": "2026-03-…",
        "Driver": "local",
        "Mountpoint": "/var/lib/docker/volumes/pinboard-data/_data",
        "Name": "pinboard-data",
        "Scope": "local"
    }
]
```
👉 “local driver = a directory on this host. Which is why ‘the volume survives’ stops being true the moment there are two hosts — the first crack that Kubernetes PersistentVolumes will fill in Session 08.”
Back it up with a throwaway container — the pattern worth memorising:
```
docker run --rm -v pinboard-data:/data -v "$PWD":/backup alpine:3.22 \
  tar czf /backup/db.tgz /data
ls -lh db.tgz
```
👉 “A container that exists for two seconds, mounts the volume read-write and your working directory as /backup, and tars one into the other. No Postgres client, no host tooling.”

Warning. Tarring a running database’s data directory is a crash-consistent copy, not a backup. For Postgres you would pg_dump — but the container pattern is exactly the same.
Bind mount = your source folder, live:
```
docker run -d --name pinboard-web -p 8080:8080 \
  --mount type=bind,src="$PWD/../../src/pinboard-web/html",dst=/usr/share/nginx/html,ro \
  -e API_URL=http://pinboard-api:8080 pinboard-web:1.0
sed -i 's|<h1>📌 Pinboard</h1>|<h1>📌 Pinboard — live from the host</h1>|' \
  ../../src/pinboard-web/html/index.html
```
👉 refresh the browser on http://localhost:8080 — the title changed with no rebuild, no restart. “Bind mount = host path straight into the container. Perfect for development, wrong for production: it depends on a path on a specific machine, and the container can write to your disk (that is why I added ,ro).”

tmpfs = memory only, never touches disk:

docker run --rm --mount type=tmpfs,dst=/scratch,tmpfs-size=16m alpine:3.22 \
  sh -c 'df -h /scratch; echo secret > /scratch/token; ls -l /scratch'

Filesystem   Size  Used Avail Use% Mounted on
tmpfs         16M     0   16M   0% /scratch

👉 “Sixteen megabytes of RAM. Nothing is written to the host disk, and it is gone when the container stops — the right place for a decrypted secret or a scratch file.”

Clean up the web container (leave pinboard-db and the volume for Demo 3):
```
docker rm -f pinboard-web
```

Demo 3: networking — the default bridge has no DNS (≈ 7 min)

What to show / say

Container names only resolve on a user-defined network. This single fact explains 90 % of “it works in Compose but not with docker run”.

Steps

On the default bridge, names do not exist:
```
docker run -d --name api-default -e APP_VERSION=1.0 pinboard-api:1.0
docker run --rm busybox:1.37 wget -qO- http://api-default:8080/
```
```
wget: bad address 'api-default:8080'
```
👉 “Both containers are on the default bridge and can reach each other by IP. There is no DNS server on it — the name means nothing.”

Prove the IP part, so nobody thinks the network is broken:

ip=$(docker inspect --format '' api-default); echo "$ip"
docker run --rm busybox:1.37 wget -qO- "http://$ip:8080/"

172.17.0.2
Pinboard API 1.0 on 8f1c2d3e4a5b (store: memory)

👉 “It works — with an address that changes every time the container is recreated. Nobody can build on that.”

A user-defined network gives you Docker’s embedded DNS at 127.0.0.11:

docker network create pinboard-net
docker run -d --name pinboard-api --network pinboard-net -e APP_VERSION=1.0 pinboard-api:1.0
docker run --rm --network pinboard-net busybox:1.37 nslookup pinboard-api
docker run --rm --network pinboard-net busybox:1.37 wget -qO- http://pinboard-api:8080/

Server:    127.0.0.11
Name:      pinboard-api
Address 1: 172.18.0.2 pinboard-api.pinboard-net
Pinboard API 1.0 on 3c4d5e6f7a8b (store: memory)

👉 “The container name is the hostname. This is exactly what Compose does for you automatically in the next session, and what a Kubernetes Service will do in Session 07.”

Who is on the network:

docker network inspect pinboard-net \
  --format ' '

--network host — no isolation, no port publishing:
```
docker run -d --name api-host --network host -e PORT=8082 pinboard-api:1.0
curl -s localhost:8082/ ; docker port api-host
```
```
Pinboard API 1.0 on <your-laptop-hostname> (store: memory)
```
👉 point at the hostname being the host’s, and at docker port printing nothing: “no -p needed, because there is no separate network namespace to publish from. Fast, and occasionally necessary — but the container can now bind any port on your machine and see all of its traffic.”
Clean up the extras (keep pinboard-net and pinboard-api for Demo 4):
```
docker rm -f api-default api-host
```

Demo 4: resource limits — OOMKilled and CPU throttling (≈ 7 min)

What to show / say

Without limits, one container can take the whole host. With limits, the cgroup we saw in Session 01 does the enforcing, and it does not negotiate.

Steps

A container with a 32 MB memory limit and a program that wants more:
```
docker run -d --name hog --memory 32m --memory-swap 32m busybox:1.37 \
  sh -c 'dd if=/dev/zero of=/dev/shm/x bs=1M count=64; sleep 60'
sleep 3
docker inspect --format 'status= exit= oom=' hog
```
```
status=exited exit=137 oom=true
```
👉 “OOMKilled: true. /dev/shm is a tmpfs, so those pages count against the container’s memory cgroup — the kernel hit 32 MB and killed the process. Exit 137 again, but this time nobody ran docker stop.”
```
docker logs hog
```
👉 “And notice: the application logged nothing about it. The evidence is in docker inspect, and in Kubernetes it will be in kubectl describe pod. This is Session 09’s most common student bug.”

The same limit applied to something real:

docker run -d --name api-limited --network pinboard-net --memory 32m \
  -p 8081:8080 pinboard-api:1.0
docker stats --no-stream api-limited

NAME          CPU %   MEM USAGE / LIMIT   MEM %   NET I/O   PIDS
api-limited   0.00%   6.1MiB / 32MiB      19.1%   1.1kB/0B  8

👉 point at LIMIT now saying 32MiB instead of the host’s RAM. “6 MB of 32. That is why the Kubernetes manifests in this course request 32Mi and limit 64Mi.”

CPU limits with the API’s own burn endpoint:
```
docker run -d --name api-cpu --cpus 0.5 -p 8082:8080 pinboard-api:1.0
curl -s "localhost:8082/api/burn?ms=6000" >/dev/null &
sleep 2 && docker stats --no-stream api-cpu
```
```
NAME      CPU %   MEM USAGE / LIMIT   MEM %   PIDS
api-cpu   49.83%  7.2MiB / 15.6GiB    0.05%   9
```
👉 “The handler is a tight loop — it wants a whole core. It gets half of one: --cpus 0.5 writes 50000 100000 into the cgroup’s cpu.max. Nothing is killed; the process is simply throttled — slower, not dead. Memory limits kill, CPU limits slow down. Remember that difference.”
Clean up:
```
docker rm -f hog api-limited api-cpu
```

Demo 5: restart policies and health checks (≈ 6 min)

What to show / say

Docker can restart a crashed container, but only this Docker, on this host — and “running” is not the same as “working”.

Steps

A container that crashes on purpose (CRASH_ON_START is built into the API for exactly this):
```
docker run -d --name crasher --restart on-failure:3 \
  -e CRASH_ON_START=true pinboard-api:1.0
sleep 8
docker inspect --format 'status= restarts= exit=' crasher
docker logs crasher | tail -2
```
```
status=exited restarts=3 exit=1
level=ERROR msg="CRASH_ON_START is set — exiting with status 1 (this is on purpose)"
```
👉 “Three retries with a growing delay, then it gives up and stays exited. Compare the four policies: no (default), on-failure[:n], always, unless-stopped. always also means restart when the daemon starts, which is how a container survives a reboot.”

Health checks — pinboard-web declares one in its Dockerfile:

grep -A2 HEALTHCHECK ../../src/pinboard-web/Dockerfile
docker run -d --name pinboard-web --network pinboard-net -p 8080:8080 \
  -e API_URL=http://pinboard-api:8080 pinboard-web:1.0
docker ps --format 'table \t'

HEALTHCHECK --interval=10s --timeout=2s --retries=3 \
  CMD wget -qO- http://127.0.0.1:8080/healthz || exit 1

NAMES          STATUS
pinboard-web   Up 4 seconds (health: starting)

Wait ~10 s and repeat:

docker ps --format 'table \t'
docker inspect --format '' pinboard-web | python3 -m json.tool | head -12

NAMES          STATUS
pinboard-web   Up 18 seconds (healthy)

👉 point at the Log array in the JSON: every probe run, its exit code and its output are kept. “docker ps gained a health column for free.”

What an unhealthy container looks like — override the check at run time:
```
docker run -d --name web-broken --network pinboard-net -p 8083:8080 \
  --health-cmd 'wget -qO- http://127.0.0.1:9999/ || exit 1' \
  --health-interval 3s --health-retries 2 pinboard-web:1.0
sleep 12
docker ps --format 'table \t' --filter name=web-broken
```
```
NAMES        STATUS
web-broken   Up 12 seconds (unhealthy)
```
👉 “The container is running and serving traffic on 8083 perfectly well — the probe is what is wrong. Docker marks it unhealthy and… does nothing else. It will not restart it, and nothing stops sending it traffic. That gap is exactly what Kubernetes liveness and readiness probes close in Session 06.”

Clean up:

docker rm -f crasher pinboard-web web-broken

Reset

Lab 03 has students build pinboard-net, pinboard-data, pinboard-db, pinboard-api and pinboard-web themselves, so remove everything the demos created — but keep the images.

cd ~/docker-kubernetes-training/demos/session03

# 1. Undo the live edit from Demo 2 step 6
sed -i 's|<h1>📌 Pinboard — live from the host</h1>|<h1>📌 Pinboard</h1>|' \
  ../../src/pinboard-web/html/index.html
grep -n '<h1>' ../../src/pinboard-web/html/index.html

# 2. Containers, network, volume, backup file
docker rm -f pinboard-db pinboard-api pinboard-web sig-shell sig-exec sig-init \
  api-default api-host api-limited api-cpu hog crasher web-broken 2>/dev/null
docker network rm pinboard-net 2>/dev/null
docker volume rm pinboard-data 2>/dev/null
rm -f db.tgz

# 3. Demo-only images
docker image rm sig:shell sig:exec 2>/dev/null

Verify:

docker ps -a                                  # empty
docker volume ls                              # empty
docker network ls                             # bridge / host / none only
docker image ls 'pinboard-*'                  # pinboard-api:1.0, pinboard-web:1.0 kept

Session 03 demos – Running containers well: config, storage, networking

Before the session

Demo 1: PID 1, signals, and why docker stop takes ten seconds (≈ 8 min)

What to show / say

Steps

Demo 2: the writable layer is a lie — volumes, bind mounts, tmpfs (≈ 8 min)

What to show / say

Steps

Demo 3: networking — the default bridge has no DNS (≈ 7 min)

What to show / say

Steps

Demo 4: resource limits — OOMKilled and CPU throttling (≈ 7 min)

What to show / say

Steps

Demo 5: restart policies and health checks (≈ 6 min)

What to show / say

Steps

Reset

Demo 1: PID 1, signals, and why `docker stop` takes ten seconds (≈ 8 min)