Session 08 · 2h

Configuration & storage

ConfigMaps, Secrets, volumes, PersistentVolumeClaims and StatefulSets — everything a Pod needs that is not in its image.

ConfigMaps Secrets Volumes PV / PVC / StatefulSets Lab 08

Session 08

What we'll cover today

1 · ConfigMaps

One image, many environments: env vars vs mounted files, update propagation, and immutable ConfigMaps.

2 · Secrets

Types, why base64 is not encryption, encryption at rest, RBAC, and the external stores you'll use in production.

3 · Volumes & persistence

emptyDir to PersistentVolumeClaims, StorageClasses and CSI, then StatefulSets for stateful workloads.

4 · Lab 08

Give Pinboard a real PostgreSQL: a Secret, a ConfigMap, a StatefulSet with a 1 Gi claim — and notes that survive.

ConfigMaps

Non-sensitive configuration as a first-class Kubernetes object.

Remind them that in session 03 and 04 they already did this with -e and env_file. Kubernetes gives the same idea an API object with a name, a namespace, RBAC and a history.

The principle

Configuration lives outside the image

Build the image once, promote the same digest through every environment.

Different greeting, theme, log format, database URL — same pinboard-api:1.0
Rebuilding to change a setting destroys the "it is the tested artefact" guarantee
Config in Git, next to the manifests, reviewed like code

ConfigMap

Plain configuration: greetings, feature flags, whole config files. Up to 1 MiB.

Secret

Same shape, different intent: credentials, tokens, TLS keys. Different RBAC, different handling.

Neither is a place for megabytes: both live in etcd and are pushed to every node that runs the Pod.

The 1 MiB limit surprises people who try to mount a data file. If a config file is big, ship it in the image or fetch it at runtime. Also flag the naming convention we use: pinboard-api-config, in the pinboard namespace, labelled with app.kubernetes.io/part-of: pinboard.

The object

Creating a ConfigMap

--from-literal, --from-file, --from-env-file
Keys become env var names or file names
Generate, review, commit — never create straight into a cluster

kubectl -n pinboard create configmap pinboard-api-config \
  --from-literal=APP_GREETING="Welcome to Pinboard" \
  --from-file=nginx.conf \
  --dry-run=client -o yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: pinboard-api-config
  namespace: pinboard
  labels:
    app.kubernetes.io/name: pinboard-api
    app.kubernetes.io/part-of: pinboard
data:
  APP_GREETING: "Welcome to Pinboard on Kubernetes"
  APP_THEME: "emerald"
  LOG_FORMAT: "json"

This is 02-api-configmap.yaml from the lab solutions, verbatim. Mention binaryData exists for non-UTF-8 content. Stress the --dry-run=client -o yaml pattern: it is how you get a correct manifest without memorising the schema, and it works for every kubectl create subcommand.

Two ways in

Environment variables vs mounted files

envFrom — every key becomes an env var; simplest and what Pinboard uses
configMapKeyRef — one key, possibly renamed
Volume mount — one file per key in a directory
Files win for real config files (nginx, app.yaml) and for live updates

envFrom:
  - configMapRef:
      name: pinboard-api-config
env:
  - name: APP_THEME
    valueFrom:
      configMapKeyRef:
        name: pinboard-api-config
        key: APP_THEME
volumeMounts:
  - name: config
    mountPath: /etc/pinboard   # one file per key

Warn about envFrom: keys that are not valid env var names are silently skipped (an event is recorded), and a missing ConfigMap keeps the Pod in CreateContainerConfigError unless you mark the reference optional: true. Explicit env entries override envFrom ones.

The gotcha

Update propagation

Consumed as	What happens when the ConfigMap changes
env / envFrom	Nothing. Values are a snapshot taken at container start → `kubectl rollout restart`
volume mount	kubelet refreshes the files within about a minute; your app must re-read them
volume with `subPath`	Never updates. Avoid `subPath` if you want live config
`immutable: true`	Rejected — delete and recreate. Cheaper on the API server and safe from typos

Common pattern: an annotation holding a checksum of the ConfigMap on the Pod template, so a config change is a rollout.

This table answers the most frequent real-world bug: "I changed the ConfigMap and nothing happened". For immutable ConfigMaps explain the mechanism — the kubelet stops watching them, which matters at thousands of Pods — and that the usual workflow is a name suffix (-v2) plus a Deployment update, which is exactly what Kustomize's configMapGenerator does.

Everything runs in the demo namespace. Apply demos/session08/configmap.yaml, pod-cm-env.yaml and pod-cm-volume.yaml, and leave kubectl -n demo logs -f cm-volume streaming in a second pane. Patch the ConfigMap — kubectl -n demo patch cm pinboard-api-config -p '{"data":{"APP_GREETING":"Config changed"}}' — then show that kubectl -n demo exec cm-env -- env | grep APP_GREETING is unchanged while the streaming log flips over by itself within about a minute. Finish by naming the fix for a real workload: kubectl rollout restart deploy/pinboard-api.

Secrets

The same object with a different name — and a much more careful set of habits.

The object

Secrets and their types

type	Used for
`Opaque`	Default. Arbitrary key/value — our `pinboard-db` credentials
`kubernetes.io/dockerconfigjson`	Registry pull credentials (`imagePullSecrets`)
`kubernetes.io/tls`	`tls.crt` + `tls.key` for Ingress and Gateway listeners
`kubernetes.io/basic-auth` · `ssh-auth`	Structured credentials with validated keys
`kubernetes.io/service-account-token`	Legacy. Pods get short-lived projected tokens instead

Types are mostly a contract for consumers: the type tells a controller which keys must exist.

Workflow

Generate the manifest, don't hand-write base64

kubectl -n pinboard create secret generic pinboard-db \
  --from-literal=POSTGRES_USER=pinboard \
  --from-literal=POSTGRES_PASSWORD=pinboard-secret \
  --from-literal=POSTGRES_DB=pinboard \
  --dry-run=client -o yaml > 01-db-secret.yaml

--dry-run=client builds the object locally and never touches the cluster
data: holds base64; stringData: takes plain text and is encoded for you on apply
Other subcommands: create secret tls, create secret docker-registry

Our 01-db-secret.yaml was generated exactly like this and then edited to use stringData so it stays readable in the repo — fine for a training password, never for a real one. Mention that kubectl apply replaces the whole Secret, so keep it in one file.

Say it out loud

base64 is encoding, not encryption

kubectl -n pinboard get secret pinboard-db -o jsonpath='{.data.POSTGRES_PASSWORD}' | base64 -d
# pinboard-secret

Anyone with read access

…reads the value. Base64 stops shoulder-surfing, nothing else.

Git is forever

A committed Secret is a leaked Secret, even after the "fix" commit.

etcd and backups

Unencrypted by default — and every etcd snapshot carries them too.

Run the command live; it lands better than any slide. Then ask "who can do that?" and answer: anyone with get secrets in the namespace — and anyone who can create a Pod there, because a Pod can mount any Secret in its namespace. That is the bridge to the next slide.

Closing the gap

Making Secrets actually secret

Encryption at rest

An EncryptionConfiguration on the API server encrypts Secrets in etcd — ideally with a KMS v2 provider, not a local key.

RBAC + namespaces

Nobody gets cluster-wide get secrets. Separate namespaces separate blast radius. Audit reads.

Never in Git — plain

If manifests must hold secrets, encrypt them: SOPS + age/KMS, or Sealed Secrets.

Short lives

Rotate. Prefer workload identity and short-lived tokens over long-lived static passwords.

Production pattern

Keep the source of truth outside the cluster

External Secrets Operator

CRDs SecretStore + ExternalSecret. The operator reads Vault / AWS Secrets Manager / Azure Key Vault / GCP and syncs a normal Kubernetes Secret.

Apps stay unchanged; rotation happens upstream.

Secrets Store CSI Driver

Mounts secrets as a volume at Pod start, straight from the provider. The value never has to exist as a Secret object in etcd.

Optional secretObjects sync if a workload needs env vars.

Both keep credentials out of Git and give you one place to rotate. Pick one per platform, not one per team.

Manifest

How pinboard-api gets its credentials

Config by envFrom, one secret key by secretKeyRef
The database Pod uses envFrom: secretRef for the whole Secret
Env vars leak into kubectl describe pod? No — but they do land in crash dumps and logs. Files are safer
Never echo $DATABASE_URL in an entrypoint

envFrom:
  - configMapRef:
      name: pinboard-api-config
env:
  - name: DATABASE_URL
    valueFrom:
      secretKeyRef:
        name: pinboard-db
        key: DATABASE_URL

From 20-api-deployment.yaml — the shape you'll apply in the lab.

Note that describe pod shows the reference, not the value, which is one small advantage over a literal env var. The DATABASE_URL key already contains the full connection string pointing at pinboard-db:5432 — the headless Service from last session.

Type the kubectl -n demo create secret generic pinboard-db ... --dry-run=client -o yaml command from two slides back and show the base64 it prints. Apply the file version (demos/session08/secret.yaml, which uses stringData), read it back with kubectl -n demo get secret pinboard-db -o yaml to show that the API server encoded it anyway, then decode it: kubectl -n demo get secret pinboard-db -o jsonpath='{.data.POSTGRES_PASSWORD}' | base64 -d. Finish with kubectl auth can-i get secrets -n demo and ask the room who in their org can run that command today.

Volumes inside the Pod

Scratch space, injected config, and the one volume type you should almost never use.

The catalogue

Volume types you will actually meet

emptyDir

Created with the Pod, deleted with it. Scratch, caches, sharing files between containers. medium: Memory for tmpfs.

configMap / secret

Keys rendered as read-only files. Secrets land in tmpfs on the node.

projected

Several sources — ConfigMap, Secret, downwardAPI, SA token — into one directory.

downwardAPI

Pod name, namespace, labels, resource limits, as files or env vars.

hostPath

A path on the node. Ties the Pod to one machine and is a well-known escape hatch. Avoid.

persistentVolumeClaim

Real, durable storage that outlives the Pod. The rest of this session.

On hostPath: it is how node-level agents (log collectors, CNI, CSI drivers) legitimately work, and it is why the restricted Pod Security Admission level forbids it for normal workloads. Our namespace already carries that label from 00-namespace.yaml. Our web Pods use emptyDir for nginx's cache and run directories precisely because the root filesystem is read-only.

Manifest

emptyDir and a projected volume

volumes belong to the Pod, volumeMounts to each container
An emptyDir survives a container restart, not a Pod reschedule
Projected volumes are how the SA token, a CA bundle and config arrive in one directory
Mount config read-only; set defaultMode for keys and certificates

volumes:
  - name: cache
    emptyDir: {}
  - name: bundle
    projected:
      sources:
        - configMap: { name: pinboard-api-config }
        - secret: { name: pinboard-db }
        - downwardAPI:
            items:
              - path: pod-name
                fieldRef: { fieldPath: metadata.name }

Tie the emptyDir example to 30-web-deployment.yaml, which mounts three of them so nginx can run with readOnlyRootFilesystem: true. That is the pattern to remember: read-only root plus small emptyDirs for the paths the process must write.

Persistent storage

PersistentVolumes, claims, StorageClasses — asking for a disk without knowing what a disk is.

Three objects

PV, PVC and StorageClass

PersistentVolume

A piece of storage in the cluster — an EBS volume, an NFS export, a local-path directory. Cluster-scoped, usually created for you.

PersistentVolumeClaim

A workload's request: size, access mode, class. Namespaced, written by the app team, mounted by the Pod.

StorageClass

A named tier — fast SSD, cheap HDD, replicated — plus the provisioner and its parameters. Written by the platform team.

You normally only ever write PVCs. kubectl get sc tells you which classes exist and which one is (default). On kind: standard, the local-path provisioner.

Static provisioning still exists — an admin pre-creates PVs — but dynamic provisioning is the norm: the claim triggers the creation. Leaving storageClassName out means "use the default class", which is exactly what our StatefulSet does so it works on kind and on a cloud alike.

Lifecycle

Provision → bind → use → reclaim

Provision

PVC created → the StorageClass's provisioner makes a volume and a PV

→

Bind

PVC and PV are matched 1:1 and go Bound

→

Use

Pod mounts the claim; the volume is attached to that node

→

Reclaim

PVC deleted → Delete destroys the volume, Retain keeps it for manual recovery

Deleting a Pod never deletes data — deleting the PVC does, if the class reclaims with Delete. Most default classes do.

Two details worth ten seconds each: volumeBindingMode: WaitForFirstConsumer delays binding until the scheduler picks a node, which is what makes zonal disks land in the right zone (kind's local-path class uses it too); and a PVC in use is protected by a finalizer, so a delete hangs in Terminating until the Pod goes away.

Ask for the right thing

Access modes and CSI

Access mode	Meaning
`ReadWriteOnce` (RWO)	Read-write by Pods on one node. Block storage: EBS, Azure Disk, local-path
`ReadWriteOncePod` (RWOP)	Read-write by exactly one Pod in the cluster. The safe choice for databases
`ReadOnlyMany` (ROX)	Read-only from many nodes
`ReadWriteMany` (RWX)	Read-write from many nodes — needs a shared filesystem: NFS, CephFS, EFS, Azure Files

CSI is the plugin interface behind all of it: every vendor ships a driver, and features like snapshots, cloning and volume expansion come with it.

The classic mistake is assuming RWX. Most block storage cannot do it, so an RWX claim simply never binds. RWOP went GA in 1.29 and is what you want for PostgreSQL — two Pods writing the same data directory is corruption, and RWO alone does not prevent it on the same node. Mention that expanding a PVC is just editing resources.requests.storage when the class allows it.

StatefulSets & databases

When Pods are not interchangeable: stable names, stable storage, ordered rollouts.

What you get

The four identity guarantees

Stable names

Ordinals, not hashes: pinboard-db-0, -1, -2. A replaced Pod gets the same name back.

Stable DNS

Via the headless Service: pinboard-db-0.pinboard-db.pinboard.svc.cluster.local always means that member.

Stable storage

volumeClaimTemplates gives each ordinal its own PVC, re-attached to the replacement Pod.

Ordered operations

Created 0→N, deleted and updated N→0, one at a time — unless you set podManagementPolicy: Parallel.

Warn them that deleting the StatefulSet leaves the PVCs behind on purpose: clean-up is deliberate, unless you configure persistentVolumeClaimRetentionPolicy. Emphasise that a StatefulSet gives identity, not clustering: it will not replicate Postgres or elect a leader. That logic lives in the application or in an operator. The retention policy field lets you choose delete-on-scale-down or delete-on-StatefulSet-delete; the default keeps everything.

Manifest

pinboard-db as a StatefulSet

serviceName must name the headless Service (session 07)
One replica: Pinboard needs durability, not high availability
The claim template creates data-pinboard-db-0, 1 Gi, on kind's default class
The Pod spec adds probes, PGDATA, and credentials via envFrom: secretRef

apiVersion: apps/v1
kind: StatefulSet
metadata: { name: pinboard-db, namespace: pinboard }
spec:
  serviceName: pinboard-db      # headless Service
  replicas: 1
  selector:
    matchLabels: { app.kubernetes.io/name: pinboard-db }
  template: { ... }             # postgres:17-alpine + probes
  volumeClaimTemplates:
  - metadata: { name: data }
    spec:
      accessModes: ["ReadWriteOnce"]
      resources: { requests: { storage: 1Gi } }

This is 10-db-statefulset.yaml with the Pod template folded away for space — show the real file in the lab. Two details from it worth calling out: PGDATA points at a sub-directory so initdb is not confused by lost+found, and the container runs non-root with a read-only root filesystem plus emptyDirs for /tmp and the socket directory.

The honest answer

Databases on Kubernetes — or managed?

Run it in-cluster when…

You use a mature operator (CloudNativePG, Percona, Strimzi)
You need the same platform everywhere, on-prem included
Someone owns backups, restores and version upgrades
You have tested a restore. Recently.

Use a managed service when…

The team is small and the database is critical
You want failover, PITR and patching as a feature
Storage performance and quorum are not your specialism
"It's just a StatefulSet" is the only plan

A bare StatefulSet is perfect for learning and for dev environments — which is exactly how we use it in the lab.

The moment to build up: kubectl -n pinboard delete pod pinboard-db-0, then refresh the browser — same notes, same PVC, new Pod, same name. Tell them to run kubectl -n pinboard get pvc,pv before and after so they see the claim survive. Finish with a ConfigMap edit plus rollout restart to change the greeting.

Session 08

Key takeaways

One image, injected config

ConfigMaps as env vars need a restart; as files they update live. immutable: true when it should never change.

Secrets need real controls

base64 is encoding. Encryption at rest, tight RBAC, nothing in Git, and an external store in production.

Claim storage, don't provision it

PVC + StorageClass + CSI. StatefulSets add stable names, stable DNS and a PVC per ordinal.

Next up

Session 09 — observability, a troubleshooting method, autoscaling and security basics.

Questions?

Up next — Session 09: Operating Kubernetes. Leave the database running, we will break it on purpose.

Configuration & storage

What we'll cover today

1 · ConfigMaps

2 · Secrets

3 · Volumes & persistence

4 · Lab 08

ConfigMaps

Configuration lives outside the image

ConfigMap

Secret

Creating a ConfigMap

Environment variables vs mounted files

Update propagation

Change a ConfigMap: env vs mounted file

Secrets

Secrets and their types

Generate the manifest, don't hand-write base64

base64 is encoding, not encryption

Anyone with read access

Git is forever

etcd and backups

Making Secrets actually secret

Encryption at rest

RBAC + namespaces

Never in Git — plain

Short lives

Keep the source of truth outside the cluster

External Secrets Operator

Secrets Store CSI Driver

How pinboard-api gets its credentials

Create a Secret, then read it straight back

Volumes inside the Pod

Volume types you will actually meet

emptyDir

configMap / secret

projected

downwardAPI

hostPath

persistentVolumeClaim

emptyDir and a projected volume

Persistent storage

PV, PVC and StorageClass

PersistentVolume

PersistentVolumeClaim

StorageClass

Provision → bind → use → reclaim

Provision

Bind

Use

Reclaim

Access modes and CSI

StatefulSets & databases

The four identity guarantees

Stable names

Stable DNS

Stable storage

Ordered operations

pinboard-db as a StatefulSet

Databases on Kubernetes — or managed?

Run it in-cluster when…

Use a managed service when…

Give Pinboard a real database

Key takeaways

One image, injected config

Secrets need real controls

Claim storage, don't provision it

Next up

Questions?