Session 10 demos – packaging: Helm, Kustomize, a word about GitOps

Block A is 30 minutes for all of this, then the capstone. Demos 1–2 are Helm, 3–4 are Kustomize, 5 is thirty seconds of Argo CD. If you fall behind, cut Demo 2 to helm template alone — never cut Demo 4, the diff is the punchline. Files next to this one:

File	Used in
`kustomize/base/`	Demos 3–4 — namespace-less manifests, no PDB/HPA, ConfigMap generated
`kustomize/overlays/dev/kustomization.yaml`	Demos 3–4 — `pinboard-dev`, 1 replica, tag 1.0, greeting override
`kustomize/overlays/prod/`	Demo 4 — `pinboard-prod`, 3 replicas, tag 1.1, `+pdb.yaml +hpa.yaml`, host patch
`helm/README.md`	Reference for “what would our own chart look like?” — not demoed live

Before the session

Cluster, images, internet check (Helm pulls from the network):

cd ~/docker-kubernetes-training/demos/session10

kubectl get nodes
kind load docker-image pinboard-api:1.0 pinboard-api:1.1 pinboard-web:1.0 --name pinboard
docker pull postgres:17-alpine && kind load docker-image postgres:17-alpine --name pinboard
helm version --short          # expect v3.x

If Session 07 left the manifest install of ingress-nginx in place, remove it — Demo 1 installs the same controller from its chart and the two would fight over host port 80:
```
kubectl delete -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/kind/deploy.yaml --ignore-not-found
```

Pre-fetch the chart so the room does not watch a download:

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update

pinboard-prod must not exist when the capstone starts — the students create it themselves. This session only renders the prod overlay; the Reset section checks it.

Demo 1: `helm install` — the same controller, packaged (≈ 7 min)

What to show / say

Session 07 installed ingress-nginx from a 700-line YAML file pinned to a URL. Here is the same software as a versioned, configurable, removable release.

Steps

helm search repo ingress-nginx --versions | head -4

NAME                            CHART VERSION   APP VERSION   DESCRIPTION
ingress-nginx/ingress-nginx     4.13.3          1.13.3        Ingress controller for Kubernetes using NGINX...
ingress-nginx/ingress-nginx     4.13.2          1.13.2        Ingress controller for Kubernetes using NGINX...

👉 two versions on every line: the chart version (the packaging) and the app version (nginx itself). They move independently — that distinction is most of what people get wrong about Helm.

Look before you install:
```
helm show values ingress-nginx/ingress-nginx | wc -l
helm show values ingress-nginx/ingress-nginx | grep -n -A3 'hostPort:' | head
```
```
1120
```
👉 1120 lines of values. That is the chart’s API surface — and the reason helm show values | less is the first command you run against any chart.

Install it, kind-style (the three settings the static manifest hard-coded):

helm install ingress-nginx ingress-nginx/ingress-nginx \
  --namespace ingress-nginx --create-namespace \
  --set controller.hostPort.enabled=true \
  --set controller.service.type=NodePort \
  --set-string controller.nodeSelector.ingress-ready=true \
  --set controller.tolerations[0].key=node-role.kubernetes.io/control-plane \
  --set controller.tolerations[0].operator=Exists \
  --set controller.tolerations[0].effect=NoSchedule \
  --wait --timeout 5m

NAME: ingress-nginx
LAST DEPLOYED: Tue May 12 10:04:11 2026
NAMESPACE: ingress-nginx
STATUS: deployed
REVISION: 1

👉 --wait blocks until the resources are ready, so a script can trust the exit code. 👉 point at REVISION: 1 — that is the thing Kustomize does not have.

helm list -A

NAME            NAMESPACE       REVISION  UPDATED     STATUS     CHART                   APP VERSION
ingress-nginx   ingress-nginx   1         2026-05-12  deployed   ingress-nginx-4.13.3    1.13.3

Where does that state live? Not in a file on your laptop:
```
kubectl -n ingress-nginx get secret -l owner=helm
```
```
NAME                                 TYPE                 DATA   AGE
sh.helm.release.v1.ingress-nginx.v1  helm.sh/release.v1   1      52s
```
👉 the rendered manifest, gzipped, in a Secret in the cluster. One Secret per revision. That is how helm rollback and helm uninstall know what they own — and why two people running helm install from different laptops still agree.

Confirm it works exactly like the manifest install did:

kubectl get ingressclass
curl -s -o /dev/null -w '%{http_code}\n' http://localhost/

NAME    CONTROLLER             PARAMETERS   AGE
nginx   k8s.io/ingress-nginx   <none>       70s
404

👉 404 = the controller is listening on port 80 and has no matching rule yet.

Demo 2: `get values`, `upgrade --set`, `template` (≈ 5 min)

What to show / say

The three commands you will actually use after the install: what did I set, change one thing, and render without applying.

Steps

What did I choose, as opposed to what the chart defaults to?

helm get values ingress-nginx -n ingress-nginx

USER-SUPPLIED VALUES:
controller:
  hostPort:
    enabled: true
  nodeSelector:
    ingress-ready: "true"
  service:
    type: NodePort
  tolerations:
  - effect: NoSchedule
    key: node-role.kubernetes.io/control-plane
    operator: Exists

👉 only the overrides. helm get values --all prints the merged 1120 lines. This command is how you recover an install you inherited.

Change one value — no file, no kubectl edit:

helm upgrade ingress-nginx ingress-nginx/ingress-nginx \
  -n ingress-nginx --reuse-values \
  --set controller.replicaCount=2
kubectl -n ingress-nginx get pods
helm history ingress-nginx -n ingress-nginx

REVISION  UPDATED     STATUS      CHART                  APP VERSION  DESCRIPTION
1         2026-05-12  superseded  ingress-nginx-4.13.3   1.13.3       Install complete
2         2026-05-12  deployed    ingress-nginx-4.13.3   1.13.3       Upgrade complete

👉 --reuse-values keeps the previous overrides; without it you start from the chart defaults and silently lose your kind settings. This trips up everyone once. 👉 the second controller Pod will stay Pending — only one node has ingress-ready=true, and only one process can own host port 80. A perfectly good accidental lesson; leave it on screen for a moment.

Undo it, with a command Kustomize has no equivalent of:

helm rollback ingress-nginx 1 -n ingress-nginx
helm history ingress-nginx -n ingress-nginx | tail -2

3   2026-05-12  deployed  ingress-nginx-4.13.3  1.13.3  Rollback to 1

👉 like kubectl rollout undo, a rollback rolls forward to an old state as a new revision.

The command for CI — render, apply nothing:
```
helm template ingress-nginx ingress-nginx/ingress-nginx \
  --set controller.replicaCount=3 | grep -c '^kind:'
helm template ingress-nginx ingress-nginx/ingress-nginx | head -20
```
👉 helm template never contacts the cluster. Lint it, diff it, scan it, pipe it into kubeconform or a policy check. 👉 and point at the Go templating in the chart’s templates/ — then say the honest thing: “this is YAML generated by a text templating language. It works, it is everywhere, and it is why Kustomize exists.” That is the bridge to Demo 3. helm/README.md in this folder has the helm create walkthrough if anyone asks for their own chart.

Demo 3: `kubectl kustomize` — render an overlay (≈ 5 min)

What to show / say

No templates, no placeholders. Plain YAML in, plain YAML out — and the overlay never copies a manifest.

Steps

Show the shape of it first:

find kustomize -type f | sort

kustomize/base/api-deployment.yaml
kustomize/base/db-secret.yaml
kustomize/base/db-statefulset.yaml
kustomize/base/ingress.yaml
kustomize/base/kustomization.yaml
kustomize/base/namespace.yaml
kustomize/base/web-deployment.yaml
kustomize/overlays/dev/kustomization.yaml
kustomize/overlays/prod/hpa.yaml
kustomize/overlays/prod/kustomization.yaml
kustomize/overlays/prod/pdb.yaml

👉 the dev overlay is one file. Everything it changes, it changes by reference.

```
cat kustomize/overlays/dev/kustomization.yaml
```
👉 read the four fields aloud: namespace, resources, replicas, images, plus the configMapGenerator merge. That is the entire dev environment.

Render it — nothing touches the cluster:

cd kustomize
kubectl kustomize overlays/dev | head -40

apiVersion: v1
kind: Namespace
metadata:
  labels:
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/part-of: pinboard
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/warn: restricted
  name: pinboard-dev
---
apiVersion: v1
data:
  APP_GREETING: Pinboard DEV — data may be wiped at any time
  APP_THEME: emerald
  LOG_FORMAT: text
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/managed-by: kustomize
    app.kubernetes.io/part-of: pinboard
  name: pinboard-api-config-dc8cmkk9c7
  namespace: pinboard-dev

👉 three things happened in those 20 lines: the Namespace object itself was renamed to pinboard-dev (the namespace: transformer does that too), the ConfigMap kept APP_THEME from the base and took APP_GREETING/LOG_FORMAT from the overlay, and the generated name carries a content hash.

Follow the hash:
```
kubectl kustomize overlays/dev | grep -E 'configMapRef|name: pinboard-api-config|image:|replicas:'
```
```
  name: pinboard-api-config-dc8cmkk9c7
  replicas: 1
      - configMapRef:
          name: pinboard-api-config-dc8cmkk9c7
      image: pinboard-api:1.0
  replicas: 1
      image: pinboard-web:1.0
  replicas: 1
      image: postgres:17-alpine
```
👉 the Deployment’s configMapRef was rewritten to the hashed name automatically. Change one character of the greeting and the hash changes, the Deployment’s Pod template changes, and a rolling update happens on the next apply — with no rollout restart and no Pod ever running against stale config. That single behaviour is the best argument for Kustomize.

Demo 4: `apply -k` and `diff -k` (≈ 6 min)

What to show / say

Two environments, one source of truth. And the command that answers “what is about to change?” before you find out the hard way.

Steps

kubectl apply -k overlays/dev

namespace/pinboard-dev created
secret/pinboard-db created
configmap/pinboard-api-config-dc8cmkk9c7 created
service/pinboard-api created
service/pinboard-db created
service/pinboard-web created
deployment.apps/pinboard-api created
deployment.apps/pinboard-web created
statefulset.apps/pinboard-db created
ingress.networking.k8s.io/pinboard created

kubectl -n pinboard-dev get deploy,sts,cm

👉 -k is -f for a directory containing a kustomization.yaml. Built into kubectl since 1.14 — nothing to install.

The comparison, client-side, no cluster involved (the grep -v drops the namespace: line that changes on every single object):

diff <(kubectl kustomize overlays/dev) <(kubectl kustomize overlays/prod) \
  | grep -E '^[<>]' | grep -v 'namespace: pinboard-'

<   name: pinboard-dev
>   name: pinboard-prod
<   APP_GREETING: Pinboard DEV — data may be wiped at any time
<   APP_THEME: emerald
<   LOG_FORMAT: text
>   APP_GREETING: Welcome to Pinboard
>   APP_THEME: amber
>   LOG_FORMAT: json
<   name: pinboard-api-config-dc8cmkk9c7
>   name: pinboard-api-config-4fbktcm97t
<   replicas: 1
>   replicas: 3
<             name: pinboard-api-config-dc8cmkk9c7
<         image: pinboard-api:1.0
>             name: pinboard-api-config-4fbktcm97t
>         image: pinboard-api:1.1
<   replicas: 1
>   replicas: 2
> kind: PodDisruptionBudget
...
> kind: HorizontalPodAutoscaler
...
<   - http:
>   - host: pinboard.localtest.me

👉 that is the whole dev/prod delta, and it is auditable in a pull request. Not one manifest was duplicated to produce it. 👉 note the two ConfigMap hashes differ, and both the ConfigMap name and the configMapRef inside the Deployment moved together — in dev and in prod independently.

Now the cluster diff — server-side, against what is actually running:

kubectl -n pinboard-dev set image deploy/pinboard-api api=pinboard-api:1.1   # simulate drift
kubectl diff -k overlays/dev
echo "exit=$?"

-         image: pinboard-api:1.1
+         image: pinboard-api:1.0
exit=1

👉 exit code 1 means “there are differences” — that is a CI gate, and it is also, in one command, what GitOps does continuously. Re-apply to converge:

kubectl apply -k overlays/dev
kubectl diff -k overlays/dev ; echo "exit=$?"

exit=0

Show that prod is only rendered, never applied here:

kubectl kustomize overlays/prod | grep -E '^kind:' | sort | uniq -c

kind: ConfigMap
kind: Deployment
kind: HorizontalPodAutoscaler
kind: Ingress
kind: Namespace
kind: PodDisruptionBudget
kind: Secret
kind: Service
kind: StatefulSet

👉 the two extra kinds are exactly the two prod-only files. And a warning worth ten seconds: an HPA and a git-managed replicas: fight each other — whoever applies last wins. Argo CD calls the fix ignoreDifferences.

Demo 5: GitOps, in one command you do not run (≈ 2 min)

What to show / say

Everything so far has been you pushing to a cluster from a laptop. Name the alternative, show the shape of it, do not install it.

Steps

Say the two commands out loud; do not type them (Argo CD is ~40 Pods and five minutes of pulls):

# NOT run in class:
kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Show the object that replaces kubectl apply -k:

cat <<'EOF'
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: pinboard-dev
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/zonoth/docker-kubernetes-training
    targetRevision: main
    path: demos/session10/kustomize/overlays/dev   # Kustomize, Helm or plain YAML
  destination:
    server: https://kubernetes.default.svc
    namespace: pinboard-dev
  syncPolicy:
    automated:
      prune: true        # delete what git no longer declares
      selfHeal: true     # revert manual kubectl changes
EOF

👉 point at selfHeal: true and connect it to Demo 4 step 3: the drift you created with set image would have been reverted within seconds, by a controller, with no human in the loop. prune: true is the other half — deleting a file deletes the object. 👉 the inversion in one sentence: “nobody has cluster credentials any more; the cluster pulls from git. Your deploy pipeline becomes a pull request.” 👉 name the alternative — Flux — and stop. That is the whole GitOps slide.

Reset

The capstone starts from a blank folder and creates pinboard-prod itself, so it must not exist. The students also install their own ingress controller — keep the Helm release only if you share this cluster with nobody.

cd ~/docker-kubernetes-training/demos/session10/kustomize
kubectl delete -k overlays/dev --ignore-not-found
kubectl delete namespace pinboard-dev --ignore-not-found
kubectl get namespace pinboard-prod    # expect: NotFound

Error from server (NotFound): namespaces "pinboard-prod" not found

Then decide about the controller — one of the two, not both:

# keep the Helm release (recommended: the capstone's Ingress needs a controller)
helm list -A

# or remove it entirely
helm uninstall ingress-nginx -n ingress-nginx && kubectl delete namespace ingress-nginx

kubectl get ns
kubectl config set-context --current --namespace=default

Nothing in this session used the demo namespace — Kustomize demos need their own namespaces to make the overlay difference visible. If Session 09’s reset was skipped, run it now so demo is empty for the next cohort.

Session 10 demos – packaging: Helm, Kustomize, a word about GitOps

Before the session

Demo 1: helm install — the same controller, packaged (≈ 7 min)

What to show / say

Steps

Demo 2: get values, upgrade --set, template (≈ 5 min)

What to show / say

Steps

Demo 3: kubectl kustomize — render an overlay (≈ 5 min)

What to show / say

Steps

Demo 4: apply -k and diff -k (≈ 6 min)

What to show / say

Steps

Demo 5: GitOps, in one command you do not run (≈ 2 min)

What to show / say

Steps

Reset

Demo 1: `helm install` — the same controller, packaged (≈ 7 min)

Demo 2: `get values`, `upgrade --set`, `template` (≈ 5 min)

Demo 3: `kubectl kustomize` — render an overlay (≈ 5 min)

Demo 4: `apply -k` and `diff -k` (≈ 6 min)