51. Helm install and template

Helm is an easy way of deploying to Kubernetes, but helm install is a bit annoying because it doesn’t save the changes it made to a local repo. That’s where helm template comes in.

helm install vs helm template

As a concrete example, let’s talk about installing Gitlab Runner from the Helm chart. This is the part of Gitlab that runs CI builds. Hosting it ourselves is cheaper (and faster) than using gitlab.com. Following the docs, we’re supposed to install it like this:

$ helm repo add gitlab https://charts.gitlab.io/
$ helm repo update
$ helm install gitlab-runner gitlab/gitlab-runner \
    --namespace gitlab-runner-ab \
    --create-namespace \
    --version 0.35.3 \
    -f gitlab-runner-values-ab.yaml

This is easy to run, but now we have to remember to run it every time we rebuild the cluster. We can no longer just kubectl apply a directory to have everything installed.

Secondly, it’s unfortunate that we don’t have a local listing of what changes were made to the cluster. Kubernetes makes it very easy to describe what components a system has, and it’s sad to lose that. Installing charts feels like going back to the dark days of looking at a server, and wondering what was apt-get install’d, and what configuration files where changed.

Enter helm template. It takes similar arguments to helm install, but instead of changing the cluster, it prints out a YAML with the changes. The command is usually mentioned in the context of writing charts, and inspecting their output, but there’s no reason we can’t use it with somebody else’s charts:

$ helm template gitlab-runner gitlab/gitlab-runner \
    --namespace gitlab-runner-ab \
    --version 0.35.3 \
    -f gitlab-runner-values-ab.yaml \
    > gitlab-runner.yaml
### We save the output to a file to apply later.
$ git add gitlab-runner.yaml
$ kubectl apply \
    --namespace gitlab-runner-ab \
    -f gitlab-runner.yaml

The template command is something we’ll run again whenever we want to change the version, so let’s put it in a Makefile:

update:
        helm repo add gitlab "https://charts.gitlab.io/" || true
        helm repo update gitlab
        helm template gitlab-runner gitlab/gitlab-runner \
          --namespace gitlab-runner-ab \
          --version 0.35.3 \
          -f gitlab-runner-values-ab.yaml \
          > gitlab-runner.yaml

Important helm template flags

A few caveats come with using helm template. The first is that Helm charts have hooks which only run at specific Helm lifecycle events like install, upgrade, or delete. By default, helm template dumps all of them, so if we’re not careful, we might end up with uninstall batch jobs in our manifest (e.g. the Longhorn chart does exactly this). We need to pass the --no-hooks flag to disable this behaviour, and then we have to be careful to run any necessary hooks ourselves when appropriate.

The second caveat is that we need an extra flag to output CRDs. The Helm best practices recommend putting CRD declarations in a separate directory instead of interleaving them with other resources in the chart. By default, helm template doesn’t output these. We need to pass the --include-crds flag to change this behaviour.

Namespaces

Another problem is that --create-namespace doesn’t work with helm template, so the namespace ends up missing. We can create it ourselves with a kustomization:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: gitlab-runner-ab
resources:
  - namespace.yaml
  - gitlab-runner.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: gitlab-runner.yaml

Conclusion

Putting it all together, we end up with the following directory layout:

gitlab-runner
├── gitlab-runner-values-ab.yaml
├── gitlab-runner.yaml
├── kustomization.yaml
├── Makefile
└── namespace.yaml

We apply this with kubectl apply -k gitlab-runner/, and we regenerate the manifest from the chart with make -C gitlab-runner/ update. We can also add this directory to a higher level kustomization, so that installing everything into the cluster is just one command:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - gitlab-runner/
  - ... other apps to deploy in the cluster ...

Doing all this, we lose Helm’s version tracking and upgrade/rollback support. On the other hand, since we store all the configuration in a repo, we can easily reason about version changes ourselves. I personally find this less stressful than hoping that Helm and the chart work properly.