52. Single directory Kubernetes manifests

I am going to describe how I store all of my Kubernetes manifests in a single directory tree. The overriding goals of this setup is to commit all of the manifests to version control, and to apply them with a single command.

The end result

The manifests directory ends up looking like this:

kube
├── Makefile
├── kustomization.yaml
├── ingresses.yaml
├── coredns
│   ├── coredns.yaml
│   └── kustomization.yaml
├── ingress-nginx
│   ├── ingress-nginx.yaml
│   ├── kustomization.yaml
│   ├── Makefile
│   ├── namespace.yaml
│   └── values.yaml
├── umami
│   ├── 01-namespace.yml
│   ├── 02-secret.yml
│   ├── 03-postgres.yml
│   ├── 04-umami.yml
│   └── kustomization.yaml
... more subdirs ...

We have a subdirectory for each app. There’s a loose ingresses.yaml at the top-level because I like to list all the ingresses in one place. The top-level and the subdirectories each contain a kustomization.yaml which lists the per-app manifest files, and overrides the namespace for all objects. Additionally, there’s a recursive Makefile to handle updating the files when they’re generated or fetched.

Kustomize

Kustomize is part of kubectl, and it has a lot of features. We’re going to focus on just the ones that enable our setup. As a basis for discussion, let’s look at the top-level kustomization.yaml, and the one for ingress-nginx.

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - coredns/
  - ingress-nginx/
  - umami/
  - ingresses.yaml

The top-level kustomization just lists the subdirectories and the ingresses.yaml file. The interesting bit here is that the resources stanza accepts not only files, but also kustomization directories. When applied with kubectl apply -k, kustomize will read through all the resources in the tree, and figure out the right global order to apply the manifests in (e.g. namespaces first, then CRDs, then service accounts, and so on).

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: ingress-nginx
resources:
  - namespace.yaml
  - ingress-nginx.yaml

The kustomization for ingress-nginx does two things. First, it overrides the namespace of any included resources with the namespace directive. This is very useful because it means we don’t have to specify the namespace field on each object in the manifests. For ingress-nginx, the manifest isn’t actually written by us, but is instead generated from a Helm chart, so we don’t even have the option of specifying the namespace.

Secondly, the kustomization lists the resource files to apply. We have a namespace.yaml with just the namespace definition, and the generated ingress-nginx.yaml. Since we generate the latter, we want to avoid modifying it directly, so being able to list multiple resource files is a must.

As a bonus, because the kustomization explicitly lists all the resource files, it means we can have other YAML files in the same directory without worrying that kubectl apply -f might try to apply them. In our example, we have values.yaml with configuration for the Helm chart.

Recursive Makefile for updates

Some of the apps like Umami have fully hand-written manifests. In other cases like CoreDNS, I just copied the example manifest from the docs. But many of them are generated from Helm charts, and need to be regenerated whenever the version changes. We do this with Makefiles. Here’s the top-level one, and the one for ingress-nginx:

update:
        make -C ingress-nginx/ update
    ... more recursive make calls ...

update:
        helm repo add ingress-nginx "https://kubernetes.github.io/ingress-nginx" || true
        helm repo update ingress-nginx
        helm template ingress-nginx ingress-nginx/ingress-nginx \
          --namespace ingress-nginx \
          --version 4.0.13 \
          -f values.yaml \
          > ingress-nginx.yaml

The top-level Makefile just recurses into the subdirectories. Our per-app Makefile first ensures that the Helm repo is added, then updates it, and finally calls helm template to generate the manifests.

Conclusion

With this setup, we can run make update in the top-level directory to ensure all the manifests are generated, and then run kubectl apply -k to apply everything. This is an idempotent operation, so we can run it as many times as we want when making infrastructure changes.