I am going to describe how I store all of my Kubernetes manifests in a single directory tree. The overriding goals of this setup is to commit all of the manifests to version control, and to apply them with a single command.
The manifests directory ends up looking like this:
kube ├── Makefile ├── kustomization.yaml ├── ingresses.yaml ├── coredns │ ├── coredns.yaml │ └── kustomization.yaml ├── ingress-nginx │ ├── ingress-nginx.yaml │ ├── kustomization.yaml │ ├── Makefile │ ├── namespace.yaml │ └── values.yaml ├── umami │ ├── 01-namespace.yml │ ├── 02-secret.yml │ ├── 03-postgres.yml │ ├── 04-umami.yml │ └── kustomization.yaml ... more subdirs ...
We have a subdirectory for each app. There’s a loose
ingresses.yaml at the top-level because I like to list all the ingresses in one place. The top-level and the subdirectories each contain a
kustomization.yaml which lists the per-app manifest files, and overrides the
namespace for all objects. Additionally, there’s a recursive
Makefile to handle updating the files when they’re generated or fetched.
Kustomize is part of
kubectl, and it has a lot of features. We’re going to focus on just the ones that enable our setup. As a basis for discussion, let’s look at the top-level
kustomization.yaml, and the one for
apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - coredns/ - ingress-nginx/ - umami/ - ingresses.yaml
The top-level kustomization just lists the subdirectories and the
ingresses.yaml file. The interesting bit here is that the
resources stanza accepts not only files, but also kustomization directories. When applied with
kubectl apply -k, kustomize will read through all the resources in the tree, and figure out the right global order to apply the manifests in (e.g. namespaces first, then CRDs, then service accounts, and so on).
apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization namespace: ingress-nginx resources: - namespace.yaml - ingress-nginx.yaml
The kustomization for
ingress-nginx does two things. First, it overrides the namespace of any included resources with the
namespace directive. This is very useful because it means we don’t have to specify the
namespace field on each object in the manifests. For
ingress-nginx, the manifest isn’t actually written by us, but is instead generated from a Helm chart, so we don’t even have the option of specifying the namespace.
Secondly, the kustomization lists the resource files to apply. We have a
namespace.yaml with just the namespace definition, and the generated
ingress-nginx.yaml. Since we generate the latter, we want to avoid modifying it directly, so being able to list multiple resource files is a must.
As a bonus, because the kustomization explicitly lists all the resource files, it means we can have other YAML files in the same directory without worrying that
kubectl apply -f might try to apply them. In our example, we have
values.yaml with configuration for the Helm chart.
Some of the apps like Umami have fully hand-written manifests. In other cases like CoreDNS, I just copied the example manifest from the docs. But many of them are generated from Helm charts, and need to be regenerated whenever the version changes. We do this with
Makefiles. Here’s the top-level one, and the one for
update: make -C ingress-nginx/ update ... more recursive make calls ...
update: helm repo add ingress-nginx "https://kubernetes.github.io/ingress-nginx" || true helm repo update ingress-nginx helm template ingress-nginx ingress-nginx/ingress-nginx \ --namespace ingress-nginx \ --version 4.0.13 \ -f values.yaml \ > ingress-nginx.yaml
Makefile just recurses into the subdirectories. Our per-app
Makefile first ensures that the Helm repo is added, then updates it, and finally calls
helm template to generate the manifests.
With this setup, we can run
make update in the top-level directory to ensure all the manifests are generated, and then run
kubectl apply -k to apply everything. This is an idempotent operation, so we can run it as many times as we want when making infrastructure changes.