I am going to describe how I store all of my Kubernetes manifests in a single directory tree. The overriding goals of this setup is to commit all of the manifests to version control, and to apply them with a single command.
The end result
The manifests directory ends up looking like this:
kube
├── Makefile
├── kustomization.yaml
├── ingresses.yaml
├── coredns
│ ├── coredns.yaml
│ └── kustomization.yaml
├── ingress-nginx
│ ├── ingress-nginx.yaml
│ ├── kustomization.yaml
│ ├── Makefile
│ ├── namespace.yaml
│ └── values.yaml
├── umami
│ ├── 01-namespace.yml
│ ├── 02-secret.yml
│ ├── 03-postgres.yml
│ ├── 04-umami.yml
│ └── kustomization.yaml
... more subdirs ...
We have a subdirectory for each app. There’s a loose ingresses.yaml
at the top-level because I like to list all the ingresses in one place. The top-level and the subdirectories each contain a kustomization.yaml
which lists the per-app manifest files, and overrides the namespace
for all objects. Additionally, there’s a recursive Makefile
to handle updating the files when they’re generated or fetched.
Kustomize
Kustomize is part of kubectl
, and it has a lot of features. We’re going to focus on just the ones that enable our setup. As a basis for discussion, let’s look at the top-level kustomization.yaml
, and the one for ingress-nginx
.
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- coredns/
- ingress-nginx/
- umami/
- ingresses.yaml
kube/kustomization.yaml
The top-level kustomization just lists the subdirectories and the ingresses.yaml
file. The interesting bit here is that the resources
stanza accepts not only files, but also kustomization directories. When applied with kubectl apply -k
, kustomize will read through all the resources in the tree, and figure out the right global order to apply the manifests in (e.g. namespaces first, then CRDs, then service accounts, and so on).
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: ingress-nginx
resources:
- namespace.yaml
- ingress-nginx.yaml
kube/ingress-nginx/kustomization.yaml
The kustomization for ingress-nginx
does two things. First, it overrides the namespace of any included resources with the namespace
directive. This is very useful because it means we don’t have to specify the namespace
field on each object in the manifests. For ingress-nginx
, the manifest isn’t actually written by us, but is instead generated from a Helm chart, so we don’t even have the option of specifying the namespace.
Secondly, the kustomization lists the resource files to apply. We have a namespace.yaml
with just the namespace definition, and the generated ingress-nginx.yaml
. Since we generate the latter, we want to avoid modifying it directly, so being able to list multiple resource files is a must.
As a bonus, because the kustomization explicitly lists all the resource files, it means we can have other YAML files in the same directory without worrying that kubectl apply -f
might try to apply them. In our example, we have values.yaml
with configuration for the Helm chart.
Recursive Makefile
for updates
Some of the apps like Umami have fully hand-written manifests. In other cases like CoreDNS, I just copied the example manifest from the docs. But many of them are generated from Helm charts, and need to be regenerated whenever the version changes. We do this with Makefiles
. Here’s the top-level one, and the one for ingress-nginx
:
update:
make -C ingress-nginx/ update
... more recursive make calls ...
kube/Makefile
update:
helm repo add ingress-nginx "https://kubernetes.github.io/ingress-nginx" || true
helm repo update ingress-nginx
helm template ingress-nginx ingress-nginx/ingress-nginx \
--namespace ingress-nginx \
--version 4.0.13 \
-f values.yaml \
> ingress-nginx.yaml
kube/ingress-nginx/Makefile
The top-level Makefile
just recurses into the subdirectories. Our per-app Makefile
first ensures that the Helm repo is added, then updates it, and finally calls helm template
to generate the manifests.
Conclusion
With this setup, we can run make update
in the top-level directory to ensure all the manifests are generated, and then run kubectl apply -k
to apply everything. This is an idempotent operation, so we can run it as many times as we want when making infrastructure changes.