57. Triggering Kubernetes rollouts from Gitlab CI

Let’s setup Gitlab CI to trigger a rolling restart of a Kubernetes deployment. This is a lightweight alternative to a something like Flux which requires an operator running in the cluster.

The goal here is to get into the nitty-gritty details, and get something working with few dependencies. Although this is not “best practices” deployment, it’s not far from it, and the setup is easy to extend.

Restarting a Kubernetes deployment with curl

First, we need to figure out how to restart a deployment. We can do it with kubectl rollout restart, but it’d be nice to not require kubectl.

The kubectl codebase is very abstract, and a bit hard to read, but the magic all happens in objectrestarter.go:41. The code adds or updates an annotation with the current date and time. Its key is kubectl.kubernetes.io/restartedAt, but it could be anything. The idea here is that, by updating the deployment’s template, all the pods become out of date, so Kubernetes will do a rolling restart of them.

We can do this ourselves with curl, a bunch of authorization flags, and a JSON patch:

$ curl 'https://kube.eu1:6443/apis/apps/v1/namespaces/default/deployments/scvalex-net' \
  --insecure \
  --cert path/to/cert.pem \
  --key path/to/key.pem \
  -X PATCH \
  -H "Accept: application/json" \
  -H "Content-Type: application/strategic-merge-patch+json" \
  --data "@-" <<EOF
{"spec": {"template": {"metadata": {"annotations": {"scvalex.net/restartedAt": "$(date +%Y-%m-%d_%T%Z)"}}}}}
EOF

Breaking this down line-by-line, we have:

• The URL is the address of the Kubernetes API server (kube.eu1:6443) followed by the path to the resource we’re changing. The format is specified in the reference docs. It contains the API namespace (apps/v1), the resource namespace (default), the kind of resource (deployments), and the resource name (scvalex-net).

• We use --insecure because the API server presents a TLS certificate signed by a non-standard CA. We could instead use --cacert path/to/ca-cert.pem if we have the CA certificate on hand.

• We use --cert and --key to specify the certificate and key that the client is using. We can get these from the client-certificate and client-key fields in ~/.kube/config (or wherever our KUBECONFIG is located). If we instead had client-certificate-data and client-key-data, we’d first have to base64 decode those values with a command like echo "LONG ENCODED STRING" | base64 -d > /tmp/cert.pem. The alternative to this certificate-based authentication is using bearer tokens, and we’ll see how those work later.

• We use -X PATCH to specify the HTTP method. This value again comes from the reference docs.

• We use -H "Accept: application/json" to indicate we want a JSON response.

• We use -H "Content-Type: application/strategic-merge-patch+json" to tell Kubernetes how we want it to handle the data we’re sending. Strategic merge just means that Kubernetes has in-built knowledge about how certain fields are supposed to work. For instance, it knows that a new annotation should be appended to the list of annotations, but an existing annotation needs to have its value changed.

• We send the data with --data "@-". The "@" tells curl to read the data from a file, and the "-" tells it that the file is STDIN. The following heredoc is the data to be sent.

The patch we send is essentially a fragment of a deployment spec, except it’s in JSON instead of YAML. Since this is a heredoc, we can shell out to date to get the current date and time. The name and value of the annotation don’t matter, as long as the name isn’t used by anything else, and the value is a fresh string every time.

{
  "spec": {
    "template": {
      "metadata": {
        "annotations": {
          "scvalex.net/restartedAt": "$(date +%Y-%m-%d_%T%Z)"
        }
      }
    }
  }
}

Permissions in a RBAC world

The above command works because I’m an admin in my cluster, and I am authorized to do anything. In order to enable Gitlab CI to do restarts as well, we need to create an account for it, and then authorize it to patch deployments.

This is a machine user, so we create a ServiceAccount. To keep related things together, we create it in the same namespace as the gitlab-runner pods.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: gitlab-deployer
  namespace: gitlab-runner-ab

The cluster is using Role Based Access Control (RBAC), so we can’t give the service account permissions directly. Instead, we first define a role with the permissions, then bind the account to it. We have to pick between a Role which is bound to a namespace, and a ClusterRole which is available in all namespaces. We choose the latter since nothing changes in the permissions from one namespace to another.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: deployment-restarter
rules:
- apiGroups: ["apps"]
  resources: ["deployments", "statefulsets"]
  verbs: ["get", "patch"]

Finally, we bind the service account to our ClusterRole. The binding is scoped to the default namespace because that’s where our deployment is running. If we wanted to authorize gitlab-deployer to change deployments in other namespaces, we’d need to create a RoleBinding in each of those as well. Or, we could use a ClusterRoleBinding to authorize it across all namespaces, but it seems a bit dodgy for our CI to, for instance, be allowed to change the coredns deployment in the kube-system namespace.

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: restart-deployments-in-default
  namespace: default
subjects:
- kind: ServiceAccount
  name: gitlab-deployer
  namespace: gitlab-runner-ab
roleRef:
  kind: ClusterRole
  name: deployment-restarter
  apiGroup: rbac.authorization.k8s.io

We now have our service account setup and properly authorized. The last bit we need is the authentication token, so that our CI job can identify itself to the API server. We get that with kubectl in two steps. First, we need the name of the secret which contains the token:

### Get the token name with describe
$ kubectl describe serviceaccounts -n gitlab-runner-ab gitlab-deployer
Name:                gitlab-deployer
Namespace:           gitlab-runner-ab
Labels:              <none>
Annotations:         <none>
Image pull secrets:  <none>
###                  👇           👇           👇
Mountable secrets:   gitlab-deployer-token-vkrw6
Tokens:              gitlab-deployer-token-vkrw6
Events:              <none>

### Or, get the token name with jq
$ kubectl get serviceaccounts -n gitlab-runner-ab gitlab-deployer -o json \
  | jq '.secrets[].name'
"gitlab-deployer-token-vkrw6"

Next, we get the value from the secret, and base64-decode it:

### Get the token value with describe
$ kubectl describe secret gitlab-deployer-token-vkrw6 -n gitlab-runner-ab
Name:         gitlab-deployer-token-vkrw6
Namespace:    gitlab-runner-ab
Labels:       <none>
Annotations:  kubernetes.io/service-account.name: gitlab-deployer
              kubernetes.io/service-account.uid: 2a8f7755-704a-4a47-ab70-e350c86fc955

Type:  kubernetes.io/service-account-token

Data
====
ca.crt:     1952 bytes
namespace:  16 bytes
###         👇              👇                👇
token:      eyJhbGciOiJSUzI1NiIsImtpZCI6InlWVUNIMVdJQX...

### Or, get the token value with jq
$ kubectl get secrets -n gitlab-runner-ab gitlab-deployer-token-vkrw6 -o json \
  | jq '.data.token | @base64d'
"eyJhbGciOiJSUzI1NiIsImtpZCI6InlWVUNIMVdJQX..."

The result is a long string representing a JSON Web Token. We put this “bearer” token in the request headers in order to authenticate. Practically, this replaces the --cert and --key flags of our curl command:

$ curl 'https://kube.eu1:6443/apis/apps/v1/namespaces/default/deployments/scvalex-net' \
  --insecure \
###  👇 this replaces --cert and --key 👇
  -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6InlWVUNIMVdJQX..." \
  -X PATCH \
  ...
EOF

Gitlab CI config

We have the command to trigger a restart, and we have the service account authorized to do this. All that’s left is to plug this into a Gitlab CI configuration. But first, what’s the address of the Kubernetes API server from within the CI job?

In my setup, the gitlab-runner doing the CI builds is in the same cluster as the deployment I want to restart, so I can just use the kubernetes.default.svc address.

Alternatively, if the API server were accessible from our Gitlab instance, we could just use its address. For instance, GKE clusters are accessible from the Internet, so we just have an IP and port to connect to. If we were running Gitlab on the same private network as our Kubernetes cluster, then an internal name would also work.

Here’s the full restart script. It’s the same as before, but we’re hiding most of curl’s output with --silent and --show-error, and we take the service account token from the GITLAB_DEPLOYER_TOKEN environment variable.

#!/bin/sh

set -e -x

curl "https://kubernetes.default.svc/apis/apps/v1/namespaces/default/deployments/scvalex-net" \
     --insecure \
     --silent \
     --show-error \
     -H "Authorization: Bearer $GITLAB_DEPLOYER_TOKEN" \
     -X PATCH \
     -H "Accept: application/json" \
     -H "Content-Type: application/strategic-merge-patch+json" \
     --data "@-" <<EOF
{
  "spec": {
    "template": {
      "metadata": {
        "annotations": {
          "scvalex.net/restartedAt": "$(date +%Y-%m-%d_%T%Z)"
        }
      }
    }
  }
}
EOF

Next, we configure the GITLAB_DEPLOYER_TOKEN variable in the Gitlab UI. It’s under Settings→CI/CD→Variables.

Finally, we add the deployment job to our .gitlab-ci.yml. The environment section is what makes this job a deployment in Gitlab’s eyes.

deploy:
  image: "curlimages/curl:7.81.0"
  stage: deploy
  only:
    - master
  environment:
    name: prod
    url: https://scvalex.net
  script:
    - cd $CI_PROJECT_DIR
    - ./trigger-restart.sh

Conclusion

That’s all there is to it. We’ve seen how to write a script to trigger a rolling restart in Kubernetes, how to add and authorize a service account to use it, and how to add it to a Gitlab CI job.

Mind you, the deployment for my website just uses the :latest container image, so triggering a restart after a build is enough to update the website to the new version. A better solution would be to tag the images with the commit hash or version, and then have the script patch the image label in the deployment template. Going back to the point that this post doesn’t describe best practices, I think using a commit hash or version would get you most of the way there. It would certainly be enough to get the full benefit from Gitlab’s environments, including rollback from the UI.