Loading dynamic configurations in Kubernetes Kustomize

However, there are times when it’s useful to be able to load configurations dynamically in Kustomize. In this article, I’ll present a use-case for dynamic configurations and methods for loading them.

A lot of times, I work with projects that use push-based Continuous Deployment with Kubernetes. Once a pull request is accepted, it’s merged to master, and a pipeline job is triggered for it automatically. Here’s what typically happens in the pipeline:

The Git commit hash provides a simple, convenient, and effective versioning strategy. It’s unique enough to distinguish each version from each other, and we can easily trace each version back to the source code.

Most importantly, the Git commit hash is something we can derive from the source repository. Once changes have been accepted, the version number is auto-generated without any additional manual steps. Even though changing a version number is a relatively small task compared to the rest of the development process, it adds up quickly in projects where there are frequent changes. I’ve found this to be an important factor in promoting rapid releases in teams.

The trade-off for using the hashes as versions is that they lack semantics for ordering and compatibility. In other words, you can’t tell which version comes after which just by looking at the version strings, and you can’t tell whether the changes between two versions broke compatibility or not. However, I’ve found these qualities to be something I can work without in practice when working with service-based software development.

Additionally, there’s a step in the pipeline to generate the final Kubernetes manifests that can be deployed to Kubernetes using kubectl. However, since this is only a demo, the actual deployment part is left out as an exercise for the reader.

First, let’s have a look at the Kustomize project. To keep the demo simple, we’ll only create a Service and a Deployment for the app in the k8s-base directory. Here’s what the Service looks like.

These are then bound to the Kustomization resource, which also sets some of the common settings such as labels. It’s placed next to the manifests in the file named kustomization.yaml.

With the files in place, we should be able to generate all the manifests using command kubectl kustomize k8s-base.

Now that we have a Kustomize project to work with, we need a way to feed the extra configurations to it via files.

One of the classic ways to inject configurations to files is to use a search and replace tools such as sed. For example, we could include placeholders in the kustomization.yaml file, but then replace the placeholders with the actual configurations. Let’s try this out by adding a placeholder for the Docker image tag in the images field.

If you look at the kustomization.yaml file, you should see the Git commit hash where the placeholder was.

The above script works when it’s run only once but it doesn’t work if you try to run it again with a different Git commit. This is because the placeholder no longer exists in the file.

We can get around this limitation by creating a dedicated template file that contains the placeholders.

We can now safely generate the final kustomization.yaml file as many times as we like from the template by forwarding the sed output to a separate file instead of replacing the contents in place.

Let’s try it out by creating a new template file. Pay attention to how the placeholder will now use the environment variable syntax.

We can provide this template to the envsubst command. Note that all of the shell variables we want to use in the template must be exported as environment variables.

By using the resources feature, we can host our dynamic configurations in a temporary project that extends the manifests in the Git repository. We can automate this using a shell script.

We can automate this using a shell script.

Let’s go through the script step by step by starting with reading the Git commit hash as we’ve done before.

Next, we’ll create a temporary directory next to the k8s-base directory to host our dynamic configurations and set it up for deletion after the script ends.

The mktemp command creates a directory with a random name, so we can call this script multiple times simultaneously without race conditions. The trap command allows us to schedule commands to be run every time a script exits. It runs even when the script ends with an error. In this case, we use trap to clean up the temporary directory.

We’ll use cat to generate the kustomization.yaml file for the temporary Kustomize project. It’s set to use the k8s-base directory as the base project and configures the image tag to use the Git commit hash we read earlier.

Finally, we’ll generate the manifests from the temporary project using the Kustomize command in kubectl. The manifests can either piped to a file or directly to the kubectl apply -f - command to deploy the resources in Kubernetes.

With this approach, we can keep the templating part separate from the Kustomize project stored in the Git repository. The script still does do templating, but it’s limited to the temporary project. In larger projects, this can be easier to reason with because you know where all templating is done.

The CLI tool includes a command for editing Kustomize projects from the command line, which we can use to feed in our dynamic configurations. Specifically, we can use the edit set image sub-command to set our image tag in the k8s-base directory.

Like in the first sed example, this command will update the kustomization.yaml file in place. However, unlike in the sed example, we can repeat the command as many times as we like.

We can also combine the Kustomize CLI tool with the temporary project approach by replacing the cat command with calls to the CLI tool.

Like earlier, we create a new temporary directory to host the temporary project. In that directory, we create a new project based on the k8s-base directory using the kustomize create command and add the image configuration. Finally, we use kustomize build to generate the Kubernetes manifests. All of these commands are run in a sub-shell to ensure the temporary directory can be deleted in the end.

In this article, I presented a use-case for when dynamic configurations can be really useful, and how we can load them in Kustomize via templating, temporary projects, and the Kustomize CLI tool.

Related posts:

Vermicomposting- Is the process of using various species of worms, usually red wigglers, white worms, and other earthworms, to create a mixture of decomposing vegetable or food waste, bedding…

Given a string with lowercase letters only, if you are allowed to replace no more than ‘k’ letters with any letter, find the length of the longest substring having the same letters after replacement…

The room is full of people sitting on cushions with their legs crossed. We will sit like this for forty minutes as we meditate together. After about twenty minutes, I hear a snore. I ignore it. It’s…