Use Kind and K3d With Tilt and Clk K8s

use kind and k3d with tilt and clk k8s

k3d is great, and support a local registry out of the box. But the time to push at every change still takes some time. Unfortunately, it does not support kubectl build.

kind does not support easily a local docker registry, but it has a way to directly push images. Unfortunately, this is very slow. We can use kubectl build to mitigate this issue.

When you want to play with tilt. You first configure a docker_build call. To have a fast code, update, repeat loop, you can:

1. run a local registry, then after each change a new image will be built, pushed and then pulled from the cluster,
2. replace docker_build by image_build (from the kubectl build extension). That way, the image will be built directly into the cluster and won’t need to go through the push and pull phases.
3. use live update, to avoid restarting a container

2 is a little bit faster than 1, because it prevents the push and pull phases. 3, by design is the fastest. I cannot imagine a way faster than 3, as it allow you to change only the pieces of information that changed.

While 3 is very tempting, it eventually makes things harder to maintain, so I recommend avoiding using it as far as possible and falling back in it only if 1 or 2 are not satisfying.

It might be tempting to make 2 the default choice then, but it has some drawbacks.

First, it is not supported in k3d.

Also, when you want to use an images as basis of another (using multi stage build), you cannot make use of custom builds (hence kubectl build). This is because tilt cannot make any assumption about what the custom build will do, not even that the custom builder will use dockerfile at all.

I personally don’t want the tiltfile to be tainted by the underlying cluster technology. But I don’t see how to prevent this. The least I can do is making it versatile enough.

I setup my clusters using the awesome clk k8s tool. Not only does it provides out of the box clusters, but is also gives extra information about their features.

To enable the kubectl build feature when available, I can simply provide the following code in my tiltfile.

load('ext://kubectl_build', 'image_build', 'kubectl_build_enable')
kubectl_build_enable(
    local('clk k8s features --field value --format plain kubectl_build',
          quiet=True))

Enable kubectl_build is global. If not enabled, image_build will simply fall back on docker_build.

If I write my docker builds with the simple docker_build instruction, I cannot make use of kubectl build. I could replace docker_build by image_build, but that would enable it even when I want to use dependent images.

Thus, images that have no base images built in the project can use image_build, while the other must use docker_build.

In some of my projects, building the base image is conditional. In that case, my code looks like this.

def has_local_registry():
    return str(
        local('clk k8s features --field value --format plain local_registry',
              quiet=True)).strip() == "True"

if use_dependent_images_p():
    builder = docker_build
    if not has_local_registry():
        warn("You plan to work on dependent images, then I need to use docker_build.\
But you don't have a local registry. It will be veryyyy sloooow.")
else:
    builder = image_build

Permalink