Clk K8s and Earthly in a Local Dev Env

I use clk k8s to bootstrap a local k8s cluster and earthly to run the tests.

I found out that it was easy to forget who can communicate with whom in a such a stack with several layers of network¹.

Because all of this is running in separate network environment, I need to make sure that all the parts can communicate with each other correctly.

When playing with a locally deployment application, I want it to be as similar to the production as possible. Therefore, I want to use a public domain name, use https and the ingress to run my tests.

1. in blue in the diagram: I MUST be able to access my program using the official ingress component, without taking any shortcut
2. also in blue in the diagram: my program should be able to access the stack using the ingress as well (also in blue). This is needed in scenario where I host my own authorization server.
3. in orange: in some cases, I may want my program to get access to some service hosted locally (at least some of my colleague believe they need this).
4. in purple: I may want to take some shortcuts and connect a debugging tool directly on my program. This is needed in cases I want to access non officially exposed ports, like when using DAP debugging

in linux

Here, we can see in blue two ways that I use to run tests, either from the host computer or from a reproducible and controller containerised environment. I do the later case, I use earthly. But in some particular cases, I run prebuilt images in a more ad-hoc way.

The deployed stack runs in a kubernetes in docker (using clk k8s).

There are three locations that need to access my exposed API:

1. the host, to run the tests
2. docker containers (and above all the earthly worker) to run the tests
3. the pods themselves to access publicly available services

To make this possible, those are the settings I have to use for those component to have my domain bound to an available network interface

1. fortunately, this is granted for all *.localtest.me addresses
2. with earthly, you can simply use the HOST command, for a more traditional docker run, use –add-host
3. fortunately, the pods have access to the docker network they are in, so you simply need to add the association my domain -> <netwprk gateway> in coredns. In fact, clk k8s add-domain will do that for you, and the function add_domain in the tilt extension as well.

To fulfil the purple line, simple kubectl port-forward are enough, and tilt make this even easier.

The orange path is also easy because the hosted service can bind on the docker network, accessible by the pods.

in mac (without docker desktop)

Here, I use the default addresses of orbstack but the same works for colima (using different addresses).

As you can see, everything is working like in linux, with the exception that the pod needs to use the vm network to get access to a hosted service. Fortunately, this is a scenario that I seldom need and I guess I will be able to mitigate with socat in docker to create a bridge. This may be a bit complicated, but an acceptable trade-of if barely needed.

in docker desktop (so, in windows?)

Here, it works like in the previous but for three exceptions

1. the vm address is not knows in advance, making it harder to associate a custom domain to this ip.
2. the pods cannot access the docker network, needing a more complicated setup, like using two instances of socat (one in the host and one in docker) to create a bridge between the pods and docker.
3. accessing host.docker.internal appears to be dependent on the technology used to get the k8s cluster. With kind, it works, with k3d, it does not. Hence I would not rely on this much.

Thus, I definitely do not recommend using docker engine for use cases such as the one I presented earlier, as communication from the inside of the cluster to the host appears to be at best very complicated.

conclusion

To have a local k8s dev env is very practical as it allows quick development cycles and reproducible tests.

To allow such setup, I warmly recommend using linux as a developer environment. Using a mac may need a bit more work, but that is kinda ok. And using docker engine (and therefore windows) is definitely not worth it.

Notes linking here

• clk k8s: accessing the host from the pod on linux and mac

Permalink