clk k8s is a tool meant to ease the creation and maintenance of a dev k8s stack.
When I deploy a local k8s stack, I sometimes want the pods to access the host. Because some services need to discuss with others using the public API, hence the ingress, hence the docker published ports.
Note that I wrote another, more generic article that may help understand this one.
One classical example is authorization. The servers should call the authorization server to check the validity of the access token, using some public address. In the local development stack, it makes sense to deploy this authority locally, in another docker container or helm application. And it makes more sense to discuss with it using a “public” interface rather than an internal network.
In more concrete term, you configure your ingresses to bind to custom domain
names that redirect to 127.0.0.1, like mystack.localtest.me. Then, you use the
command clk k8s add-domain mystack.localtest.me to make this domain actually
resolve to the host from the perspective of the pods. That way, every actors of
your program, whether it runs on the host or inside the k8s stack, can connect
and use the exposed services.
As it happens, this is not as simple as it seems.
in linux
So far, I used the gateway IP of the docker bridge network as IP address to connect to when wanting to access host services.
Let’s simplify the scenario by running a simple nc command on the host.
while true; do nc -l -p 9999; done
Then, you can generally access it using the bridge gateway IP.
docker run alpine sh -c "nc 172.17.0.1 9999 < <(date)"
You should see the date appear on the listening side of the connection.
This also works from a pod running in the local cluster.
kubectl create deployment --image nginx nginx
kubectl exec deployment/nginx -- apt update
kubectl exec deployment/nginx -- apt install netcat-traditional
kubectl exec deployment/nginx -- nc 172.17.0.1 9999 < <(date)
This also shows the date on the listening side.
There is nothing difficult here.
in mac
The listening nc command needs to be slightly different.
while true; do nc -l 0.0.0.0 9999; done
with docker desktop
When people install docker in mac, they generally use the official docker desktop one. So let’s try it first.
docker run alpine sh -c "nc 172.17.0.1 9999 < <(date)"
This does nothing and fails.
They recommend using host.docker.internal.
Let’s try it.
docker run alpine sh -c "nc host.docker.internal 9999 < <(date)"
This one works.
So far so good. Let’s try it in the cluster.
clk k8s create-cluster --flow
kubectl create deployment --image nginx nginx
kubectl exec deployment/nginx -- apt update
kubectl exec deployment/nginx -- apt install netcat-traditional
kubectl exec deployment/nginx -- nc host.docker.internal 9999 < <(date)
It times out.
We can see that the ip address is correctly resolved though.
kubectl exec deployment/nginx -- apt install -y bind9-dnsutils
kubectl exec deployment/nginx -- dig +short host.docker.internal
192.168.65.254
Let’s try it with k3d instead of kind
clk k8s remove
clk k8s --distribution k3d create-cluster --flow
kubectl create deployment --image nginx nginx
kubectl exec deployment/nginx -- apt update
kubectl exec deployment/nginx -- apt install netcat-traditional
kubectl exec deployment/nginx -- nc host.docker.internal 9999 < <(date)
Indeed, host.docker.internal is not even resolved.
Let’s make sure of this.
kubectl exec deployment/nginx -- apt install -y bind9-dnsutils
kubectl exec deployment/nginx -- dig +short host.docker.internal
There is no returned ip address, showing that host.docker.internal does not resolve.
Let’s try using its IP address directly.
kubectl exec deployment/nginx -- nc 192.168.65.254 9999 < <(date)
Again, time out.
In fact, even though it is published on the host, the hosted service is generally run in docker as well.
So let’s try with the server part running inside docker.
docker run -i -t --rm -p 9999:9999 alpine sh -c "while true;do /usr/bin/nc -l -p 9999 -e echo hi;done"
docker run alpine sh -c "nc ${ip} 9999 < /dev/null"
This also times out.
Using the host.docker.internal address
docker run alpine sh -c "nc ${ip} 9999 < /dev/null"
hi
And again in k8s
kubectl create deployment --image nginx nginx
kubectl exec deployment/nginx -- apt update
kubectl exec deployment/nginx -- apt install netcat-traditional
kubectl exec deployment/nginx -- nc host.docker.internal 9999 < /dev/null
hi
So, using host.docker.internal works in k8s, using kind.
With k3d though, it still does not find the host.docker.internal address and times out with 192.168.65.254.
We could dig into the networking stuff, but I don’t think that it is worth it, because they are other more promising docker stacks. Also, the host.docker.internal name is not practical for it makes more complicated the scenario mentioned above with a custom domain like mystack.localtest.me.
Also, I don’t like docker desktop much, because it can only be managed using the graphical user interface, which is not IaC friendly.
trying with colima
/Users/sam/.nix-profile/bin/colima start --network-address
...
> Successfully created context "colima"
> Current context is now "colima"
Let’s first try with the default bridge gateway ID address
docker network inspect bridge | /Users/sam/.nix-profile/bin/jq -r ".[0].IPAM.Config[0].Gateway"
172.17.0.1
docker run alpine sh -c "nc ${ip} 9999 < <(date)"
It fails.
This is expected, because colima runs inside a virtual machine, it can access the host only using the colima network interface created for this.
/Users/sam/.nix-profile/bin/colima list --json | /Users/sam/.nix-profile/bin/jq -r .address
192.168.107.2
Let’s try with this interface, substituting 2 by 1 to get to the host.
docker run alpine sh -c "nc ${ip%*.*}.1 9999 < <(date)"
I can see the date in the listening side, meaning that it works.
But that is not practical, because I have to run a specific colima command to know the ip address.
with the listening part running with docker run -p
As explained earlier, most of my tools are running in docker. So the scenario that I am more interested in is when the server part is also a container.
First, let’s run the nc listening part in docker.
docker run -i -t --rm -p 9999:9999 alpine sh -c "while true;do /usr/bin/nc -l -p 9999 -e echo hi;done"
First, let’s discuss with it from the host.
nc localhost 9999 < /dev/null
hi
That’s ok, now let’s try to discuss with it using the bridge gateway IP from a docker container.
docker run alpine sh -c "nc ${ip} 9999 < /dev/null"
hi
That sounds promising, let’s try with the colima address.
docker run alpine sh -c "nc ${ip} 9999 < /dev/null"
hi
Now, let’s try from another network.
docker network create a
docker run --network a alpine sh -c "nc ${ip} 9999 < /dev/null"
hi
Great. Let’s just try with the colima ip address as well.
docker run --network a alpine sh -c "nc ${ip} 9999 < /dev/null"
hi
Everything works as expected here.
with k8s
Now, let’s try from a k8s deployment.
/Users/sam/.local/bin/clk k8s --distribution kind create-cluster
kubectl create deployment --image nginx nginx
kubectl exec deployment/nginx -- apt update
kubectl exec deployment/nginx -- apt install netcat-traditional
kubectl exec deployment/nginx -- nc "${ip}" 9999 < <(date)
hi
And using the colima address
kubectl exec deployment/nginx -- nc "${ip}" 9999 < <(date)
hi
Sound like everything is working well.
trying with orb
docker network inspect bridge | /Users/sam/.nix-profile/bin/jq -r ".[0].IPAM.Config[0].Gateway"
192.168.215.1
That is not practical as a lot of tools assume this IP address is 172.17.0.1.
Running the listening part in the host
while true; do nc -l 0.0.0.0 9999; done
docker run alpine sh -c "nc ${ip} 9999 < <(date)"
It fails. Like in colima, the virtual machine prevents accessing the host from the container using the gateway ip.
Now, trying from docker to docker.
The listening part now looks like (I took the liberty of putting it already in a separate network to test this case as well):
docker run -i -t --rm --network b -p 9999:9999 alpine sh -c "while true;do /usr/bin/nc -l -p 9999 -e echo hi;done"
Then,
docker run alpine nc "${ip}" 9999 < /dev/null
hi
Great. Now from k8s
kubectl create deployment --image nginx nginx
kubectl exec deployment/nginx -- apt update
kubectl exec deployment/nginx -- apt install netcat-traditional
kubectl exec deployment/nginx -- nc "${ip}" 9999 < /dev/null
hi
So it looks like orb also deals correctly with connecting the pods using the gateway ip.
conclusion
To ease developing in a k8s dev stack, I would definitely avoid docker desktop.
orb and colima work quite well regarding this use case.
orb is fast, but the strange gateway ip address may lead to issues. Also, it is closed source.
colima works well, is open source and has plenty of customization features. I can even enter its virtual machine and install custom stuff in there.
Therefore, I would tend to use colima.