Konubinix' opinionated web of thoughts

K3d

Fleeting

Tool to ease playing with k3s inside docker, done by rancher.

It uses flannel to deal with networking. Flannel does not handle networkpolicy.

Flannel doesn’t control how containers are networked to the host, only how the traffic is transported between hosts and doesn’t implement network policy controller. For network policy, other projects such as Calico can be used.

— https://banzaicloud.com/docs/pipeline/security/network-policy/network-plugins/

It seems like there is a minimal implementation of networkpolicy inside k3s, but it does not appear to deal with even basic use cases.

running Cilium in K3D

important k3s arguments here are --disable-network-policy and --flannel-backend=none.

— https://sandstorm.de/de/blog/post/running-cilium-in-k3s-and-k3d-lightweight-kubernetes-on-mac-os-for-development.html

cannot use a cilium init-container in the k3d/k3s combo, because it tries to run a script in the context of the above containers using the /bin/bash interpreter (but the k3s images are based on busybox, and only have a /bin/sh interpreter available).

— https://sandstorm.de/de/blog/post/running-cilium-in-k3s-and-k3d-lightweight-kubernetes-on-mac-os-for-development.html

docker exec -it k3d-foo-agent-0 mount bpffs /sys/fs/bpf -t bpf docker exec -it k3d-foo-agent-0 mount –make-shared /sys/fs/bpf

docker exec -it k3d-foo-server-0 mount bpffs /sys/fs/bpf -t bpf docker exec -it k3d-foo-server-0 mount –make-shared /sys/fs/bpf

— https://sandstorm.de/de/blog/post/running-cilium-in-k3s-and-k3d-lightweight-kubernetes-on-mac-os-for-development.html

helm repo add cilium https://helm.cilium.io/

helm install cilium cilium/cilium –version 1.9.1 \ –namespace kube-system \ –set kubeProxyReplacement=partial \ –set hostServices.enabled=false \ –set externalIPs.enabled=true \ –set nodePort.enabled=true \ –set hostPort.enabled=true \ –set bpf.masquerade=false \ –set image.pullPolicy=IfNotPresent \ –set ipam.mode=kubernetes

helm upgrade cilium cilium/cilium –version 1.9.1 \ –namespace kube-system \ –reuse-values \ –set hubble.listenAddress=":4244" \ –set hubble.relay.enabled=true \ –set hubble.ui.enabled=true

— https://sandstorm.de/de/blog/post/running-cilium-in-k3s-and-k3d-lightweight-kubernetes-on-mac-os-for-development.html

k3d with cgroup v2

I think I’m seeing the same or similar issue. When I rollback to rancher/k3s:v1.19.7-k3s1, the cluster starts fine.

— https://github.com/rancher/k3d/issues/493

k3d cluster create –verbose –trace –image rancher/k3s:v1.19.8-k3s1 wsop Doesn’t work on Fedora 33 as there is different error

> docker logs –follow k3d-wsop-server-0 2>&1 … time=“2021-03-03T11:52:51.432307543Z” level=fatal

— https://github.com/rancher/k3d/issues/493

Hi @fr33ky , thanks for your input. I guess when you run docker info, you see that Cgroup Version is 2? I’m on the same docker version on Ubuntu, but using cgroup v1… no problems here

— https://github.com/rancher/k3d/issues/493

Using Debian Sid, in the meantime, I personally switched back to cgroup v1. I added systemd.unified_cgroup_hierarchy=0 to my GRUB_CMDLINE_LINUX_DEFAULT (/etc/default/grub) and then ran update-grub

— https://github.com/rancher/k3d/issues/493

k3d: unable to start built container

Basically it seems like k3d is not a good environment for kubectl-buildkit, and the only option for it is if you are using a registry, as described by @pdevine, although that eliminates one of the use cases (not transferring bytes up and down from the registry, and needing a registry)

— https://github.com/vmware-tanzu/buildkit-cli-for-kubectl/issues/46#:~:text=Basically%20it%20seems,clarified%20the%20environment

install a custom cni with k3d

Custom CNI

Run K3s with –flannel-backend=none and install your CNI of choice

— https://rancher.com/docs/k3s/latest/en/installation/network-options/

install calico alongside flannel

Installing Calico for policy and flannel (aka Canal) for networking

— https://projectcalico.docs.tigera.io/getting-started/kubernetes/flannel/flannel

Notes linking here