Tool to ease playing with k3s inside docker, done by rancher.
It uses flannel to deal with networking. Flannel does not handle networkpolicy.
Flannel doesn’t control how containers are networked to the host, only how the traffic is transported between hosts and doesn’t implement network policy controller. For network policy, other projects such as Calico can be used.
It seems like there is a minimal implementation of networkpolicy inside k3s, but it does not appear to deal with even basic use cases.
running Cilium in K3D
- External reference: https://sandstorm.de/de/blog/post/running-cilium-in-k3s-and-k3d-lightweight-kubernetes-on-mac-os-for-development.html
important k3s arguments here are
cannot use a cilium init-container in the k3d/k3s combo, because it tries to run a script in the context of the above containers using the /bin/bash interpreter (but the k3s images are based on busybox, and only have a /bin/sh interpreter available).
docker exec -it k3d-foo-agent-0 mount bpffs /sys/fs/bpf -t bpf docker exec -it k3d-foo-agent-0 mount –make-shared /sys/fs/bpf
docker exec -it k3d-foo-server-0 mount bpffs /sys/fs/bpf -t bpf docker exec -it k3d-foo-server-0 mount –make-shared /sys/fs/bpf
helm repo add cilium https://helm.cilium.io/
helm install cilium cilium/cilium –version 1.9.1 \ –namespace kube-system \ –set kubeProxyReplacement=partial \ –set hostServices.enabled=false \ –set externalIPs.enabled=true \ –set nodePort.enabled=true \ –set hostPort.enabled=true \ –set bpf.masquerade=false \ –set image.pullPolicy=IfNotPresent \ –set ipam.mode=kubernetes
helm upgrade cilium cilium/cilium –version 1.9.1 \ –namespace kube-system \ –reuse-values \ –set hubble.listenAddress=":4244" \ –set hubble.relay.enabled=true \ –set hubble.ui.enabled=true
k3d with cgroup v2
- External reference: https://github.com/rancher/k3d/issues/493
I think I’m seeing the same or similar issue. When I rollback to rancher/k3s:v1.19.7-k3s1, the cluster starts fine.
k3d cluster create –verbose –trace –image rancher/k3s:v1.19.8-k3s1 wsop Doesn’t work on Fedora 33 as there is different error
> docker logs –follow k3d-wsop-server-0 2>&1 … time=“2021-03-03T11:52:51.432307543Z” level=fatal
Hi @fr33ky , thanks for your input. I guess when you run docker info, you see that Cgroup Version is 2? I’m on the same docker version on Ubuntu, but using cgroup v1… no problems here
Using Debian Sid, in the meantime, I personally switched back to cgroup v1. I added systemd.unified_cgroup_hierarchy=0 to my GRUB_CMDLINE_LINUX_DEFAULT (/etc/default/grub) and then ran update-grub
k3d: unable to start built container
- External reference: https://github.com/vmware-tanzu/buildkit-cli-for-kubectl/issues/46
Basically it seems like k3d is not a good environment for kubectl-buildkit, and the only option for it is if you are using a registry, as described by @pdevine, although that eliminates one of the use cases (not transferring bytes up and down from the registry, and needing a registry)
install a custom cni with k3d
- External reference: https://rancher.com/docs/k3s/latest/en/installation/network-options/
Run K3s with –flannel-backend=none and install your CNI of choice
install calico alongside flannel
- External reference: https://projectcalico.docs.tigera.io/getting-started/kubernetes/flannel/flannel
Installing Calico for policy and flannel (aka Canal) for networking
Notes linking here
- clk k8s
- clk parameters
- cognitive resistance to devops mindset
- Comment lister les images installés dans k3d
- docker looses the dns configuration
- how to share data using a local kubernetes cluster (blog)
- making k3d work again in debian testing (blog)
- persistent local volumes with k3d kubernetes
- use kind and k3d with tilt and clk k8s