Route missing in kubernetes node with kuberouter as the CNI

Anyone who is evaluating into having a networking solution for their kubernetes cluster without having a lot of moving parts in the cluster, kuberouter provides pod networking, ability to enforce network policies, IPVS/LVS service proxy among other things.

The problem which we faced specifically while running this in our clusters was missing routes upon restart of the node, or sometimes in the case when the node was joining the cluster as part of the worker node.

For us, the issue would come around as a the kiam (which we were using for identity management for pods inside the k8s clusters) pod would go into CrashLoopBackOff as described by me in the github issue https://github.com/uswitch/kiam/issues/49, as the dns resolution would fail (more on that later)

We were using the latest version of Coreos, but we found out that the version 1576.5.0 of Coreos was not plagued by this problem.

This has been defined in detail in the github issue

The problem was that there was race condition caused by systemd-networkd.service trying to manager the tunnels and modigying the routes causing the missing routes. Whenever networkd was restarting, all the tunnels would go away with it.

It is best described by Niel here.

To fix this, a file in the networkd dir /etc/systemd/network/ so it starts ignoring those interfaces and doesn’t manage them as described by Lomkju

[Match]
Name=tun* kube-bridge kube-dummy-if

[Link]
Unmanaged=yes

It was tested to be working for the following coreos version as mentioned by Lomkju

Container Linux by CoreOS 1967.6.0 (Rhyolite)
Kernel: 4.14.96-coreos-r1