mmacleod.ca

Zero Trust K3s Network With Cilium

I wanted to implement full zero-trust networking within my k3s cluster which uses the Cilium CNI, which has custom CiliumClusterwideNetworkPolicy and CiliumNetworkPolicyresources, which extend what is possible with standard Kubernetes NetworkPolicy resources.

Cilium defaults to allowing traffic, but if a policy is applied to an endpoint, it switches and will deny any connect not explicitely allowed. Note that this is direction dependent, so ingress and egress are treated separately.

Zero trust policies require you to control traffic in both directions. Not only does your database need to accept traffic from your app, but your app has to allow the connection to the database.

This is tedious, and if you don’t get it right it will break your cluster and your ability to tell what you’re missing. So I figured I’d document the policies required to keep your cluster functional.

Note that my k3s cluster has been deployed with --disable-network-policy, --disable-kube-proxy, --disable-servicelb, and --disable-traefik, because these services are provided by Cilium (or ingress-nginx, in the case of traefik).

Lastly, while the policies below apply to k3s, they’re probably a good starting point for other clusters – the specifics will be different, but you’re always going to want to allow traffic to your DNS service, etc.

Hubble UI

Before attempting any network policies, ensure you’ve got hubble ui and hubble observe working. You should verify that the endpoints and ports used in the policies below match your cluster.

Cluster Wide Policies

These policies are applied cluster wide, without regard for namespace boundaries.

Default Deny

Does what it says on the tin.

apiVersion: "cilium.io/v2"
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: "default-deny"
spec:
  description: "Empty ingress and egress policy to enforce default-deny on all endpoints"
  endpointSelector:
    {}
  ingress:
  - {}
  egress:
  - {}

Allow Health Checks

Required to allow cluster health checks to pass.

apiVersion: "cilium.io/v2"
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: "health-checks"
spec:
  endpointSelector:
    matchLabels:
      'reserved:health': ''
  ingress:
    - fromEntities:
      - remote-node
  egress:
    - toEntities:
      - remote-node

Allow ICMP

ICMP is useful with IPv4, and absolutely necessary for IPv6. This policy allows select ICMP and ICMPv6 request types globally, both within and outside the cluster.

apiVersion: "cilium.io/v2"
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: "allow-icmp"
spec:
  description: "Policy to allow select ICMP traffic globally"
  endpointSelector:
    {}
  ingress:
  - fromEntities:
    - all
  - icmps:
    - fields:
      - type: EchoRequest
        family: IPv4
      - type: EchoReply
        family: IPv4
      - type: DestinationUnreachable
        family: IPv4
      - type: TimeExceeded
        family: IPv4
      - type: ParameterProblem
        family: IPv4
      - type: Redirect 
        family: IPv4
      - type: EchoRequest
        family: IPv6
      - type: DestinationUnreachable
        family: IPv6
      - type: TimeExceeded
        family: IPv6
      - type: ParameterProblem
        family: IPv6
      - type: RedirectMessage
        family: IPv6
      - type: PacketTooBig
        family: IPv6
      - type: MulticastListenerQuery
        family: IPv6
      - type: MulticastListenerReport
        family: IPv6
  egress:
  - toEntities:
    - all
  - icmps:
    - fields:
      - type: EchoRequest
        family: IPv4
      - type: EchoReply
        family: IPv4
      - type: DestinationUnreachable
        family: IPv4
      - type: TimeExceeded
        family: IPv4
      - type: ParameterProblem
        family: IPv4
      - type: Redirect 
        family: IPv4
      - type: EchoRequest
        family: IPv6
      - type: EchoReply
        family: IPv6
      - type: DestinationUnreachable
        family: IPv6
      - type: TimeExceeded
        family: IPv6
      - type: ParameterProblem
        family: IPv6
      - type: RedirectMessage
        family: IPv6
      - type: PacketTooBig
        family: IPv6
      - type: MulticastListenerQuery
        family: IPv6
      - type: MulticastListenerReport
        family: IPv6

Allow Kube DNS

This pair of policies allows the cluster to query DNS.

apiVersion: "cilium.io/v2"
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: "allow-to-kubedns-ingress"
spec:
  description: "Policy for ingress allow to kube-dns from all Cilium managed endpoints in the cluster"
  endpointSelector:
    matchLabels:
      k8s:io.kubernetes.pod.namespace: kube-system
      k8s-app: kube-dns
  ingress:
  - fromEndpoints:
    - {}
    toPorts:
    - ports:
      - port: "53"
        protocol: UDP
---
apiVersion: "cilium.io/v2"
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: "allow-to-kubedns-egress"
spec:
  description: "Policy for egress allow to kube-dns from all Cilium managed endpoints in the cluster"
  endpointSelector:
    {}
  egress:
  - toEndpoints:
    - matchLabels:
        k8s:io.kubernetes.pod.namespace: kube-system
        k8s-app: kube-dns
    toPorts:
    - ports:
      - port: "53"
        protocol: UDP

Kubernetes Services

These policies are applied to the standard kubernetes services running in the kube-system namespace.

Kube DNS

Kube DNS (or Core DNS in some k8s distros) needs to talk to the k8s API server and also to DNS resolvers outside the cluster.

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: kube-dns
  namespace: kube-system
spec:
  endpointSelector:
    matchLabels:
      k8s:io.kubernetes.pod.namespace: kube-system
      k8s-app: kube-dns
  ingress:
  - fromEntities:
    - host
    toPorts:
    - ports:
      - port: "8080"
        protocol: TCP
      - port: "8181"
        protocol: TCP
  egress:
  - toEntities:
    - world
    toPorts:
    - ports:
      - port: "53"
        protocol: UDP
  - toEntities:
    - host
    toPorts:
    - ports:
      - port: "6443"
        protocol: TCP

Metrics Server

The metrics service needs to talk to most of the k8s services.

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: metrics-server
  namespace: kube-system
spec:
  endpointSelector:
    matchLabels:
      k8s:io.kubernetes.pod.namespace: kube-system
      k8s-app: metrics-server
  ingress:
  - fromEntities:
    - host
    - remote-node
    - kube-apiserver
    toPorts:
    - ports:
      - port: "10250"
        protocol: TCP
  egress:
  - toEntities:
    - host
    - kube-apiserver
    - remote-node
    toPorts:
    - ports:
      - port: "10250"
        protocol: TCP
  - toEntities:
    - kube-apiserver
    toPorts:
    - ports:
      - port: "6443"
        protocol: TCP

Local Path Provisioner

The local path provisioner only seems to talk to the k8s API server.

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: local-path-provisioner
  namespace: kube-system
spec:
  endpointSelector:
    matchLabels:
      k8s:io.kubernetes.pod.namespace: kube-system
      app: local-path-provisioner
  egress:
  - toEntities:
    - host
    - kube-apiserver
    toPorts:
    - ports:
      - port: "6443"
        protocol: TCP

Cilium Services

These policies apply to the Cilium services themselves. I deployed mine to the cilium namespace, so adjust as necessary if you deployed Cilium to the kube-system namespace.

Hubble Relay

The hubble-relay service needs to talk to all cilium and hubble components in order to consolidate a cluster-wide view.

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  namespace: cilium
  name: hubble-relay
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: hubble-relay
  ingress:
  - fromEntities:
      - host
    toPorts:
    - ports:
      - port: "4222"
        protocol: TCP
      - port: "4245"
        protocol: TCP
  - fromEndpoints:
    - matchLabels:
        app.kubernetes.io/name: hubble-ui
    toPorts:
    - ports:
      - port: "4245"
        protocol: TCP
  egress:
  - toEntities:
    - host
    - remote-node
    - kube-apiserver
    toPorts:
      - ports:
        - port: "4244"
          protocol: TCP

Hubble UI

The hubble-ui provides the tools necessary to actually observe traffic in the cluster.

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  namespace: cilium
  name: hubble-ui
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: hubble-ui
  ingress:
  - fromEntities:
      - host
    toPorts:
    - ports:
      - port: "8081"
        protocol: TCP
  egress:
  - toEndpoints:
    - matchLabels:
        app.kubernetes.io/name: hubble-relay
    toPorts:
      - ports:
        - port: "4245"
          protocol: TCP
  - toEntities:
    - kube-apiserver
    toPorts:
      - ports:
        - port: "6443"
          protocol: TCP

Cert Manager

These policies will help if you’re using cert-manager.

Cert Manager

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  namespace: cert-manager
  name: cert-manager
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: cert-manager
  ingress:
  - fromEntities:
      - host
    toPorts:
    - ports:
      - port: "9403"
        protocol: TCP
  egress:
  - toEntities:
    - kube-apiserver
    toPorts:
      - ports:
        - port: "6443"
          protocol: TCP
  - toEntities:
    - world
    toPorts:
      - ports:
        - port: "443"
          protocol: TCP
        - port: "53"
          protocol: UDP

Webhook

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  namespace: cert-manager
  name: webhook
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: webhook
  ingress:
  - fromEntities:
      - host
    toPorts:
    - ports:
      - port: "6080"
        protocol: TCP
  - fromEntities:
      - kube-apiserver
    toPorts:
    - ports:
      - port: "10250"
        protocol: TCP
  egress:
  - toEntities:
    - kube-apiserver
    toPorts:
      - ports:
        - port: "6443"
          protocol: TCP

CA Injector

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  namespace: cert-manager
  name: cainjector
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: cainjector
  egress:
  - toEntities:
    - kube-apiserver
    toPorts:
      - ports:
        - port: "6443"
          protocol: TCP

External DNS

This policy will allow external-dns to communicate with API driven DNS services. To update local DNS services via RFC2136 updates, change the world egress port from 443 TCP to 54 UDP.

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  namespace: external-dns
  name: external-dns
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: external-dns
  ingress:
  - fromEntities:
      - host
    toPorts:
    - ports:
      - port: "7979"
        protocol: TCP
  egress:
  - toEntities:
    - kube-apiserver
    toPorts:
      - ports:
        - port: "6443"
          protocol: TCP
  - toEntities:
    - world
    toPorts:
      - ports:
        - port: "443"
          protocol: TCP

Ingress-Nginx & OAuth2 Proxy

These policies will be helpful if you use ingress-nginx and oauth2-proxy. Note that I deployed them to their own namespaces, so you may need to adjust if you deployed them to the same namespace.

Ingress-Nginx

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  namespace: ingress-nginx
  name: ingress-nginx
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
  ingress:
  - fromEntities:
      - kube-apiserver
    toPorts:
    - ports:
      - port: "8443"
        protocol: TCP
  - fromEntities:
      - host
    toPorts:
    - ports:
      - port: "10254"
        protocol: TCP
  - fromEntities:
      - world
    toPorts:
    - ports:
      - port: "80"
        protocol: TCP
      - port: "443"
        protocol: TCP
  egress:
  - toEntities:
    - kube-apiserver
    toPorts:
      - ports:
        - port: "6443"
          protocol: TCP
  - toEndpoints:
    - matchLabels:
        k8s:io.kubernetes.pod.namespace: oauth2-proxy
        app.kubernetes.io/name: oauth2-proxy
    toPorts:
    - ports:
      - port: "4180"
        protocol: TCP

OAuth2 Proxy

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  namespace: oauth2-proxy
  name: oauth2-proxy
spec:
  endpointSelector:
    matchLabels:
      app.kubernetes.io/name: oauth2-proxy
  ingress:
  - fromEndpoints:
    - matchLabels:
        k8s:io.kubernetes.pod.namespace: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
    toPorts:
    - ports:
      - port: "4180"
        protocol: TCP
  - fromEntities:
    - host
    toPorts:
    - ports:
      - port: "4180"
        protocol: TCP
  egress:
  - toEntities:
    - world
    toPorts:
      - ports:
        - port: "443"
          protocol: TCP

Conclusion

These policies should get your cluster off the ground (or close to it). You’ll still need to add additional policies for your actual workloads (and probably extend the ingress-nginx one).

More posts