This is a small follow up to my previous post that introduces another Kubernetes feature: Endpoints. The official documentations says that Endpoints are deprecated, but we can still use this to understand how Kubernetes implements some sort of load-balancing requests to services across multiple pods.

We begin with our manifest.yaml to apply a Kubernetes Deployment just like the last time, except the manifest is a lot simpler this time: It deploys 2 replicas, one on each worker ignore the node labels

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx-host
        image: nginx:latest
        ports:
        - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-svc
spec:
  selector:
    app: nginx
  type: NodePort
  ports:
  - port: 80
    targetPort: 80

You can see one pod running on each of the worker nodes:

> minikube kubectl -p mmd -- get pods -o wide
NAME                     READY   STATUS    RESTARTS   AGE   IP           NODE      NOMINATED NODE   READINESS GATES
nginx-7fb98f8f7c-sv57p   1/1     Running   0          37m   10.244.2.2   mmd-m03   <none>           <none>
nginx-7fb98f8f7c-w648g   1/1     Running   0          37m   10.244.1.2   mmd-m02   <none>           <none>
> minikube kubectl -p mmd -- get svc -o wide
NAME         TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)        AGE     SELECTOR
nginx-svc    NodePort    10.97.146.53   <none>        80:31831/TCP   3d12h   app=nginx

Requests to our service can now be served by any of the Pods.

In the last post we saw that packets of both the requests to the workerNodeIp:NodePort combo and clusterIp:ServicePort were matched by our NAT table and ultimately had their destination ip:addr combo rewritten to forward the request to the singular nginx pod with the IP at 10.244.1.2:

Chain KUBE-SEP-SADCJIHRQW7RJ62U (1 references)
 pkts bytes target     prot opt in     out     source               destination 
    .... 
    1    60 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/nginx-svc */ tcp to:10.244.1.2:80

This is an Endpoint. An Endpoint is a singular Pod that implements the actual service. Theorectially we could link a Service to a Pod directly because no one ever (atleast in my experience) thinks of creating Endpoints by hand, but from reading Kelsey Hightower’s Kubernetes book, it seems that the fundamental design principle of Kubernetes is to have things very loosely-coupled. When you create a Service, you don’t just directly specify a deployment whose pods will become the implementors of the Service. Rather you specify the matching labels that the Service looks for in Pods. One reason could be that there are 2 different deployments that launch 2 pods each with a common label and a Service matches this common label thus providing 4 pods to service traffic. I guess for these reasons of flexibility, Kubernetes introduced the Endpoint concept that can allow a Service to match pods from different deployments

Coming back to our current case, we launch 2 pods in our Deployment and the Service’s matchLabels matches the labes on both the pods, thus creating 2 endpoints. Traffic to the service can be directed to any one of the pods, but how is this redirection done ?

Well, turns out that you can extend IPtables with modules that can match packets in a custom way and one of them is the statistic module !

For example, you can use bpf bytecode in the rule to execute bpf code code on the incoming packet and use the result of the executation to ACCEPT or DROP packets as in the example in the man page

For example, to read only packets matching 'ip proto 6', insert
the following, without the comments or trailing whitespace:

       4               # number of instructions
       48 0 0 9        # load byte  ip->proto
       21 0 1 6        # jump equal IPPROTO_TCP
       6 0 0 1         # return     pass (non-zero)
       6 0 0 0         # return     fail (zero)

iptables -A OUTPUT -m bpf --bytecode '4,48 0 0 9,21 0 1 6,6
             0 0 1,6 0 0 0' -j ACCEPT

Kubernetes (more specifically the CNI plugin used by Kubernetes to implement networking) uses the statistic module that can match incoming packets with a probability:

Chain KUBE-SVC-HL5LMXD5JFHQZ6LN (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 KUBE-MARK-MASQ  tcp  --  *      *      !10.244.0.0/16        10.97.146.53         /* default/nginx-svc cluster IP */ tcp dpt:80
    0     0 KUBE-SEP-SADCJIHRQW7RJ62U  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/nginx-svc -> 10.244.1.2:80 */ statistic mode random probability 0.50000000000
    0     0 KUBE-SEP-QX4RFH7MAHSLU35M  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* default/nginx-svc -> 10.244.2.2:80 */

So whe a request comes matching our Kubernetes Service, the packet (actually only the first packet in a connection) gets a random coin toss to decide if it should have its destination re-written to be routed to 10.244.1.2 or 10.244.2.2. Pretty cool, eh ?

There was a bit of confusion for me here. I assumed that the NAT rules for rewriting packets runs for every packet, but that doesn’t seem to be the case. Some AI assisted searching and there is an old LKML email-thread about how NAT is run only for the FIRST packet in a new connection and all the subsequent packets for the same “connection” having the same destination.

This is good because otherwise literally every packet sent by our cURL would be distributed among the 2 pods, which would be disastrous for a connection oriented protocol like TCP, can you imagine sending your HTTP request as 5 packets and the prefix of your request URL ending up in Pod 1 and the path part of the URL ending up in Pod 2 ? LOL.

The word “connection” should tingle your senses (atleast it did mine). We are dealing with IP packets here, one layer below connection oriented protocols like TCP where the server knows the order of the packets, the seq number, the client etc. What does a “connection” mean here ? It turns out that there is (yet another) module called conntrack in the Linux networking system that uses the combo of (src_ip, srcp_port, destination_ip, destination_port and L4 protocol: tcp/udp/icmp etc) to map each packet to a connection. Since this combo will be unique (as in a client requests files from a service will have a unique ip address + source port), we can use these 5 parameters to literally track every packet back to a connection. Conntrack and connections are crucial across many of the iptables functionality and it looks like NAT (in both pre and post-routing hooks) depends on conntrack to rewrite the destination/source per connections.

Conntrack deserves a series of posts in itself and thankfully someone more knowledgeable than me has already written about it. The posts go deep into the details of conntrack usage and implementation so if you are interesting, definitely check it out.

To sum it up, its kinda simpler if we work backwards from a Pod -> client to see how the chain of rules and chains in our IPTables are setup to ensure that requests from clients reach our service’s pods (I am going to try to do this as ASCII art as a tribute to the Netfilter HOWTO)

CLIENT (via ClusterIP)
   │
   ▼
PREROUTING
   │
   ▼
KUBE-SERVICES  (all k8s services)
   │
   ▼
KUBE-SVC-HL5LMXD5JFHQZ6LN   (nginx-svc)
   │
   ▼
KUBE-SEP-QX4RFH7MAHSLU35M   (endpoint / DNAT)
   │
   ▼
Pod  10.244.1.2:80


CLIENT (via NodePort)
   │
   ▼
PREROUTING
   │
   ▼
KUBE-SERVICES
   │
   ▼
KUBE-NODEPORTS
   │
   ▼
KUBE-SVC-HL5LMXD5JFHQZ6LN
   │
   ▼
KUBE-SEP-QX4RFH7MAHSLU35M
   │
   ▼
Pod  10.244.1.2:80