Netfilter,IPtables and Kubernetes: how do they work together?
The simplest explanation I could find was this Linux Howto document from 2002 by Rusty Russell, the author of ipchains, netfilters and iptables , where the stages of network packet-processing is described. There is an ASCII art image of the netfilters system in Linux in there that is still surprisingly relavant in 2026.
--->PRE------>[ROUTE]--->FWD---------->POST------>
Conntrack | Mangle ^ Mangle
Mangle | Filter | NAT (Src)
NAT (Dst) | | Conntrack
(QDisc) | [ROUTE]
v |
IN Filter OUT Conntrack
| Conntrack ^ Mangle
| Mangle | NAT (Dst)
v | FilterWhat is Netfilter
The lifecyle of an incoming (and outgoing) network packet goes through certain stages:
- Packet comes in
- Packet is tracked wrt to an existing or a new connection
- The networking stack in the kernel decides if the packet must be delivered to a local process or if it has to be sent outwards (via another network interface) to a different machine
- Same for packets generated by our local processes (
curl, ftp clients etc). A decision must be made if the packet must be sent out to the internet or delivered to another local processes - Before the packet is sent out, processs it (swap source IP address etc)
- Before packets are delivered to processes, sent from processess to the outside world or forwarded from one interface to another, accept or drop them.
Admins and programmers would want to control at each stage to enable or disable certain kind of behaviours, for example:
- Do not allow connections initiated by external servers to our machines, ingress is only allowed for packets related to a connection already created by a local process
- Allow this VM running in our local machine to be able to talk to the internet
- If 2 VMs are running , allow them to be networked to each other and let them also be able to make calls to the internet
Netfilter is the framework that allows this extensibility of the Linux networking stack without users having to write drivers or extensions to the kernel and recompiling and running custom kernels.
The mechanism is provided through NETFILTER_HOOKS, a set of pre-defined stages in the flow of a packet in/out of the networking stack where hook can be injected by a user to control the packet processing behaviour.
IPtables is a system built on top of Netfilter. Kernel modules can register tables and hook them up into the various hook points offered by Netfilter for packet processing.
There seem to be a few different mechanisms to inject these hooks, but the most common ones seem to be iptables (the older) and nftables (the newer) mechanism, where user writes these hooks as rules
What is not covered
I don’t want to go too deep into how these are setup as Kernel modules or how how the table structures exist (also honestly, I haven’t done any experiments on that front either). I set out to write this as a guide for some practical understanding of iptables and usecases and how they are used by tools we use day to day, such as containers and Kubernetes Pods/services etc. A lot of the work previously done by kernel modules seems to be done using eBPF these days; for example, there is Cilium a Kubernetes CNI plugin used widely to setup HTTP aware load-balances,proxies for their workloads on Kubernetes.
Likewise, NFtables are also not covered because we have no need for any custom tables beyond the existing tables from iptables. The scenarios described in this post add chains and rules to existing tables to setup networking for containers and VMS.
Back to Netfilters.
This is the image representation of the ASCII diagram from before
There are only 5 tables as far as I know: nat, filter, raw, mangle, security and each table provides chains that hooks into one or more of the hook points. The hook pointers are fixed per table: that is chains in the filter table are applied only in the INPUT, FORWARD and OUTPUT hooks, whereas the nat table has chains running in the PREROUTING, OUTPUT and POSTROUTING hooks; Chains in the raw table can apply at PREROUTING and OUTPUT hooks whereas chains in the mangle table apply at all hooks.
If there are multiple chains from different tables for the same hook, then the chains are run as per a fixed order (in NFtables, the order can be arbitrarily changed). For example, the order of priority of chains at some of the hooks are
PREROUTING
raw
mangle
nat (DNAT / destination translation)
INPUT
mangle
filter
security
FORWARD
mangle
filter
securityMost of the commands you will see here uses the iptables userspace CLI. This cli tool provides a user-friendly way to add or remove rules to chains and chains to tables. Technically, users can add or remove new tables via iptables too, but to do so you need to write and load kernel modules or use nftables.
Kubernetes seems to support both iptables and nftables, but most installations seem to use iptables by default. For our deep-dive here we will mostly deal with the standard filter and nat tables exploring how kubernetes installed chains rewrite packets to support kubernetes services.
Filtering
The filter table, as the name suggests, contains chains and rules to filter packets. These chains are attached to the hook points: NF_IP_FORWARD (when it is decided that a packet must be routed from one interface to another), NF_IP_LOCAL_IN (before a packet is delivered to a process) and NF_IP_LOCAL_OUT (when a packet leaves a process)
Scenario 1
Container runtimes on your local machine can setup a software bridge devices to allow 2 containers to talk to one another. The containers will have a default route to forward packets to this bridge if it can’t find a route to the resolved IP address. Take the usecase of a curl request to npmjs.com within the container. The IP address of npmjs.com resolves to an address that lies outside of the current local network, so the default route is to forward those packets to the bridge which then forwards to the right physical host or knows where to further forward the packets.
In my case, the bridge simply forwards the the packets to my physical network card and the packets are sent to the actual host of npmjs.com through the public internet.The packets sent from my bridge to my physical ethernet/wireless interface were by default dropped, so I added a rule to the FORWARD chain in the filter table to accept all packets coming from br0 (the software bridge) to wlp0s0 (my physical network device)
iptables -t filter -A FORWARD -i br0 -o wlp0s0 -j ACCEPTThe filter table is the default table if -t is unspecified, so you will often see commands modify the filter table to just skip the -t option
iptables -A FORWARD -i br0 -o wlp0s0 -j ACCEPTScenario 2
Another rule would be to drop all packets unless they are for an SSH connection to port 22:
Here we set the rule to allow incoming packets to a local process if the destination port is 22 and the protocol is TCP. The default policy of the input chain is to drop the packets. The rules are evaluated in order, kind of like a case/match expression in programming languages:
iptables -A INPUT -p tcp --dport 22 -j ACCEPT
iptables -P INPUT DROPNAT
NAT stands for Network Address Translation. When a computer is connect simultaneously to a local networks and an external one, there will be a need for applications to deal with different IP addresses, one for the local networks and the other for the public internet.
Scenario 1
Take the case of 2 virtual machines on a localhost. I setup a bridge between the VMs and gave it each addresses in the 172.16.0.1/28 address range. The default route is set to that of the bridge.
When these VMs need to talk to the rest of the internet, the packets sent from these VMs will contain 172.16.0.x as the source IP. This 172.16.0.x IP range is classified as a private IP range for use within private networks and thus when a website wants to respond to this address, it quite simply doesn’t know how to to. This is also the same case with internet routes in our homes, where all of our devices have a private IP within the Wifi/LAN network and the router has an external public IP Address.
To solve this issue, we setup a post-routing rule that swaps the source IP of the VM with that of the public internet interface so that external servers can send their responses to us. This is set by adding a MASQUERADE rule to the POSTROUTING chain (masquerade means to disguise or hide)
iptables -t nat -A POSTROUTING -o wlp0s0 -j MASQUERADEWhat this rule basically says is that when a packet leaves (-o output) through the wlp0s0 interface, swap the source IP in the packet with the IP address of the wlp0s0 interface so that our destination knows how to send the reply packets for our request. The reason we have to use masquerade is because our interface wlp0s0 can get a different public IP address over time (DHCP) because the number of IPV4 addresses are limited and broadband/home interner providers have a fixed pool of IP addresses that get assigned and reassigned to different devices as demand changes and devices get restarted
Do we need a rule to explicitly enable the reverse: That is to allow response packets coming in from external systems and swap the destination IP to that of the VM when the remote server responds to our packets ? There is another Linux networking feature called connntrack that keeps track of (by default) of all connections in and out of our interfaces. When we allow connection initiated from our VMs to forward packets, conntrack remembers this connection and on incoming reply packets allows them to continue traveling towards the VM. If that is not the case by default, we can enable it on the FORWARD chain to allow incoming packets for established connections by updating the forward chain
iptables -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPTSince -t is omitted, it updates the filter table.
IPtables: Targets and Policies
All these chains contain rules that match certain properties of packets (source IP, destination IP, etc.) and each rule has a target: the action to take if the packet matches (e.g. ACCEPT, DROP, REJECT, or jump to another chain). When no rule matches a packet, the chain falls back to its policy: a default action of either ACCEPT or DROP, applicable only to built-in chains (INPUT, OUTPUT, FORWARD). You can look up the current policies with:
iptables -L -v -n
# -L list rules
# -v verbose
# -n numeric, just display the IPs in the rules' conditions as isNow that we know about netfilters (hooks in the networking stack where we can attach code or rules), iptables: an organization of chains that contain rules which attach at one of the hook points that are table specific and NAT, lets see how these tools are employed to build something most of us would use everyday (or atleast frequently): Kubernetes Services
Software Bridge
The software bridge in Linux is an extremely flexible virtual device for networking. It can operate at 2 levels:
- At Layer 2 (like a physical Ethernet switch), forwarding ethernet frames between the interfaces plugged into it based on MAC addresses. You can spin up a network namespace (eg for a container), setup a virtual ethernet pair with one end in the namspace and another on the host connected to the bridge, routing packets between the host and the namespace as if 2 physical machines connected with a ethernet cable
or
- Assign an IP address to the bridge and to the virtual ethernet devices in the network namespace thus making the bridge act like a packet switch
In our Kubernetes scenario, our VMs and Bridge have both a MAC address and an IP address thus allow us to route IP packets from the host -> VMs and vice-versa. We will not go too deep into bridges here, but for a more in-depth reading on bridges and virtual devices in Linux I would highly encourage you to read Iximiuz’s course and container networking tutorial
Kubernetes Networking
Have you used a Service in Kubernetes? Chances are you have. Have you ever wondered how it works, like how can pods run on different nodes and pods on the same node can communicate with one another or send traffic to a pod on the other node?
If you don’t know the answer to this question, fear not, for this post will guide you through my setup and how I traced the flow of packets through the system until it landed on the right port within a container’s network namespace where our nginx process is listening, buried deep within the system.
Let’s start with the setup first. I will deploy an nginx server as a Kubernetes Pod via a Deployment. Then we create a Service for the nginx server exposing a NodePort, that is each worker node in the cluster exposing a port to external traffic. If you know the ip address of the node, you can reach the nginx server via:
http://ip-of-node:targetPort
There might be a confusing with a clusterIP which is an IP address within the cluster that other pods can use to reach our nginx service. This IP cannot be reached outside of the cluster. We will also take a look at the clusterIP later to see how it is fake and isn’t really an IP but is used by Kubernetes to route traffic to the right pod/container within the cluster.
The setup
VM setup
It is difficult to setup a physical multi-node cluster (I am running a few experiments for setting up a cluster with an Rpi, X86_64 and a Macbook, but that’s probably for another post). So instead I used minikube on Linux to setup a cluster with multiple virtual machines as worker nodes on my Desktop. I used the kvm driver to setup the VMs.
For our intents and purposes, Virtual Machines vs Physical worker nodes should not make a difference. I will explain how the networking is setup for virtual machines and the only difference between physical worker nodes and virtual nodes would be the routes in our host’s routing tables.
Once you follow the instructions on minikube’s page to install the kvm driver (libvirt and qemu), you can start a cluster with 1 control-plane node and 2 workers nodes as simply as
minikube start --driver=kvm2 -p mmd --nodes 3This starts a cluster named mmd with 2 worker nodes and 1 control plane node. I encountered errors initially due to groups permission issue with the libvirt group. To use libvirt daemon for managing VMs, the user has to be made part of the libvirt user group. Unfortunately my shell didn’t pickup the updated group membership until I rebooted my system, so if you see Permission denied or minikube failure issue, try to see if your user is part of the libvirt group in the shell first.
Once the cluster has been started, you can ssh into the worker VMs using
minikube ssh -p mmd --node mmd-node02/03or run kubectl against the cluster using
minikube kubectl -p mmd -- get pods -A The VMs are connected to one another (and the host) through the bridge
> virsh net-list --all
Name State Autostart Persistent
--------------------------------------------
default active yes yes
mk-mmd active yes yes
> virsh net-info mk-mmd
Name: mk-mmd
UUID: 7119a156-5b6a-477f-8d0a-f3747e099151
Active: yes
Persistent: yes
Autostart: yes
Bridge: virbr1
> virsh net-dumpxml mk-mmd
<network connections='3'>
<name>mk-mmd</name>
<uuid>7119a156-5b6a-477f-8d0a-f3747e099151</uuid>
<bridge name='virbr1' stp='on' delay='0'/>
<mac address='52:54:00:42:7c:5d'/>
<dns enable='no'/>
<ip address='192.168.39.1' netmask='255.255.255.0'>
<dhcp>
<range start='192.168.39.2' end='192.168.39.253'/>
<host mac='fe:e9:8d:3c:5d:8e' name='mmd' ip='192.168.39.217'/>
<host mac='7a:e5:d3:9f:09:61' name='mmd-m02' ip='192.168.39.133'/>
<host mac='ce:ba:58:f7:6b:cd' name='mmd-m03' ip='192.168.39.167'/>
</dhcp>
</ip>
</network>Kubernetes Service Setup
- We create a deployment for
nginxthat runs exactly 1 pod in a specific worker node - We create a
NodePortservice for nginx. The service provisions a port on each worker node such that the service can be reached through the node’s IP address:curl http://ip-addr-of-node:targetPort - Scenario 1: We investigate the how the networking works when making requests directly to the IP of the node that’s running the pod and through the other node’s IP
- Scenario 2: We touch upon how pods can reach the service internally by resolving the
nginx.default.svc.cluster.localDNS record
This is the manifest to create the Deployment and the NodePort service. I set a nodeSelector label in the deployment so that the replica lands on the m02 node. Before that add a label to our m02 node
minikube kubectl -p mmd -- label node mmd-m02 role=nginx-hostspecify the role label in the Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
nodeSelector:
role: nginx-host # Pod gets deployed on the node with this label
containers:
- name: nginx-host
image: nginx:latest
ports:
- containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
name: nginx-svc
spec:
selector:
app: nginx
type: NodePort
ports:
- port: 80
targetPort: 80Apply the manifest and once the service is created and the pod is up and running we can make requests to the IP address of the worker (Note the fact it is the IP addr of the worker)
> minikube kubectl -p mmd -- apply -f manifest.yaml
> minikube kubectl -p mmd -- get pods
NAME READY STATUS RESTARTS AGE
nginx-6f764748fc-4wk85 1/1 Running 0 9h
### Scenario 1: Making requests to the nginx pod via Node 03's NodePort
```sh
> minikube ip -p mmd -n mmd-m03
192.168.39.167
> minikube ip -p mmd -n mmd-m02
192.168.39.133
>curl http://$(minikube ip -p mmd --node mmd-m03):31831/
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
<style>
...
<p><em>Thank you for using nginx.</em></p>
</body>
</html>How does our host know to route the packets for our curl’s request to the Worker node ?
If I do ip route show this is the routing table on my host
...
192.168.39.0/24 dev virbr1 proto kernel scope link src 192.168.39.1
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 The first rule says that for all addresses in the range 192.168.39.0/24 route it to the device with the ip 192.168.39.1 which happens to be the bridge device created for our VM <-> VM network.
So we have a route from host -> node m03. How does m03 route our incoming requests to the Pod on m02.
This is where the magic of iptables comes into the picture.
When the packet reaches our worker node, the packet must either be routed to another interface or sent to a local process on the worker node itself.
There is a PREROUTING hook that runs before the routing decision is made and our iptables for m03 does some tricks here. We can look at what happens by investigating the
chains and the rules in the PREROUTING hook of the NAT table.
> minikube ssh -p mmd -n mmd-m03
# Inside the worker shell
> sudo iptables -t nat -L -v -n
Chain PREROUTING (policy ACCEPT 37 packets, 10218 bytes)
pkts bytes target prot opt in out source destination
71 12592 KUBE-SERVICES all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */
....
....
Chain KUBE-SERVICES (2 references)
pkts bytes target prot opt in out source destination
.....
0 0 KUBE-SVC-HL5LMXD5JFHQZ6LN tcp -- * * 0.0.0.0/0 10.97.146.53 /* default/nginx-svc cluster IP */ tcp dpt:80
2 120 KUBE-NODEPORTS all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL
Chain KUBE-NODEPORTS (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-EXT-HL5LMXD5JFHQZ6LN tcp -- * * 0.0.0.0/0 127.0.0.0/8 /* default/nginx-svc */ tcp dpt:31831 nfacct-name localhost_nps_accepted_pkts
1 60 KUBE-EXT-HL5LMXD5JFHQZ6LN tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/nginx-svc */ tcp dpt:31831
Chain KUBE-EXT-HL5LMXD5JFHQZ6LN (2 references)
pkts bytes target prot opt in out source destination
1 60 KUBE-SVC-HL5LMXD5JFHQZ6LN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain KUBE-SVC-HL5LMXD5JFHQZ6LN (2 references)
pkts bytes target prot opt in out source destination
1 60 KUBE-SEP-SADCJIHRQW7RJ62U all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/nginx-svc -> 10.244.1.2:80 */
Chain KUBE-SEP-SADCJIHRQW7RJ62U (1 references)
pkts bytes target prot opt in out source destination
....
1 60 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/nginx-svc */ tcp to:10.244.1.2:80Our PREROUTING chain sends our packet through the following chains:
PREROUTING -> KUBE-SERVICES -> KUBE-NODEPORTS -> KUBE-EXT-HL5LMXD5JFHQZ6LN -> KUBE-SVC-HL5LMXD5JFHQZ6LN -> KUBE-SEP-SADCJIHRQW7RJ62U
(Ignore the comments within the /* */ section) Following the chain we endup with a DNAT target that routes the tcp connection to the ip,port combo 10.244.1.2
DNAT or destination NAT is a target in a rule that rewrites the destination ip,port in the packet to the values mentioned in the rule.
So when our worker node gets a TCP packet with the destination port 31831 (the second rule in our KUBE-NODEPORTS chain) meant for the nginx service, the packet processing rule follows the chains until it lands on our DNAT target rule which then rewrites the destination ip and port in our packet.
The packet is now ready to be routed. The new ip of the packet is now not 127.0.0.1 or 0.0.0.0 which means that this packet is not meant for any process in the localhost. Where should the packet be sent to then ? Lets consult the routing table in our worker
# on m03
> ip route show
ip route show
default via 192.168.122.1 dev eth1 proto dhcp src 192.168.122.74 metric 1024
10.244.0.0/24 via 192.168.39.217 dev eth0
10.244.1.0/24 via 192.168.39.133 dev eth0
192.168.39.0/24 dev eth0 proto kernel scope link src 192.168.39.167 metric 1024
192.168.122.0/24 dev eth1 proto kernel scope link src 192.168.122.74 metric 1024
192.168.122.1 dev eth1 proto dhcp scope link src 192.168.122.74 metric 1024 and voila ! The route for 10.244.1.2 must be routed via 192.168.39.133 which maps to the ip address of our worker m02 (where the nginx pod is running)
<host mac='7a:e5:d3:9f:09:61' name='mmd-m02' ip='192.168.39.133'/>Lets peek inside our worker m02 routing table and nat chains to see how m02 hands off the packet to the nginx pod/container
> ip route show
default via 192.168.122.1 dev eth1 proto dhcp src 192.168.122.177 metric 1024
...
10.244.1.2 dev veth85b23f64 scope host
...This seems to be a veth device that is attached to a container. We can verify this by inspecting the running containers and the addr associated with the veth devices within them
> sudo crictl ps
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD NAMESPACE
6c9692da0d57f 5f52b9aca7a79 13 hours ago Running nginx-host 0 1c9b1927ce044 nginx-6f764748fc-4wk85 default
ee1108dce6687 4626fe10df5b9 14 hours ago Running kindnet-cni 1 f0fbf1f141558 kindnet-8g44n kube-system
f77ac4d52ac87 d6a28daf3e6b0 14 hours ago Running kube-proxy 1 08679416f77a6 kube-proxy-n4pfh kube-system
> sudo crictl inspect 6c9692da0d57f | grep "pid"
"pid": 1816,
> $ sudo nsenter -t 1816 -n ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
.....
3: eth0@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 7a:a4:12:50:88:8f brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.244.1.2/24 brd 10.244.1.255 scope global eth0
valid_lft forever preferred_lft foreverSo confirmed ! Our simple curl request from the host goes through a really long-winding path of:
- Reaching the worker node
m03and having the destination ip + port rewritten in thePREROUTINGchain - Routed to the worker node
m03by the routing table onm03 - On
m02routed to the veth device inside thenginxcontainer’s network namespace where the nginx process is listening on port 80
Complicated, am I right ?
What happens if we make a curl request directly to the m02 node’s NodePort service from the host ?
Lets look at the PREROUTING table on the node m02 again and surprise, the chain is almost exactly similar to what we had seen in m03 with incoming packets for
the NodePort service having its dnat re-written to the ip:port combo of 10.244.1.2:80 thus getting routed to our container !
$ sudo iptables -t nat -L -v -n
Chain PREROUTING (policy ACCEPT 5 packets, 498 bytes)
pkts bytes target prot opt in out source destination
127 19160 KUBE-SERVICES all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */
Chain KUBE-SERVICES (2 references)
pkts bytes target prot opt in out source destination
....
0 0 KUBE-SVC-HL5LMXD5JFHQZ6LN tcp -- * * 0.0.0.0/0 10.97.146.53 /* default/nginx-svc cluster IP */ tcp dpt:80
1 60 KUBE-NODEPORTS all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL
Chain KUBE-NODEPORTS (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-EXT-HL5LMXD5JFHQZ6LN tcp -- * * 0.0.0.0/0 127.0.0.0/8 /* default/nginx-svc */ tcp dpt:31831 nfacct-name localhost_nps_accepted_pkts
0 0 KUBE-EXT-HL5LMXD5JFHQZ6LN tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/nginx-svc */ tcp dpt:31831
Chain KUBE-EXT-HL5LMXD5JFHQZ6LN (2 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- * * 0.0.0.0/0 0.0.0.0/0 /* masquerade traffic for default/nginx-svc external destinations */
0 0 KUBE-SVC-HL5LMXD5JFHQZ6LN all -- * * 0.0.0.0/0 0.0.0.0/0
Chain KUBE-SVC-HL5LMXD5JFHQZ6LN (2 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ tcp -- * * !10.244.0.0/16 10.97.146.53 /* default/nginx-svc cluster IP */ tcp dpt:80
0 0 KUBE-SEP-SADCJIHRQW7RJ62U all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/nginx-svc -> 10.244.1.2:80 */
Chain KUBE-SEP-SADCJIHRQW7RJ62U (1 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-MARK-MASQ all -- * * 10.244.1.2 0.0.0.0/0 /* default/nginx-svc */
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/nginx-svc */ tcp to:10.244.1.2:80The solution to handling the pre-routing chain is quite elegant ! All the pre-routing for all services whether it is traffic from within or external from cluster start processing at POSTROUTING -> KUBE-SERVICES. From KUBE-SERVICES, if the traffic is from within the cluster, then it gets processed directly via the KUBE-SVC-HL5... chain which handles the DNAT for our nginx service. If the traffic originated from the outside, then it is handled via the KUBE-NODEPORTS -> KUBE-EXT-HL5LMXD5JFHQZ6LN -> KUBE-SVC-HL5.. chain thus all ending up in the same DNAT rule. Elegant, isn’t it ?
How pods can reach the service internally by resolving the nginx.default.svc.cluster.local DNS record
We saw how the PREROUTING chain rewrites the destination ip:port combo to allow the packets originating outside the cluster to be routed to the right container. What happens when we try to reach the nginx service from within the cluster from another pod ?
We can see from the kubectl get output that each kubernetes services has an associated IP address
> k get services -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
kubernetes ClusterIP 10.96.0.1 <none> 443/TCP 28h <none>
nginx-svc NodePort 10.97.146.53 <none> 80:31831/TCP 14h app=nginxRunning an nslookup from another pod/container resolves the same IP for the DNS
> minikube kubectl -p mmd -- exec -it dnsutils -- nslookup nginx-svc.default.svc.cluster.local
Server: 10.96.0.10
Address: 10.96.0.10#53
Name: nginx-svc.default.svc.cluster.local
Address: 10.97.146.53Remember the PREROUTING tables from the previous section, yes the same rules now apply to within cluster traffic as well !
Each container in the worker node is connected to the worker node via a veth pair
> ip addr show
...
6: veth6adf44b8@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
link/ether c6:47:04:73:98:b7 brd ff:ff:ff:ff:ff:ff link-netns cni-55c28e65-ac40-f2d7-ede3-8acb435f8d11Just like a packet coming into the eth0 interface connect to the host (and bridge) any packet originating from a pod/container is also processed via the PREROUTING chain (since it is coming INTO the worker node, of sorts) and our PREROUTING chain performs the same dance as above through the KUBE-SERVICES chain
Chain KUBE-SERVICES (2 references)
pkts bytes target prot opt in out source destination
0 0 KUBE-SVC-ERIFXISQEP7F7OF4 tcp -- * * 0.0.0.0/0 10.96.0.10 /* kube-system/kube-dns:dns-tcp cluster IP */ tcp dpt:53
0 0 KUBE-SVC-JD5MR3NA4I4DYORP tcp -- * * 0.0.0.0/0 10.96.0.10 /* kube-system/kube-dns:metrics cluster IP */ tcp dpt:9153
0 0 KUBE-SVC-HL5LMXD5JFHQZ6LN tcp -- * * 0.0.0.0/0 10.97.146.53 /* default/nginx-svc cluster IP */ tcp dpt:80
0 0 KUBE-SVC-NPX46M4PTMTKRN6Y tcp -- * * 0.0.0.0/0 10.96.0.1 /* default/kubernetes:https cluster IP */ tcp dpt:443
9 829 KUBE-SVC-TCOU7JCQXEZGVUNU udp -- * * 0.0.0.0/0 10.96.0.10 /* kube-system/kube-dns:dns cluster IP */ udp dpt:53
5 300 KUBE-NODEPORTS all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCALSee how rule 3 matches the destination 10.97.146.53: which is our clusterIP for the nginx-svc ? And the target is our Chain KUBE-SVC-HL5LMXD5JFHQZ6LN which handled the NodePort traffic as well ?
** Magic ! Pure Magic ! **
Conclusion
I had been putting away exploring this topic for almost a year now. The idea to explore how Kubernetes services were setup was something that I thought of last September, but work and other commitments sadly didn’t leave me the time to explore it further. And now that I finally did, I am positively shocked at the elegance of these ideas ! The same PREROUTING chains in both the workers even when the Pod was running on only one node ? Mind boggling ! There is an entirely different topic of how routing works with multiple pods per service, but that’s probably left for another post. Even exploring and mapping this much turned out to be a time-consuming exercise and I haven’t even began to explore conntrack or the MARK & MASQUERADE techniques yet, but hey mor content for the future am I right ?
Nonetheless, exploring Kubernetes internals has been a deeply satisfying experience (thanks to LLMs that help so much with debugging and explaining very tersely written Linux documentation) and I cannot stop but marvel at the elegance of simple ideas composed together beautifully, just like chains in an iptable !