01 - Kubernetes Networking: CNI, Cilium with eBPF and Network Policy

Kubernetes Networking: CNI, Cilium with eBPF and Network Policy

Kubernetes abstracts the network in a masterly way: each Pod gets a routable IP address, containers within the same Pod share the network namespace, and Pods can communicate with each other on any node without NAT. Yet, the apparent simplicity hides a real complexity: how does this model actually work? Who deals with assign IPs, route traffic between nodes, implement network policies?

The answer lies in Container Network Interface (CNI), a standard that defines how networking plugins should integrate with Kubernetes. In this article we'll explore the Kubernetes networking model from the inside: how kube-proxy works, why Cilium with eBPF is it replacing traditional solutions, and how implement Network Policy to isolate workloads in production clusters.

The Kubernetes Network Model

Kubernetes imposes a networking model with four fundamental requirements that any CNI implementation must comply with:

• All Pods can communicate with all other Pods without NAT
• All nodes can communicate with all Pods without NAT
• The IP that a Pod sees as its own and the same IP that others use to reach it
• Containers within a Pod share network namespaces and IPs

This "flat network model" greatly simplifies application reasoning: a microservice does not need to know whether the service it calls is on the same node or on a node remote. The complexity is moved to the cluster network layer.

How Communication Between Pods Works

When a Pod A wants to communicate with a Pod B on different nodes, the typical flow is the following with a solution based on overlay network (e.g. VXLAN):

1. The package exits Pod A's container through the interface eth0
2. Enter the node namespace through a veth pair
3. The CNI plugin intercepts it and encapsulates it in VXLAN (or GRE, Geneve...)
4. The packet traverses the physical network to the Pod B node
5. The CNI on the destination node de-encapsulates the packet
6. The packet arrives at Pod B through its veth pair

With solutions based on BGP or native routing (like Cilium in native routing mode), encapsulation is not necessary and performance improves significantly.

Container Network Interface (CNI)

The CNI is a CNCF specification that defines how container runtimes should invoke network plugins. Kubernetes uses CNI to delegate network management to plugins: when the kubelet creates a Pod, it calls the configured CNI plugin to allocate the address IP and configure the network interface.

Main CNI Plugins

kube-proxy: The Old Approach

kube-proxy is the Kubernetes component responsible for implementing the Services: when a client calls a Service, kube-proxy ensures that the traffic arrives at one of the Pod backend. Traditionally uses iptables for this purpose.

The problem with iptables is that complexity scales linearly with the number of rules. In a cluster with 10,000 Services and 100,000 endpoints, iptables manages millions of rules. Every packet must pass through this chain of rules, with significant impact on the latency and CPU of the node.

The in-tree alternative e IPVS (IP Virtual Server), which uses a hash table for O(1) lookup instead of iptables linear scan. But IPVS also has limitations: It does not support advanced policies and still requires the management of additional iptables rules.

Cilium and eBPF: The Future of Kubernetes Networking

eBPF (extended Berkeley Packet Filter) is a Linux kernel technology which allows you to run sandboxed programs directly in the kernel, without modifying it e without kernel modules. Cilium uses eBPF to intercept and process network traffic at the kernel level, completely bypassing iptables and kube-proxy.

Benefits of Cilium with eBPF

• Performance: Latency reduced by up to 30%, throughput increased by 60% compared to iptables
• Scalability: O(1) lookup with BPF maps instead of iptables linear scan
• Observability: Hubble provides real-time visibility of L3/L4/L7 traffic
• Layer 7 Policy: Policies based on HTTP path, method, header, gRPC method, Kafka topic
• Service Mesh without sidecar: mTLS and load balancing implemented in the kernel, zero sidecar overhead
• Kube proxy replacement: Cilium can completely replace kube-proxy

Cilium installation

We install Cilium on a Kubernetes cluster using Helm, configuring it to replace kube-proxy and enable Hubble for observability:

# Aggiungi il repo Helm di Cilium
helm repo add cilium https://helm.cilium.io/
helm repo update

# Installa Cilium con kube-proxy replacement e Hubble abilitati
helm install cilium cilium/cilium \
  --version 1.16.0 \
  --namespace kube-system \
  --set kubeProxyReplacement=true \
  --set k8sServiceHost=API_SERVER_HOST \
  --set k8sServicePort=API_SERVER_PORT \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true \
  --set ipam.mode=kubernetes

# Verifica installazione
cilium status --wait

# Verifica connettivita
cilium connectivity test

Cilium in Production: Advanced Configuration

For a production cluster, here is a complete Helm values configuration with native routing (without VXLAN encapsulation) to maximize performance:

# cilium-values-production.yaml
kubeProxyReplacement: true
k8sServiceHost: "10.0.0.1"  # indirizzo API server
k8sServicePort: "6443"

# Native routing mode (senza overlay VXLAN)
# Richiede che la rete sottostante supporti il routing dei pod CIDR
tunnel: disabled
autoDirectNodeRoutes: true
ipv4NativeRoutingCIDR: "10.244.0.0/16"

# IPAM
ipam:
  mode: kubernetes

# BGP per annunciare i pod CIDR ai router
bgp:
  enabled: true
  announce:
    podCIDR: true
    lbIP: true

# Hubble - osservabilita Layer 7
hubble:
  enabled: true
  metrics:
    enabled:
      - dns:query;ignoreAAAA
      - drop
      - tcp
      - flow
      - icmp
      - http
  relay:
    enabled: true
    replicas: 2
  ui:
    enabled: true

# Monitoring con Prometheus
prometheus:
  enabled: true
  serviceMonitor:
    enabled: true

# Encryption (WireGuard)
encryption:
  enabled: true
  type: wireguard

# Load Balancing con DSR (Direct Server Return)
loadBalancer:
  mode: dsr  # elimina un hop di rete nel path di ritorno

# Limita connessioni per Pod
bpf:
  mapDynamicSizeRatio: 0.0025

Network Policy in Kubernetes

By default, all Pods in a Kubernetes cluster can freely communicate with each other. This is convenient for development but is a security issue in production. The NetworkPolicy allow you to define entry and egress rules for Pods.

Default Deny: The Fundamental Best Practice

The first step to isolate workloads and apply a policy default-deny on each namespace. This denies all traffic not explicitly authorized:

# default-deny-all.yaml
# Nega tutto il traffico ingress e egress nel namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}  # seleziona TUTTI i pod nel namespace
  policyTypes:
    - Ingress
    - Egress
---
# Permetti il traffico DNS (necessario per la risoluzione dei nomi)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns
  namespace: production
spec:
  podSelector: {}
  policyTypes:
    - Egress
  egress:
    - ports:
        - protocol: UDP
          port: 53
        - protocol: TCP
          port: 53

NetworkPolicy for a Typical Application

Let's see how to isolate a three-tier application: frontend, backend, database. Only the frontend accepts traffic from outside, only the backend can reach the DB:

# network-policies-app.yaml

# Frontend: accetta traffico dall'ingress controller
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-ingress-to-frontend
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: frontend
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: ingress-nginx
          podSelector:
            matchLabels:
              app.kubernetes.io/name: ingress-nginx
      ports:
        - protocol: TCP
          port: 8080
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: backend
      ports:
        - protocol: TCP
          port: 8000
    - ports:
        - protocol: UDP
          port: 53
---
# Backend: accetta solo dal frontend, puo raggiungere DB
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-backend
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
    - Ingress
    - Egress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: frontend
      ports:
        - protocol: TCP
          port: 8000
  egress:
    - to:
        - podSelector:
            matchLabels:
              app: database
      ports:
        - protocol: TCP
          port: 5432
    - ports:
        - protocol: UDP
          port: 53
---
# Database: accetta solo dal backend
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-backend-to-database
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: database
  policyTypes:
    - Ingress
  ingress:
    - from:
        - podSelector:
            matchLabels:
              app: backend
      ports:
        - protocol: TCP
          port: 5432

CiliumNetworkPolicy: Policy Layer 7

Standard Kubernetes NetworkPolicies operate at Layer 3/4 (IP and port). Cilium extends this with CiliumNetworkPolicy, which allows content-based policies of communication: HTTP path, gRPC method, Kafka topic, DNS query.

HTTP Policy with Cilium

We allow the frontend to call only specific backend endpoints:

# cilium-http-policy.yaml
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: frontend-to-backend-l7
  namespace: production
spec:
  endpointSelector:
    matchLabels:
      app: backend
  ingress:
    - fromEndpoints:
        - matchLabels:
            app: frontend
      toPorts:
        - ports:
            - port: "8000"
              protocol: TCP
          rules:
            http:
              - method: "GET"
                path: "/api/v1/products.*"
              - method: "POST"
                path: "/api/v1/orders"
              - method: "GET"
                path: "/health"

DNS Policy with Cilium

Limit the DNS domains that a Pod can resolve (useful for preventing data exfiltration):

# cilium-dns-policy.yaml
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: restrict-dns-egress
  namespace: production
spec:
  endpointSelector:
    matchLabels:
      app: backend
  egress:
    # Permetti solo DNS verso il cluster DNS
    - toEndpoints:
        - matchLabels:
            k8s:io.kubernetes.pod.namespace: kube-system
            k8s:k8s-app: kube-dns
      toPorts:
        - ports:
            - port: "53"
              protocol: ANY
          rules:
            dns:
              - matchPattern: "*.internal.company.com"
              - matchPattern: "*.svc.cluster.local"
              - matchPattern: "api.stripe.com"

Hubble: Network Observability

Hubble and the Cilium observability layer. Provides real-time visibility on all cluster network traffic, with communications identities based on Kubernetes labels instead of IP addresses.

# Installa il CLI di Hubble
export HUBBLE_VERSION=$(curl -s https://raw.githubusercontent.com/cilium/hubble/master/stable.txt)
curl -L --fail --remote-name-all \
  https://github.com/cilium/hubble/releases/download/$HUBBLE_VERSION/hubble-linux-amd64.tar.gz
tar xzvf hubble-linux-amd64.tar.gz
sudo mv hubble /usr/local/bin

# Port-forward al relay Hubble
cilium hubble port-forward &

# Osserva il traffico in tempo reale
hubble observe --namespace production --follow

# Filtra per Pod specifici
hubble observe \
  --namespace production \
  --from-pod frontend-7d9d6b8f-abc12 \
  --to-pod backend-5c4f8d9-xyz99 \
  --follow

# Mostra solo i drop (traffico bloccato dalle policy)
hubble observe \
  --namespace production \
  --verdict DROPPED \
  --follow

# Statistiche per service
hubble observe \
  --namespace production \
  --output json | jq '.flow.destination.namespace'

Network Policy Debugging and Troubleshooting

Debugging NetworkPolicies is one of the most common challenges in Kubernetes. Here's one approach systematic:

# 1. Verifica quali NetworkPolicy si applicano a un Pod
kubectl get networkpolicies -n production -o wide

# 2. Test di connettivita con un Pod temporaneo
kubectl run test-pod \
  --image=nicolaka/netshoot \
  --rm \
  -it \
  --restart=Never \
  -n production \
  -- bash

# All'interno del Pod:
# Test TCP
nc -zv backend-service 8000
# Test DNS
nslookup backend-service.production.svc.cluster.local
# Test HTTP
curl -v http://backend-service:8000/health

# 3. Con Cilium, usa il tool di policy verification
cilium policy get  # mostra tutte le policy caricate

# 4. Testa la connettivita specifica
kubectl exec -n production frontend-pod -- \
  curl -v http://backend-service:8000/api/v1/products

# 5. Con Hubble, vedi perche un pacchetto viene droppato
hubble observe \
  --namespace production \
  --verdict DROPPED \
  --from-pod frontend-7d9d6b8f \
  --follow

Multi-Namespace Isolation

In multi-tenant clusters, it is critical to isolate namespaces. By default, NetworkPolicies they do not block cross-namespace traffic. Here's how to implement complete isolation:

# Isola completamente un namespace da tutti gli altri
# (ma permette il traffico interno al namespace)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: namespace-isolation
  namespace: tenant-a
spec:
  podSelector: {}
  policyTypes:
    - Ingress
    - Egress
  ingress:
    # Permetti solo traffico dallo stesso namespace
    - from:
        - podSelector: {}
    # Permetti da monitoring namespace
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: monitoring
  egress:
    # Permetti solo traffico verso lo stesso namespace
    - to:
        - podSelector: {}
    # Permetti DNS
    - ports:
        - protocol: UDP
          port: 53
    # Permetti verso monitoring namespace
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: monitoring

Performance Benchmark: Cilium vs kube-proxy

Benchmarks show significant differences between Cilium with eBPF and kube-proxy with iptables in clusters with a high number of Services:

Best Practices for Kubernetes Networking

Anti-Patterns to Avoid

Conclusions and Next Steps

The Kubernetes networking model, with its flat, IP-per-Pod approach, is elegant in design its simplicity yet sophisticated in implementation. Choosing the CNI plugin is one of the most important architectural decisions for a production cluster: influence the performance, security and observability features available.

Cilium with eBPF is the most advanced CNI available today: replace kube-proxy with superior performance, offers Layer 7 policies, integrates observability with Hubble and can replace a traditional service mesh for mTLS. For new clusters in production, and the choice recommended by the CNCF community and large players such as Google, Amazon and Microsoft who use it in their managed Kubernetes services.

Network policies, implemented correctly with a default-deny approach, reduce the attack surface drastically if a Pod is compromised. I'm not optional in production: they are a fundamental safety requirement.