Understanding Kubernetes Networking: Pods, Services, CNI, and Service Mesh

Kubernetes has become the de facto standard for container orchestration. According to the CNCF Annual Survey, more than 90% of organizations running containers in production rely on Kubernetes in some form. Yet despite its widespread adoption, networking remains one of the most misunderstood parts of the platform.

The reason is simple: Kubernetes networking solves a problem that traditional infrastructure rarely encounters. In a conventional environment, servers have stable IP addresses, applications stay on the same machines for months, and communication paths remain predictable. Kubernetes operates under completely different assumptions. Containers are constantly created, destroyed, rescheduled, and scaled across multiple nodes. The network must continue working even while the underlying infrastructure changes every second.

This article explores how Kubernetes networking works under the hood, why Container Network Interface (CNI) plugins exist, and how technologies such as Calico, Flannel, Cilium, and Service Mesh fit into the overall architecture.

Why Kubernetes Networking Is Different

At the heart of Kubernetes lies a fundamental challenge: workloads are ephemeral.

A Pod, which represents the smallest deployable unit in Kubernetes, is not designed to live forever. Pods can be recreated during deployments, replaced after failures, moved to different nodes by the scheduler, or scaled up and down according to demand.

Every time this happens, the Pod receives a new IP address.

In traditional infrastructure, applications often communicate with a known server address. In Kubernetes, that approach would quickly fail because the destination may disappear moments later. The platform therefore needs a networking model that allows constantly changing workloads to communicate reliably without applications having to track individual IP addresses.

This requirement drives almost every networking component inside a Kubernetes cluster.

The Three Networks Inside Kubernetes

Although Kubernetes networking often appears complex, it can be understood as three separate but interconnected layers.

The first layer is the node network. This is the physical or virtual infrastructure connecting cluster nodes. Kubernetes does not create or manage this network. Whether nodes are connected through data-center switches, cloud VPCs, or virtual networks is largely irrelevant to Kubernetes itself.

The second layer is the pod network. Every Pod receives its own IP address and must be able to communicate directly with every other Pod in the cluster without Network Address Translation (NAT).

The third layer is the service network. Since Pod addresses constantly change, Kubernetes provides virtual addresses called Services that act as stable entry points for groups of Pods.

Together, these three layers create the illusion of a stable and predictable environment despite the dynamic nature of container workloads.

The Node Network: The Foundation Beneath Kubernetes

The node network is the simplest part of the architecture.

Every Kubernetes node has one or more network interfaces and participates in a conventional IP network. Nodes communicate with each other using standard Layer 2 switching or Layer 3 routing.

The control plane depends on this connectivity to manage worker nodes. Components such as kubelet, the API server, and etcd exchange information through the node network continuously.

However, there is an important limitation.

The node network only knows about node IP addresses. It has no knowledge of Pod IP addresses. From the perspective of routers and switches, Pods do not exist.

This creates the central networking challenge of Kubernetes: how can a Pod on one node communicate with a Pod on another node if the underlying infrastructure only understands node addresses?

How Pods Communicate on the Same Node

Communication becomes relatively straightforward when Pods reside on the same node.

When Kubernetes creates a Pod, the CNI plugin builds a virtual Ethernet pair, commonly known as a veth pair. One end is placed inside the Pod's network namespace, while the other remains attached to a virtual bridge on the host.

This bridge behaves much like a software switch.

As a result, traffic between Pods on the same node never leaves the machine. Packets are switched locally through the virtual bridge, providing low latency and minimal overhead.

From the application's perspective, it simply communicates with another IP address. The underlying complexity remains invisible.

Cross-Node Communication: The Real Challenge

Things become more interesting when Pods are located on different nodes.

Imagine a Pod on Node A attempting to send traffic to a Pod on Node B.

The source Pod knows the destination Pod's IP address. The problem is that the physical network does not. Routers between nodes understand node addresses but have no route to the Pod subnet.

Without additional mechanisms, the packet would simply be dropped.

To solve this problem, Kubernetes networking solutions generally rely on one of two approaches: overlay networking or direct routing.

Overlay Networks and VXLAN

Overlay networking creates a virtual network on top of the existing infrastructure.

Instead of teaching the physical network about Pod addresses, the original packet is encapsulated inside another packet. The outer packet contains node IP addresses that the underlying network already understands.

The most common encapsulation technology used in Kubernetes environments is VXLAN.

When a Pod sends traffic to a remote Pod, the CNI plugin wraps the packet inside a VXLAN tunnel and forwards it to the destination node. The receiving node removes the encapsulation header and delivers the original packet to the target Pod.

This approach offers remarkable flexibility because it works almost everywhere. The underlying network does not need special routing capabilities or awareness of Kubernetes.

That simplicity explains why Flannel became one of the most popular networking solutions for learning environments and small clusters.

However, encapsulation introduces overhead. Every packet becomes slightly larger, reducing the effective Maximum Transmission Unit (MTU). Additional processing is also required to encapsulate and decapsulate traffic, consuming CPU resources.

While these costs are usually acceptable for small deployments, they become increasingly noticeable in large-scale environments handling millions of packets per second.

Direct Routing and BGP

An alternative approach avoids encapsulation entirely.

Instead of hiding Pod addresses from the infrastructure, direct-routing solutions teach the network how to reach Pod subnets.

Each node receives responsibility for a specific Pod CIDR block. Routing information is then distributed throughout the cluster so that traffic can be forwarded directly to the correct node.

This is where Border Gateway Protocol (BGP) enters the picture.

BGP is the same protocol that powers large portions of the internet. Within Kubernetes, solutions such as Calico use BGP to advertise Pod routes between nodes.

When a Pod sends traffic to another Pod, the packet travels using standard routing mechanisms. No encapsulation occurs, and the packet remains unchanged throughout its journey.

The result is lower latency, reduced CPU overhead, and improved throughput.

For organizations operating high-performance environments, this efficiency can become a significant advantage.

Why Kubernetes Avoids NAT

One of Kubernetes' core networking principles is that Pods should communicate using their real IP addresses.

This differs from traditional Docker networking, where containers often sit behind NAT and share the host's address.

Avoiding NAT provides several important benefits.

Observability becomes easier because applications can see the true source of traffic. Security controls can identify specific Pods instead of entire nodes. Network policies can enforce communication rules between workloads with much greater precision.

Without real Pod addresses, implementing fine-grained security boundaries would become significantly more difficult.

This design decision is one of the reasons Kubernetes networking feels more complex than traditional container networking, but it also enables far stronger security and operational visibility.

The Role of CNI

Kubernetes itself does not build networks.

Instead, it defines a set of networking requirements and delegates implementation to a Container Network Interface (CNI) plugin.

The CNI specification acts as a contract between Kubernetes and networking providers.

When a new Pod is created, Kubernetes invokes the configured CNI plugin. The plugin allocates an IP address, creates network interfaces, configures routing, applies firewall rules, and integrates the Pod into the cluster network.

When the Pod is deleted, the same plugin cleans up those resources.

This separation allows different networking technologies to coexist under a common interface while giving operators freedom to choose the implementation that best fits their environment.

Services: Solving the Service Discovery Problem

Even with Pod-to-Pod connectivity solved, applications still face another challenge.

Pods come and go constantly.

A frontend service cannot reliably connect to a backend Pod whose address may change during the next deployment.

Kubernetes addresses this problem through Services.

A Service receives a stable virtual IP address and acts as a permanent endpoint for a group of Pods selected by labels.

Applications communicate with the Service rather than individual Pods. Kubernetes then distributes requests among healthy backend instances.

This abstraction enables rolling updates, automatic recovery, and horizontal scaling without requiring configuration changes in client applications.

Comparing Flannel, Calico, and Cilium

Flannel remains one of the simplest networking solutions available. Its VXLAN-based architecture makes deployment straightforward and minimizes infrastructure requirements. For learning Kubernetes or running small clusters, Flannel often provides everything needed.

Calico takes a more production-oriented approach. By leveraging routing instead of pure overlays, it delivers better performance and introduces one of Kubernetes' most important security features: Network Policies.

These policies allow administrators to define exactly which workloads may communicate with each other. In modern zero-trust environments, such controls are increasingly viewed as mandatory rather than optional.

Cilium represents the newest generation of Kubernetes networking. Built around eBPF technology, it moves packet processing into the Linux kernel itself.

This architecture enables advanced observability, lower latency, sophisticated security controls, and highly efficient traffic handling. Major cloud providers and large-scale platforms have increasingly adopted Cilium as eBPF technology has matured.

Network Policies and Zero-Trust Networking

In many Kubernetes environments, connectivity alone is not enough.

Without restrictions, every Pod may potentially communicate with every other Pod. This flat network model creates unnecessary attack surfaces and increases the impact of compromised workloads.

Network Policies provide a mechanism for defining explicit communication rules.

For example, a frontend service may be allowed to communicate with the backend, while direct access to the database remains prohibited. If an attacker compromises a frontend Pod, lateral movement opportunities become significantly more limited.

This approach aligns closely with modern zero-trust security principles, which assume that no workload should be trusted by default.

Service Mesh: The Next Layer Above Networking

As microservice architectures grow, networking requirements extend beyond basic connectivity.

Organizations begin demanding encrypted service-to-service communication, automatic retries, traffic shaping, observability, and deployment controls.

These capabilities are typically delivered through a Service Mesh.

Solutions such as Istio and Linkerd introduce sidecar proxies that intercept traffic entering and leaving Pods. Applications continue operating normally, while the mesh transparently adds advanced networking functionality.

This architecture allows teams to implement mutual TLS encryption, collect detailed telemetry, enforce traffic policies, and perform canary deployments without modifying application code.

In large environments containing dozens or hundreds of services, these capabilities can dramatically simplify operations.

Choosing the Right Networking Stack

There is no universal answer to the question of which CNI plugin is best.

For labs, educational environments, and small clusters, Flannel remains an excellent starting point due to its simplicity.

For most production deployments, Calico offers a strong balance of performance, security, maturity, and operational stability.

Organizations pursuing advanced observability, large-scale networking, and modern cloud-native architectures increasingly look toward Cilium and eBPF-based solutions.

The optimal choice ultimately depends on infrastructure constraints, operational expertise, security requirements, and performance goals.

Conclusion

Kubernetes networking exists to solve a deceptively difficult problem: maintaining reliable communication between workloads that constantly appear, disappear, and move throughout a cluster.

Node networks provide the underlying transport layer. Pod networks enable direct workload communication. Services create stable endpoints for dynamic applications. CNI plugins implement the actual networking model, while Service Mesh platforms add advanced traffic management and security capabilities.

Understanding how these pieces fit together is essential for anyone operating Kubernetes in production. Once the underlying principles become clear, many of the platform's seemingly mysterious networking behaviors begin to make perfect sense.