Why Is Kubernetes Networking So Confusing?
Translated from the original Korean post. 한국어 원문 보기 →
Timed out. Again.
The deploy went green. The browser gives me nothing but a timeout. kubectl says the pod is Running, happy as can be. And I have no idea why I can't reach it.
Everybody hits this wall the first time they touch Kubernetes networking.
I did. Years ago, first cluster, I figured: pod's up, so I'll just curl the pod IP. Nope. At the time I couldn't understand why anyone would build something this convoluted. Looking back the cause was boring — I didn't know that pods, Services, and nodes are three separate layers, each with its own networking rules. I was treating them as one thing, so nothing lined up.
Networking comes down to one question: at which layer does a packet move, from whom, to whom. Once that layer split clicks, most of the confusion goes away.
How a pod gets an IP
In Kubernetes, IPs are assigned per pod, not per container. Multiple containers in one pod share the same IP and talk to each other over localhost.
On a single node it's simple.
| 엔티티 | IP 주소 | 네트워크 |
|---|---|---|
| 내 노트북 | 192.168.1.10 | 홈 LAN |
| 쿠버네티스 노드 | 192.168.1.2 | 홈 LAN |
| Pod A | 10.244.0.2 | 클러스터 내부 |
| Pod B | 10.244.0.3 | 클러스터 내부 |
When Kubernetes comes up it creates an internal network — usually 10.244.0.0/16 — and pods draw their IPs from it. Pods on the same node can talk directly.
One thing you have to internalize: pod IPs change. Restart a pod, roll out an update, and the IP is gone. So hardcoding a pod IP into a config file or your code is planting a time bomb. It runs fine for weeks, then one pod restarts and the connection dies. Those failures are especially annoying to chase down, precisely because everything works most of the time.
What breaks with multiple nodes
Add a second node and the picture gets messy. If each node independently carves out 10.244.0.0/16, you end up with pods on different nodes holding the same IP. Same address in two places, and routing has nothing to work with.
So Kubernetes nails down a few requirements. Every pod must reach every other pod without NAT. Every node must reach every pod, and every pod must reach every node.
The fun part: Kubernetes itself specifies how these requirements must be satisfied and then doesn't implement any of it. It publishes the contract and delegates the implementation. The thing that takes that delegation is the CNI (Container Network Interface) plugin. Interface separated from implementation — which is why you can swap the network layer to fit your environment.
The CNI plugins
You pick a CNI at cluster install time, and from then on that plugin owns per-node subnet allocation, routing setup, and node-to-node traffic. All the actual plumbing happens there.
The main options:
| CNI | 특징 |
|---|---|
| Calico | 범용성 좋음, 강력한 네트워크 정책 지원 |
| Flannel | 단순함, '그냥 작동하는' 클래식한 선택 |
| Cilium | eBPF 기반, 고성능, 뛰어난 관측성 |
| Weave Net | 멀티클라우드 설정이 쉬움 |
Managed Kubernetes (EKS, GKE, AKS) ships a default CNI, but you can replace it. Lately I've seen noticeably more teams moving to Cilium. The eBPF-backed observability seems to be the big draw. Half of debugging a network problem is figuring out where the packet died, and anyone who's operated a cluster knows what a tool that shows you that is worth.
Once the CNI is in, each node gets its own non-overlapping subnet — node 1 takes 10.244.0.0/24, node 2 takes 10.244.1.0/24, and so on. No address collisions, so routing works, and only then can every pod actually reach every other pod.
Why Services exist
Pods disappear and come back, as covered. Their IPs change with them. So how does your frontend call your backend reliably? How do you chase an address that keeps moving?
You don't. That's what a Service is for. It does three things: groups pods with a label selector, holds a stable IP and DNS name of its own, and load-balances traffic across the matching pods.
Think of it as a fixed sign hung in front of a shifting crowd of pods. Callers only look at the sign; they don't care how many pods came up or died behind it. That one layer of abstraction is the heart of Kubernetes networking.
Three Service types cover almost everything you'll run into day to day.
ClusterIP: the default internal Service
Reachable only from inside the cluster. This is the type you'll use most.
apiVersion: v1
kind: Service
metadata:
name: backend
spec:
type: ClusterIP # 생략해도 됨 (기본값)
ports:
- port: 80 # 서비스가 노출하는 포트
targetPort: 8080 # 파드가 실제 받는 포트
selector:
app: backend # app=backend 레이블 파드 선택
Apply it and you get:
kubectl apply -f backend-service.yaml
kubectl get svc backend
# NAME TYPE CLUSTER-IP PORT(S) AGE
# backend ClusterIP 10.100.20.4 80/TCP 5s
Now any other pod in the cluster can call it:
curl http://backend # DNS로 해결
curl http://10.100.20.4 # IP로도 가능하지만 이름 추천
In practice the layout is usually frontend pods calling a backend ClusterIP Service, and backend pods calling Redis and MySQL ClusterIP Services.
Each tier scales and redeploys on its own schedule, and because the Service holds a stable endpoint underneath, the caller's config never has to change. Anyone who's run microservices knows how much that "the endpoint doesn't move" property matters in operations. The moment services depend on each other by IP, deploys start tripping over each other.
Debugging has an order to it. When a connection fails, kubectl exec into a pod and run nslookup backend, then curl -v http://backend. If DNS resolves but curl hangs, it's probably a selector mismatch. Narrowing in that order splits DNS vs. selector vs. the pod itself fast.
NodePort: opening the door from outside
Opens the same port on every node so external traffic can get in.
apiVersion: v1
kind: Service
metadata:
name: myapp-service
spec:
type: NodePort
ports:
- port: 80 # 서비스 포트 (내부)
targetPort: 80 # 파드 포트
nodePort: 30008 # 노드 포트 (외부, 30000-32767)
selector:
app: myapp
NodePort tangles three ports together, which trips people up at first.
- nodePort: 외부에서 접근하는 포트 (30008)
- port: 클러스터 내부 서비스 포트 (80)
- targetPort: 파드가 실제 받는 포트 (80)
Leave nodePort out and one gets picked automatically from the 30000-32767 range.
kubectl apply -f nodeport-service.yaml
# 외부에서 접근
curl http://192.168.1.2:30008
curl http://192.168.1.3:30008 # 모든 노드에서 같은 포트
Multiple pods get load-balanced automatically, and it doesn't matter which node you hit even if the pods are spread across several. Know one node address and you're in.
NodePort is a bad fit for a production-facing service, though. You're exposing an arbitrary high port in the 30000s, and you get no TLS termination and no host-based routing at all. Demos and internal testing are about as far as I'd take it.
LoadBalancer: production-grade exposure
Provisions a real load balancer from your cloud provider.
apiVersion: v1
kind: Service
metadata:
name: voting-app
spec:
type: LoadBalancer
ports:
- port: 80
targetPort: 80
selector:
app: voting-app
Apply it and an external IP shows up:
kubectl get svc voting-app
# NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
# voting-app LoadBalancer 10.100.55.9 34.102.77.14 80:31842/TCP 45s
Point DNS at 34.102.77.14 and the service is public. On a cloud provider, that single line — type: LoadBalancer — is a command that provisions real cloud LB resources. Convenient, and each LB carries a bill. Slap a LoadBalancer on every Service and they pile up while the invoice quietly grows. That's why production setups usually put one Ingress in front and park the Services behind it.
Watch out for one thing. In environments that don't support the LoadBalancer type (VirtualBox, home labs), EXTERNAL-IP just sits at <pending> forever. There's no cloud controller, so nothing is there to provision it. Behavior falls back to NodePort.
If you want LoadBalancer semantics on bare metal, install MetalLB.
Picking a type
The decision is simpler than it looks. Called only from inside the cluster? ClusterIP. Needs external access on a cloud provider? LoadBalancer, usually behind an Ingress. Needs external access on bare metal or a demo box? NodePort or MetalLB.
| 타입 | 용도 | 외부 접근 | 프로덕션 적합성 |
|---|---|---|---|
| ClusterIP | 내부 서비스 | X | ⭐⭐⭐ |
| NodePort | 개발/데모 | O | ⭐ |
| LoadBalancer | 공개 서비스 | O | ⭐⭐⭐ |
Get clear on who calls this Service and the type picks itself. Using an externally exposed type for something nothing outside ever calls is leaving a door wide open when you could just lock it.
The failures you'll keep seeing
These patterns show up over and over in the field. Handy for going from symptom to cause.
EXTERNAL-IP stuck at <pending> means you used the LoadBalancer type in an environment with no cloud controller. Switch to NodePort or install MetalLB. If you hit a NodePort and get "Connection refused," the selector usually isn't matching any pods — run kubectl get endpoints <svc> first and see whether ENDPOINTS is empty.
DNS resolves but curl fails: there are matching pods, but they aren't Ready. Check your readinessProbe. Pod-to-pod traffic failing across nodes is a CNI problem nine times out of ten, and the first move is kubectl get pods -n kube-system to confirm the CNI pods are Running.
There's a pattern here. Most networking failures are half-solved the moment you correctly identify which layer broke. Service, selector, pod, or CNI. Splitting by layer beats tearing into the yaml at random.
Wrapping up
Kubernetes networking looks complicated at first because it looks like one thing. Pull it apart and it reduces to two axes: pod networking and the Service abstraction. CNI owns pod-to-pod traffic, and Services provide the endpoint that doesn't move. It's a design that separates what changes (pods) from what must not (Services) — hold it that way and it gets easier to reason about.
ClusterIP for internal traffic, NodePort or LoadBalancer for external depending on where you're running. Get comfortable with just that and most day-to-day networking problems start to untangle.
Come to think of it, that timeout error that ate hours of my life years ago was just this: I never made the Service, and I tried to attach straight to the pod. I just didn't know about the layers.
Was this post helpful?
One click helps me write the next one