InfoDrip
KO

How Kubernetes DNS Actually Works — CoreDNS, ndots, and Why Your Lookups Are Slow

·Platform Decision·8 min read

Translated from the original Korean post. 한국어 원문 보기 →

How Do Pods Actually Find Each Other in Kubernetes?

There's something slightly unsettling about your first Kubernetes cluster. The frontend pod talks to the backend service just fine. The backend connects to the database without a fuss. You never hardcoded a single IP, and yet everything finds everything. For a while I just accepted it.

Then the question shows up. How do pods actually locate each other? Pods die and come back with new IPs constantly, and somehow the connections keep working.

The answer is DNS. More specifically, CoreDNS. That quiet little workload sitting in your cluster is doing basically all of the service discovery. Most people never look at it until an external API call starts crawling for no apparent reason. That's how I ended up looking.

So let's walk through what this hidden worker actually does, and where the occasional DNS slowdown comes from.

CoreDNS Is the Cluster's Phone Book

CoreDNS became the default DNS server in Kubernetes 1.13, back in 2018. Before that it was kube-dns. CoreDNS is lighter and its plugin architecture makes configuration far more flexible, so it took over completely.

Strip away the details and CoreDNS is a phone book. A pod asks "where's the service I'm trying to reach?" and CoreDNS answers "here's the IP."

It normally runs as a Deployment in the kube-system namespace, with 2 replicas for availability. In front of it sits a Service named kube-dns, which provides a stable ClusterIP.

One thing worth flagging: the switch happened, but the Service is still called kube-dns. The name stuck around for backward compatibility. Mismatches like this — name says one thing, reality is another — will trip you up during debugging more than once. Either way, every pod in the cluster points at that ClusterIP as its DNS server.

What Happens Inside a Pod During a Lookup

Here's the sequence when a pod resolves a service name.

Step 1: the query fires A pod wants to reach my-service in its own namespace. It sends the query to whatever DNS server is configured in its /etc/resolv.conf.

Step 2: it lands on CoreDNS When kubelet starts the pod, it writes the kube-dns Service IP into /etc/resolv.conf automatically. The application doesn't have to know or care. Every DNS query goes to CoreDNS by default.

Step 3: internal or external? CoreDNS looks at the query and decides whether it's a cluster-internal name or an outside domain. Something like my-service.default.svc.cluster.local gets resolved against the Kubernetes API and comes back as a ClusterIP. Something like google.com gets forwarded upstream.

Step 4: the IP comes back The pod gets its answer and opens the connection. When everything is healthy, the whole round trip takes a few milliseconds. The catch is how easily that "when everything is healthy" assumption breaks. More on that shortly.

Kubernetes DNS Naming — This Is All of It

Kubernetes service names follow one predictable pattern.

서비스이름.네임스페이스.svc.cluster.local

Piece by piece: the service name is whatever you called your Service resource. The namespace is where it lives (default, production, whatever). The svc marks it as a Service resource. And cluster.local is the cluster's default domain, which you almost never change.

So a backend-api service in the production namespace has this full name:

backend-api.production.svc.cluster.local

In practice, though, you can just write backend-api from within the same namespace. Kubernetes fills in the rest for you. That auto-completion is convenient — and it's also exactly where the ndots trap starts. Convenience and footguns usually grow from the same root.

What's Actually in /etc/resolv.conf

Exec into any pod, open /etc/resolv.conf, and you'll see something like this.

nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

Line by line.

That 10.96.0.10 on the nameserver line is the ClusterIP of the kube-dns Service. Every DNS query starts there.

The search line is the list of suffixes to append when you give a short name. Type my-service and the resolver tries my-service.default.svc.cluster.local, then my-service.svc.cluster.local, and so on. This is why short names work inside your own namespace.

ndots:5 is where things get interesting. It gets its own section below.

That "Kubernetes fills in the rest" magic from the previous section? It's this search line. Not magic — just string concatenation. And because it's that simple, the way it accidentally multiplies your query count is equally simple.

The ndots Problem — Why Your DNS Is Slow

This is the important part, and a surprising number of people never run into an explanation of it. I only clocked it while chasing down a payment API integration that was responding just a little too slowly.

ndots:5 means one thing: if a name has fewer than 5 dots in it, don't treat it as a fully qualified domain — try the search domains first.

Which blows up the moment you call an external API. api.stripe.com has 2 dots. Not 5. So CoreDNS works through this list in order:

  1. api.stripe.com.default.svc.cluster.local → nothing (NXDOMAIN)
  2. api.stripe.com.svc.cluster.local → nothing (NXDOMAIN)
  3. api.stripe.com.cluster.local → nothing (NXDOMAIN)
  4. api.stripe.com → finally

Every single external API call costs you 3 extra DNS queries. Cause and effect, separated: the cause is the default ndots value, the effect is query amplification. With few pods and rare outbound calls you'll never notice. But once your pod count climbs and external integrations pile up, traffic toward CoreDNS and your upstream resolver multiplies and you get a bottleneck. Classic threshold behavior — fine, fine, fine, and then suddenly slow.

Fixing it

Option one: put a trailing dot on the domain.

// 이렇게 하지 말고
fetch('https://api.stripe.com/charges')

// 이렇게 하세요
fetch('https://api.stripe.com./charges')

That final dot says "this is already fully qualified." The search domains get skipped entirely. It's a cheap fix — pure code change — but if your outbound calls are scattered across the codebase, you'll miss some.

Option two: change the ndots value itself. You can set it lower in the Pod spec via dnsConfig.

apiVersion: v1
kind: Pod
spec:
  dnsConfig:
    options:
    - name: ndots
      value: "2"

The upside is that it applies to the whole pod at once. The downside is the mirror-image risk: drop ndots too low and short service names in your own namespace can bypass the search domains and head straight out to the public internet. Lump a chatty-internal workload and a chatty-external workload under the same value and one of them loses. Which one you optimize for depends on what that pod actually does. There isn't a single right answer here.

Reading the CoreDNS Config

CoreDNS is driven by a config file called the Corefile, stored in the coredns ConfigMap in kube-system.

kubectl get configmap coredns -n kube-system -o yaml

A typical Corefile looks like this:

.:53 {
    errors
    health
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {
        pods insecure
        fallthrough in-addr.arpa ip6.arpa
    }
    prometheus :9153
    forward . /etc/resolv.conf
    cache 30
    loop
    reload
    loadbalance
}

What the main plugins do: kubernetes watches the Kubernetes API and resolves cluster-internal names. forward hands off anything outside the cluster to the upstream resolver. cache holds DNS answers for 30 seconds to cut load. health and ready expose endpoints for Kubernetes health checking. prometheus serves metrics on port 9153.

From an operations standpoint the two to care about first are cache and prometheus. The cache value is your first line of defense against traffic volume, and the Prometheus metrics are the only way to prove DNS is actually slow rather than just feeling slow. Saying "I think something's off with DNS" in an incident review carries very different weight than showing query counts and response latency on a graph.

Debugging DNS — Commands Worth Memorizing

The handful of commands I reach for when DNS breaks.

1. Check the CoreDNS pods

kubectl get pods -n kube-system -l k8s-app=kube-dns

2. Test resolution from a throwaway pod

kubectl run dnsutils --image=registry.k8s.io/e2e-test-images/jessie-dnsutils:1.3 --restart=Never -- sleep 3600
kubectl exec -it dnsutils -- nslookup kubernetes.default

A healthy cluster returns the ClusterIP of the kubernetes service. A timeout or SERVFAIL is your signal right there.

3. Read the CoreDNS logs

kubectl logs -n kube-system -l k8s-app=kube-dns

Look for REFUSED, SERVFAIL, I/O timeout. In my experience it splits one of two ways here: either the upstream DNS server is having a bad day, or a network policy is blocking port 53. The first is a problem outside CoreDNS, the second is inside your cluster — and which one you pick determines who you page. Letting the logs point you in a direction first is how you save an hour.

Wrapping Up

If Kubernetes networking is the highway, DNS is the navigation. Doesn't matter how well the roads are paved — if you can't find your way, the car stops.

The nice thing about digging into CoreDNS is that a vague annoyance like "why is this external API call slow sometimes?" turns into a structural answer. Even a single ndots tweak is a small architecture decision about whether internal or external traffic gets priority. DNS isn't a piece of plumbing you can ignore; it sits on top of both your operating costs and your response latency.

The less visible a component is, the more true this gets: nobody thinks about it while it works, and the whole cluster wobbles when it stops. CoreDNS lives exactly there. Next time I'll take apart another layer of Kubernetes networking the same way.

Was this post helpful?

One click helps me write the next one

#Kubernetes#CoreDNS#DNS#Networking#DevOps